Informative Features for Model Comparison
Given two candidate models, and a set of target observations, we address the problem of measuring the relative goodness of fit of the two models. We propose two new statistical tests which are nonparametric, computationally efficient (runtime complexity is linear in the sample size), and interpretable. As a unique advantage, our tests can produce a set of examples (informative features) indicating the regions in the data domain where one model fits significantly better than the other. In a real-world problem of comparing GAN models, the test power of our new test matches that of the state-of-the-art test of relative goodness of fit, while being one order of magnitude faster.
Wittawat Jitkrittum is a postdoctoral researcher working with Bernhard Schoelkopf at Max Planck Institute for Intelligent Systems, Germany. He earned his PhD from Gatsby Unit, University College London with a thesis on informative features for comparing distributions. He received a best paper award at NeurIPS 2017 and the ELLIS PhD award 2019 for outstanding dissertation. Wittawat has broad research interests covering kernel methods, deep generative models, and approximate Bayesian inference. He is a co-organizer of the first Southeast Asia Machine Learning School (SEAMLS 2019) in Jakarta, Indonesia, and the first Machine Learning Research School (MLRS 2019) in Bangkok, Thailand.