Multiple Model Comparison Test.

This repository contains resources for finding the best model with multiple model comparison. The algorithm is described in our paper,

Kernel Stein Tests for Multiple Model Comparison
Jen Ning Lim, Makoto Yamada, Bernhard Schölkopf, Wittawat Jitkrittum
NeurIPS 2019

How to install?

Requires numpy, autograd, matplotlib and SciPy. The package can be installed with pip command.

pip install -e /path/to/the/folder/of/this/repo/after/clone

Or alternatively,

pip install git+https://github.com/jenninglim/model-comparison-test.git

Once installed, you should be able to do import reltest without any error.

Demo

See notebooks/demo_reltest.ipynb.

Reproducing results

See reproduce-results.

Disclaimer

The $p$-values produced from the modules may not be accurate. At the time of writting this code there were several problems with the scipy.stats.truncnorm module. See 1, 2, 3 and 4. This may have been fixed in the pull request 5 in SciPy 1.4.0.

The current implementation relies on an accurate approximation of the inverse CDF in the tail regions of the truncated normal (see 6). The inverse CDF is used to calculate the rejection threshold.

See also

• Kernel Relative Test of Fit: Code Paper
• Kernel Goodness of Fit: Code, Paper