Limitations of the empirical Fisher approximation: Code

This repository contains code to reproduce the experiments in

Limitations of the empirical Fisher approximation
Frederik Kunstner, Lukas Balles and Philipp Hennig
arxiv.org/abs/1905.12558

Go to

• Installation
• Usage
• Known issues

---

Installation

• Main files: Download the repository and install the modules in the libs folder:

  mkdir limitation-ef
  git clone https://github.com/fKunstner/limitations-empirical-fisher.git limitation-ef
  cd limitation-ef/libs/eftk
  pip install .
  cd ../efplt
  pip install .
  

• Dependencies: Install through pip install [x]:

  numpy, scipy, matplotlib, scikit-learn and tqdm.

• Datasets: the content of datasets.zip should be stored in ~/data,
  (edit eftk/datasets.py for another location)

---

Usage

The main file to run the experiment is experiments/main.py.

Calling python experiments/main.py -h should output the following help:

usage: main.py [-h] (-misspec | -optim | -vecfield) [-run] [-plot] [-appendix]
               [-save] [-show]

Experiment Runner

optional arguments:
  -h, --help  show this help message and exit

Experiment selection:
  -misspec    Misspecification experiment
  -optim      Optimization experiment
  -vecfield   Vector field visualization

Action selection:
  At least one of [-run, -plot] is required

  -run        Runs the experiment and save results as a .pk file
  -plot       Plots the result from a .pk file (requires -save and/or -show)
  -appendix   Also run/plot the experiments in the appendix

Plotting options:
  At least one of [-save, -show] is required if plotting

  -save       Save the plots
  -show       Show the plots

TL;DR:

typical use: $ python main.py -expName -run -plot -show

  (-misspec | -optim | -vecfield)   Choose experiment
  -run                              Run expensive computations (grid search) and save results
  -plot (-show | -save)             Plot results and (show or save) plots
  -appendix                         Run additional experiments

• -vecfield experiment (Fig. 1): -run needed, but short (~10min)

  python main.py -vecfield -plot -show
  

• -misspec experiment (Fig. 2): -run not needed

  python main.py -misspec -plot -show
  

• -optim experiment (Fig. 3) -run needed, large grid search (~2hours)

  python main.py -optim -run
  python main.py -optim -plot -show
  

• -appendix (Fig. 4, 5, 6)

  python main.py -misspec -plot -show -appendix
  python main.py -optim -run -plot -show -appendix
  

---

Known issues

• Plotting needs a working Latex installation (the figures use Matplotlib's Latex backend)
• On Ubuntu, you might need to install apt-get install dvipng