Alexander Fishkov1*, Kajetan Schweighofer2*, Mykyta Ielanskyi2, Nikita Kotelevskii1, Mohsen Guizani1, Maxim Panov1
In this work, we extend recent advances on uncertainty estimation in classification to the regression setting. Under parameteric assumptions, we derive closed-form expressions for regression uncertainty measures and MC estimators thereof. Our broad evaluation on synthetic and real-world regression datasets provides guidance for selecting reliable uncertainty measures for the task at hand. Implementations of all considered uncertainty measures can be found here: uncertainty_measures.py.
conda env create -f environment.yml
Important are python=3.12, pytorch=2.4 and torchvision=0.19 for reproducability.
For the toy tasks dots and arrow, execute the notebooks notebooks/check_dots.ipynb and notebooks/check_arrows.ipynb.
They both contain calls to the generate_and_save_data function of the specific task, which creates the datasets necessary for training and testing.
For cityscapes, follow the instructions in datasets/README.md first and then execute datasets/generate_cityscapes.ipynb to preprocess the dataset for training.
To train models, call train.py. Example configs can be seen in run_experiment.sh.
After these runs are finished, one needs to obtain probs on the test dataset for individual ensemble models and save the test targets for convinience.
This is done by calling evaluate.py.
All configs can be seen in run_evaluation.sh.
Similarly for active learning, which is run by calling active_learning.py.
The final results for individual experiments are obtained throught the analyze_* and generate_* notebooks.
To play around with the difference between the usual parameterization of the Gaussian likelihood and the natural parametrization, there is the toy sine example in toy_regression.ipynb.