This codebase provides a Pytorch implementation for the paper CIDER: How to Exploit Hyperspherical Embeddings for Out-of-Distribution Detection? at ICLR 2023.
Out-of-distribution (OOD) detection is a critical task for reliable machine learning. Recent advances in representation learning give rise to distance-based OOD detection, where testing samples are detected as OOD if they are relatively far away from the centroids or prototypes of in-distribution (ID) classes. However, prior methods directly take off-the-shelf contrastive losses that suffice for classifying ID samples, but are not optimally designed when test inputs contain OOD samples. In this work, we propose CIDER, a novel representation learning framework that exploits hyperspherical embeddings for OOD detection. CIDER jointly optimizes two losses to promote strong ID-OOD separability: a dispersion loss that promotes large angular distances among different class prototypes, and a compactness loss that encourages samples to be close to their class prototypes. We analyze and establish the unexplored relationship between OOD detection performance and the embedding properties in the hyperspherical space, and demonstrate the importance of dispersion and compactness. CIDER establishes superior performance, outperforming the latest rival by 19.36% in FPR95.
Remarks: We are actively working on improving the codebase for reproducibility and ease of use. Stay tuned for more updates :).
Aug 12: In alignment with prior works on the ImageNet-100 subset (the script for generating the subset is provided here), we've also finetuned CIDER with the default hyperparameters (e.g., 10 epochs with ResNet-34) and report the performance below for reference. The results are averaged over 3 seeds:
| OOD | FPR95 | AUROC | AUPR |
|---|---|---|---|
| SUN | 32.84 ± 1.86 | 92.24 ± 0.38 | 91.72 ± 0.28 |
| Places365 | 45.31 ± 1.74 | 90.10 ± 0.48 | 90.90 ± 0.42 |
| Textures | 10.03 ± 0.23 | 98.21 ± 0.02 | 98.37 ± 0.02 |
| iNaturalist | 15.42 ± 2.38 | 97.28 ± 0.31 | 97.80 ± 0.22 |
| AVG | 25.90 ± 1.47 | 94.46 ± 0.29 | 94.70 ± 0.22 |
The checkpoint is available here.
Apr 28: Updated prototype initialization with ID training set (Thanks zjysteven); changed default weight scale from 2.0 to 1.0 in train_cider_cifar100.sh for better performance.
The default root directory for ID and OOD datasets is datasets/. We consider the following (in-distribution) datasets: CIFAR-10, CIFAR-100, and ImageNet-100.
Small-scale OOD datasets For small-scale ID (e.g. CIFAR-10), we use SVHN, Textures (dtd), Places365, LSUN-C (LSUN), LSUN-R (LSUN_resize), and iSUN.
OOD datasets can be downloaded via the following links (source: ATOM):
- SVHN: download it and place it in the folder of
datasets/small_OOD_dataset/svhn. Then runpython utils/select_svhn_data.pyto generate test subset. - Textures: download it and place it in the folder of
datasets/small_OOD_dataset/dtd. - Places365: download it and place it in the folder of
datasets/ood_datasets/places365/test_subset. We randomly sample 10,000 images from the original test dataset. - LSUN: download it and place it in the folder of
datasets/small_OOD_dataset/LSUN. - LSUN-resize: download it and place it in the folder of
datasets/small_OOD_dataset/LSUN_resize. - iSUN: download it and place it in the folder of
datasets/small_OOD_dataset/iSUN.
For example, run the following commands in the root directory to download LSUN-C:
cd datasets/small_OOD_dataset
wget https://www.dropbox.com/s/fhtsw1m3qxlwj6h/LSUN.tar.gz
tar -xvzf LSUN.tar.gz
The directory structure looks like this:
datasets/
---CIFAR10/
---CIFAR100/
---small_OOD_dataset/
------dtd/
------iSUN/
------LSUN/
------LSUN_resize/
------places365/
------SVHN/Large-scale OOD datasets For large-scale ID (e.g. ImageNet-100), we use the curated 4 OOD datasets from iNaturalist, SUN, Places, and Textures, and de-duplicated concepts overlapped with ImageNet-1k. The datasets are created by Huang et al., 2021 .
The subsampled iNaturalist, SUN, and Places can be downloaded via the following links:
wget http://pages.cs.wisc.edu/~huangrui/imagenet_ood_dataset/iNaturalist.tar.gz
wget http://pages.cs.wisc.edu/~huangrui/imagenet_ood_dataset/SUN.tar.gz
wget http://pages.cs.wisc.edu/~huangrui/imagenet_ood_dataset/Places.tar.gz
The directory structure looks like this:
datasets/
---ImageNet100/
---ImageNet_OOD_dataset/
------dtd/
------iNaturalist/
------Places/
------SUN/Evaluate pre-trained checkpoints
Our checkpoints can be downloaded here for CIFAR-100 and CIFAR-10. Create a directory named checkpoints/[ID_DATASET] in the root directory of the project and put the downloaded checkpoints here. For example, for CIFAR-10 and CIFAR-100:
checkpoints/
---CIFAR-10/
------ckpt_c10/
------checkpoint_500.pth.tar
---CIFAR-100/
------ckpt_c100/
------checkpoint_500.pth.tar
The following scripts can be used to evaluate the OOD detection performance:
sh scripts/eval_ckpt_cifar10.sh ckpt_c10 #for CIFAR-10
sh scripts/eval_ckpt_cifar100.sh ckpt_c100 # for CIFAR-100
Evaluate custom checkpoints
If the default directory to save checkpoints is not checkpoints, create a softlink to the directory where the actual checkpoints are saved and name it as checkpoints. For example, checkpoints for CIFAR-100 (ID) are structured as follows:
checkpoints/
---CIFAR-100/
------name_of_ckpt/
---------checkpoint_500.pth.tarTrain from scratch
We provide sample scripts to train from scratch. Feel free to modify the hyperparameters and training configurations.
sh scripts/train_cider_cifar10.sh
sh scripts/train_cider_cifar100.sh
Fine-tune from ImageNet pre-trained models
We also provide fine-tuning scripts on large-scale datasets such as ImageNet-100.
sh scripts/train_cider_imgnet100.sh # To be updated
If you find our work useful, please consider citing our paper:
@inproceedings{ming2023cider,
title={How to Exploit Hyperspherical Embeddings for Out-of-Distribution Detection?},
author={Yifei Ming and Yiyou Sun and Ousmane Dia and Yixuan Li},
booktitle={The Eleventh International Conference on Learning Representations },
year={2023},
url={https://openreview.net/forum?id=aEFaE0W5pAd}
}
For more in-depth discussions on the method and extensions, feel free to drop an email at ming5@wisc.edu :)