Evidential Prior Guided Neural Collapse for Open World Object Detection

🔥 Highlights

• Challenges with Open World Object Detection: Existing OWOD methods may raise unknown recall but fail to assign reliable, discriminative confidence to unknown objects—degrading further across incremental stages.

• ENC DETR: Evidential Prior Guided Neural Collapse DETR unifies representation stabilization and class-agnostic objectness scoring for robust open world learning.

• Stable Representation Space: ENC-DETR employs equiangular base vectors to globally align features with predefined optimal classifiers, maximising inter-class separability, and evidential prior further pulls features toward their prototypes, keeping the representation space stable throughout all incremental stages.

• Discriminative Objectness: ENC-DETR Fuse evidential classification evidence and regression latent cues to build reliable self-supervised signals.

• “Query Conflict” Insight: We identify a query conflict issue inherent in pseudo-label based self-supervised learning of unknown classes, which leads models into decision conflicts. To address this, ENC-DETR introduces an uncertainty-based regularization at decision boundaries, enabling fine-grained adjustments.

• Significant Gains: On M-OWODB / S-OWODB, unknown-class AP improves ≈2–9× while mitigating representation collapse & catastrophic forgetting.

📌 Overview

Overall architecture of ENC-DETR: (a) Evidential prior guided ETF classification head: Normalizes features, aligns with predefined EBVs, transforms distances into evidence, and incorporates a prior to produce Dirichlet parameters. The mean of distribution predicts the probabilities and quantifies dissonance uncertainty. (b) Joint objectness head: Fuses classification and regression evidential spaces to learn a robust, class-agnostic objectness score. (c) Evidential regression head: Uses four MLP layers to predict Normal Inverse Gamma parameters gives box coordinates, while the distribution captures box regression uncertainty. For more, see the manuscript.

🧩 ENC-DETR: Uncertainty Awareness and Representation Stability

(a) Vanilla DNN lacks uncertainty modeling. (b) Evidential DNN models distributions over distributions, enabling uncertainty quantification. (c) EBVs-optimized DNN maintains a stable feature space during incremental learning. (d) Evidential DNN with NC achieves both uncertainty quantification and stable representation. For more, see the manuscript.

📊 Visualize of ENC-DETR

📈 Results

	Task1			Task2			Task3			Task4
Method	U-Rec	U-AP	mAP	U-Rec	U-AP	mAP	U-Rec	U-AP	mAP	mAP
OW-DETR	7.5	0.1	59.2	6.2	0.1	42.9	5.7	0.1	30.8	27.8
PROB	19.4	1.2	59.5	17.4	0.8	44.0	19.6	0.5	36.0	31.5
ALLOW	13.6	4.9	59.3	10.0	0.7	40.6	9.8	0.4	37.3	30.6
ENC	20.2	9.2	61.0	17.4	9.1	48.0	19.6	9.1	38.4	34.1

🛠️ Requirements and Installation

Python Environment

We have trained and tested our models on Ubuntu 16.04, CUDA 11.1/11.3, GCC 5.4.0, Python 3.10.4

conda create --name enc python==3.10.4
conda activate enc
pip install -r requirements.txt
pip install torch==1.12.0+cu113 torchvision==0.13.0+cu113 torchaudio==0.12.0 --extra-index-url https://download.pytorch.org/whl/cu113

Backbone features

Download the self-supervised backbone from here and add in models folder.

Compiling CUDA operators

cd ./models/ops
sh ./make.sh
# unit test (should see all checking is True)
python test.py

📂 Dataset Preparation

The file structure:

ENC/
└── data/
    └── OWOD/
        ├── JPEGImages
        ├── Annotations
        └── ImageSets
            ├── OWDETR
            ├── TOWOD
            └── VOC2007

The splits are present inside data/OWOD/ImageSets/ folder.

1. Download the COCO Images and Annotations from coco dataset into the data/ directory.
2. Unzip train2017 and val2017 folder. The current directory structure should look like:

ENC/
└── data/
    └── coco/
        ├── annotations/
        ├── train2017/
        └── val2017/

4. Move all images from train2017/ and val2017/ to JPEGImages folder.
5. Use the code coco2voc.py for converting json annotations to xml files.
6. Download the PASCAL VOC 2007 & 2012 Images and Annotations from pascal dataset into the data/ directory.
7. untar the trainval 2007 and 2012 and test 2007 folders.
8. Move all the images to JPEGImages folder and annotations to Annotations folder.

Currently, we follow the VOC format for data loading and evaluation

🔧 Prepare EBVs

Run EBVs\EBVs.ipynb to generate Equiangular Basis Vectors and load them in models\enc_deformable_detr.py

🚀 Training

Training on single node

To train ENC-DETR on a single node with 4 GPUS, run

bash ./run.sh

**note: you may need to give permissions to the .sh files under the 'configs' and 'tools' directories by running chmod +x *.sh in each directory.

By editing the run.sh file, you can decide to run each one of the configurations defined in \configs:

1. M_OWOD_BENCHMARK.sh - training for tasks 1-4 on the MOWOD Benchmark.
2. S_OWOD_BENCHMARK.sh - training for tasks 1-4 on the SOWOD Benchmark.
3. IL_10_10.sh - training for incremental setting 10_10 on VOC2007 dataset
4. IL_15_5.sh - training for incremental setting 15_5 on VOC2007 dataset
5. IL_19_1.sh - training for incremental setting 19_1 on VOC2007 dataset

Training on slurm cluster

To train ENC-DETR on a slurm cluster having 2 nodes with 8 GPUS each (not tested), run

bash run_slurm.sh

**note: you may need to give permissions to the .sh files under the 'configs' and 'tools' directories by running chmod +x *.sh in each directory.

🧪 Test

To evaluate ENC-DETR on a single node with 4 GPUS, run

bash ./run_eval.sh

**note: you may need to give permissions to the .sh files under the 'configs' and 'tools' directories by running chmod +x *.sh in each directory.

By editing the run_eval.sh file, you can decide to run each one of the configurations defined in \configs:

1. EVAL_M_OWOD_BENCHMARK.sh - evaluation of tasks 1-4 on the MOWOD Benchmark.
2. EVAL_S_OWOD_BENCHMARK.sh - evaluation of tasks 1-4 on the SOWOD Benchmark.
3. EVAL_IL.sh - evaluation of all incremental setting on VOC2007 dataset

**note: you may need to give permissions to the .sh files under the 'configs' and 'tools' directories by running chmod +x *.sh in each directory.

Note: Please check the [xxx], [xxx] repository for more training and evaluation details.

👍 Acknowledgements

ENC-DETR builds on previous works' code bases such as [xxx], [xxx], [xxx]. If you found ENC-DETR useful please consider citing these works as well.