[AAAI 2025] RCTrans: Radar-Camera Transformer via Radar Densifier and Sequential Decoder for 3D Object Detection

Yiheng Li, Yang Yang and Zhen Lei

MAIS&CASIA, UCAS

Introduction

This repository is an official implementation of RCTrans.

News

• [2024/12/18] Camera Ready version is released.
• [2024/12/13] Codes and weights are released.
• [2024/12/10] RCTrans is accepted by AAAI 2025 🎉🎉.

Environment Setting

conda create -n RCTrans python=3.8
conda activate RCTrans
pip install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0 -f https://download.pytorch.org/whl/cu111/torch_stable.html
pip install flash-attn==0.2.2 --no-build-isolation
pip install mmdet==2.28.2
pip install mmsegmentation==0.30.0
cd mmdetection3d
pip install -v -e .
cd ..
pip install ipython
pip install fvcore
pip install spconv-cu111==2.1.21
pip install yapf==0.40.0
pip install setuptools==59.5.0
pip install ccimport==0.3.7
pip install pccm==0.3.4
pip install timm

Data Preparation

python tools/create_data_nusc.py --root-path ./data/nuscenes --out-dir ./data --extra-tag nuscenes_radar --version v1.0

Folder structure

RCTrans
├── projects/
├── mmdetection3d/
├── tools/
├── ckpts/
├── data/
│   ├── nuscenes/
│   │   ├── maps/
│   │   ├── samples/
│   │   ├── sweeps/
│   │   ├── v1.0-test/
|   |   ├── v1.0-trainval/
|   ├── nuscenes_radar_temporal_infos_train.pkl
|   ├── nuscenes_radar_temporal_infos_val.pkl
|   ├── nuscenes_radar_temporal_infos_test.pkl

Or you can directly use our pre-generated pickles here. Val Train Test

Train & Inference

Train

export PYTHONPATH=$PYTHONPATH:/xxx/xxx/RCTrans/
bash tools/dist_train.sh projects/configs/RCTrans/rcdetr_90e_256×704_swinT.py 8 --work-dir work_dirs/xxx/

Evaluation

bash tools/dist_test.sh projects/configs/RCTrans/rcdetr_90e_256×704_swinT.py ckpts/xxx.pth 8 --eval bbox

Tracking

# following the scripts of CenterPoint.

Speed

python tools/benchmark.py projects/configs/test_speed/rcdetr_90e_256×704.py --checkpoint ckpts/xxx.pth

Visualize

python tools/visualize.py
# We also recommand to use the Visualization codes from BEVFormer, which is really nice.

Weights

Download these backbones: Swin_T, ResNet-18, ResNet-50, VovNet, and put them into the RCTrans/ckpts/.

We give the pre-trained in Table 1: Swint-train, ResNet18-train, ResNet50-train.

Acknowledgements

We thank these great works and open-source codebases: MMDetection3d, BEVFormer, DETR3D, PETR, StreamPETR, CMT, CenterPoint, FUTR3D.

Citation

If you find our work is useful, please give this repo a star and cite our work as:

@article{li2024rctrans,
  title={RCTrans: Radar-Camera Transformer via Radar Densifier and Sequential Decoder for 3D Object Detection},
  author={Li, Yiheng and Yang, Yang and Lei, Zhen},
  journal={arXiv preprint arXiv:2412.12799},
  year={2024}
}