CoCMT

This repo is the official implementation of "CoCMT: Communication-Efficient Cross-Modal Transformer for Collaborative Perception" (IROS 2025).

Environment Setup

Step 1: Clone Repository

git clone https://github.com/taco-group/COCMT.git
cd COCMT

Step 2: Install MMDetection3D Dependencies

First, install the required MMDetection3D ecosystem packages:

pip install mmcv-full==1.6.0
pip install mmdet==2.24.0
pip install mmsegmentation==0.29.1
pip install mmdet3d==1.0.0rc5

For more details on MMDetection3D installation, please refer to the CMT project.

Step 3: Install CoCMT

cd mmdetection3d

pip install -v -e .

cd ..

pip install -v -e .

pip install -r requirements.txt

python setup.py develop

python3 opencood/utils/setup.py build_ext --inplace

pip list

Step 4: Setup Directories

mkdir opencood/logs

cp -r opencood/modality_assign opencood/logs/heter_modality_assign

Dataset Preparation

OPV2V Dataset

Please refer to OpenCOOD for dataset download and preparation.

V2V4Real Dataset

Please refer to V2V4Real for dataset download and preparation.

Train

OPV2V 3 Branch

# Train CoCMT Hetero-LiDAR-Camera based on nofusion weights
python opencood/tools/lightning_train.py --hypes_yaml opencood/hypes_yaml/opv2v/CMT/hetero/CMT-Camera_Lidar_att_fuse.yaml --load_cmt_camera_nofusion_weights --cmt_camera_nofusion_weights_path [CMT_Camera_NoFusion_weights]
--load_cmt_lidar_nofusion_weights --load_cmt_lidar_nofusion_weights [CMT_LiDAR_NoFusion_weights]


# Train CoCMT Hetero-LiDAR-Camera based on nofusion weights and frozen stage1: nofusion weights
python opencood/tools/lightning_train.py --hypes_yaml opencood/hypes_yaml/opv2v/CMT/hetero/CMT-Camera_Lidar_att_fuse.yaml --load_cmt_camera_nofusion_weights --cmt_camera_nofusion_weights_path [CMT_Camera_NoFusion_weights]
--load_cmt_lidar_nofusion_weights --load_cmt_lidar_nofusion_weights [CMT_LiDAR_NoFusion_weights] --fix_camera_backbone --fix_lidar_backbone

# Train CoCMT Homo-LiDAR based on nofusion weights
python opencood/tools/lightning_train.py --hypes_yaml opencood/hypes_yaml/opv2v/CMT/lidar/CMT-L_attfuse.yaml --load_cmt_lidar_nofusion_weights --load_cmt_lidar_nofusion_weights [CMT_LiDAR_NoFusion_weights]

# Train CoCMT Homo-LiDAR based on nofusion weights and frozen stage1: nofusion weights
python opencood/tools/lightning_train.py --hypes_yaml opencood/hypes_yaml/opv2v/CMT/lidar/CMT-L_attfuse.yaml --load_cmt_lidar_nofusion_weights --load_cmt_lidar_nofusion_weights [CMT_LiDAR_NoFusion_weights] --fix_lidar_backbone

# Train CoCMT Homo-Camera based on nofusion weights
python opencood/tools/lightning_train.py --hypes_yaml opencood/hypes_yaml/opv2v/CMT/camera/CMT-C_attfuse.yaml --load_cmt_camera_nofusion_weights --load_cmt_camera_nofusion_weights [CMT_Camera_NoFusion_weights]

# Train CoCMT Homo-Camera based on nofusion weights and frozen stage1: nofusion weights
python opencood/tools/lightning_train.py --hypes_yaml opencood/hypes_yaml/opv2v/CMT/camera/CMT-C_attfuse.yaml --load_cmt_camera_nofusion_weights --load_cmt_camera_nofusion_weights [CMT_Camera_NoFusion_weights] --fix_camera_backbone

# Train CoCMT LiDAR nofusion weights
python opencood/tools/lightning_train.py --hypes_yaml opencood/hypes_yaml/opv2v/CMT/lidar/CMT-L_pointpillar.yaml

# Train CoCMT Camera nofusion weights
python opencood/tools/lightning_train.py --hypes_yaml opencood/hypes_yaml/opv2v/CMT/camera/CMT-C_resnet50.yaml

V2V4Real

# Train CoCMT Homo-LiDAR based on nofusion weights
python opencood/tools/lightning_train.py --hypes_yaml opencood/hypes_yaml/v2v4real/CMT/lidar/CMT-L_attfuse_v2v4real.yaml --load_cmt_lidar_nofusion_weights --load_cmt_lidar_nofusion_weights [CMT_LiDAR_NoFusion_weights]

# Train CoCMT Homo-LiDAR based on nofusion weights and frozen stage1: nofusion weights
python opencood/tools/lightning_train.py --hypes_yaml opencood/hypes_yaml/v2v4real/CMT/lidar/CMT-L_attfuse_v2v4real.yaml --load_cmt_lidar_nofusion_weights --load_cmt_lidar_nofusion_weights [CMT_LiDAR_NoFusion_weights] --fix_lidar_backbone

# Train CoCMT nofusion weights
python opencood/tools/lightning_train.py --hypes_yaml opencood/hypes_yaml/v2v4real/CMT/lidar/CMT-L_pointpillar_v2v4real.yaml

Eval

OPV2V

bash test_epochs.sh <inference_script> <model_dir> <fusion_method> <epoch list> --lightning --qbm --range "102.4,102.4"
bash test_epochs.sh opencood/logs/cmt_camera_lidar_att_fuse_2024_03_25_10_57_59 intermediate 0,1 --lightning --qbm --range "102.4,102.4"

V2V4Real

bash test_epochs.sh <inference_script> <model_dir> <fusion_method> <epoch list> --lightning --qbm --homo --range "70.4,38.4"
bash test_epochs.sh opencood/logs/cmt_lidar_intermediatefusion_v2v4real intermediate 0,1 --lightning --qbm --homo --range "70.4,38.4"

Acknowledgement

We build our framework on top of CMT, HEAL, OpenCOOD, and V2V4Real. Please refer to the following repos:

• CMT: https://github.com/junjie18/CMT
• HEAL: https://github.com/yifanlu0227/HEAL
• OpenCOOD: https://github.com/DerrickXuNu/OpenCOOD
• V2V4Real: https://github.com/ucla-mobility/V2V4Real