M-BEV: Masked BEV Perception for Robust Autonomous Driving （AAAI 2024）

Demo for M-BEV

[AAAI 2024] M-BEV: This repo is the implementation of "M-BEV: Masked BEV Perception for Robust Autonomous Driving "

M-BEV is a perception framework to improve the robustness for camera-based autonomous driving methods. We develop a novel Masked View Reconstruction (MVR) module in our M-BEV. It mimics various missing cases by randomly masking features of different camera views, then leverages the original features of these views as self-supervision, and reconstructs the masked ones with the distinct spatio-temporal context across camera views. Via such a plug-and-play MVR, our M-BEV is capable of learning the missing views from the resting ones, and thus well generalized for robust view recovery and accurate perception in the testing.

Preparation

This implementation is built upon detr3d and petrv2. Follow their instructions to prepare for the Envirorments, besides you need to install timm and torchvision.

• Detection Data
  Follow the mmdet3d to process the nuScenes dataset (https://github.com/open-mmlab/mmdetection3d/blob/master/docs/en/data_preparation.md).

• Segmentation Data
  Download Map expansion from nuScenes dataset (https://www.nuscenes.org/nuscenes#download).

• Ckpts To verify the performance on the val set, we provide normally trained petrv2 and petrv2 trained with M-BEV weights for comparsion, the code is wpg1.

Fast inference

You can download the ckpts and put the models in the ckpts folder, then modify the config file according to your environment

tools/dist_test.sh projects/configs/petrv2/petrv2_vovnet_gridmask_mbev_p4_800x320.py /ckpts/mebv.pth 8 --eval bbox

tools/dist_test.sh projects/configs/petrv2/petrv2_vovnet_gridmask_p4_800x320.py /ckpts/petrv2.pth 8 --eval bbox

Acknowledgement

Many thanks to the authors of mmdetection3d, detr3d and petr.