Add support for multi-modal NuScenes Detection #1339
Conversation
| @@ -23,6 +24,18 @@ def __init__(self, root_path, augmentor_configs, class_names, logger=None): | |||
| cur_augmentor = getattr(self, cur_cfg.NAME)(config=cur_cfg) | |||
| self.data_augmentor_queue.append(cur_augmentor) | |||
|
|
|||
| def disableAugmentation(self, augmentor_configs): | |||
There was a problem hiding this comment.
I will be better to use disable_augmentation as the function name to keep the naming style consistent along pcdet
There was a problem hiding this comment.
Ok, I will adjust the function name.
| @@ -130,6 +131,30 @@ def __getitem__(self, index): | |||
| """ | |||
| raise NotImplementedError | |||
|
|
|||
| def set_lidar_aug_matrix(self, data_dict): | |||
There was a problem hiding this comment.
I think it will be better to move this function to data_augmentor.py and call it after forward all data augmentation.
There was a problem hiding this comment.
The function set_lidar_aug_matrix needs to be called during testing and identity matrix is generated, so it may not be appropriate to move it data_augmentor.py.
| @@ -137,6 +181,60 @@ def __getitem__(self, index): | |||
| 'gt_names': info['gt_names'] if mask is None else info['gt_names'][mask], | |||
| 'gt_boxes': info['gt_boxes'] if mask is None else info['gt_boxes'][mask] | |||
| }) | |||
| if self.use_camera: | |||
There was a problem hiding this comment.
It will be better to add a new function to include all following added codes. I think this will make __getitem__ easier to understand.
There was a problem hiding this comment.
I will define a function called load_camera_info to include these codes.
There was a problem hiding this comment.
Get it. Thanks a lot for your contribution. I think after the following adjustment, we can merge this PR. I will make sure with shaoshuai @sshaoshuai.
|
Great. @sshaoshuai will have a final check and then merge it. |
|
Hi @chenshi3 , thank you very much for your work on supporting TransFusion! May I ask if it's possible to train TransFusion-Lidar model with earlier version of pcdet? Could you please give me some hints on how should I change the related codes? (I am not sure if it's enough to just add the transfusion-lidar head and other model-related codes, or should I also modify other parts to support it?) |
|
Hi @chenshi3, thank you very much for your contribution. However, I have a question about the data augmentation. In the config file, rotation and scale augmentations are applied to the point cloud. However, this augmentation alone has nothing to do with the image being augmented in the next step. Can you explain this? Does it affect the efficiency of the model? |
Hi,
I'm submitting a pull request to add support for multi-modal detection on the NuScenes dataset. This PR consists of four main components:
(1) Support for the TransFusion-Lidar head.
(2) Implementation of the fade strategy, which disables data augmentations in the last several epochs during training.
(3) Support for the multi-modal NuScenes dataset, including data processing and dataset usage.
(4) Introduction of the multi-modal detector BEVFusion, including support for image backbones, LSS Transform, and Bev pooling.
Best regards,
Chen Shi