Errors encountered while validating the model

[jsc122]

After the ct _ bev _ backbone file was modified and training, errors in the following figure were encountered while verifying on the validation set.
At the same time, the source code of the model is used to verify the three classes trained under the profile rdiou_3cat.yaml, and the following error is also reported.
Therefore, I would like to ask the general solution to this mistake. I look forward to and thank you for your reply!

[hlsheng1]

Hi, maybe the nan problem. Pls refer to issue 15.

[jsc122]

Thanks for your reply. What you mean is that to solve the problem that all the detection accuracy of the validation set are zero, I can refer to issue 15?

[hlsheng1]

Hi, maybe the training loss is nan and model parameters are nans in your training process. I fixed this issue in the latest version.

[jsc122]

Thanks for your quick modification, but it seems like a new error right now.
When I trained with the new anchor_head_rdiou.py and anchor_head_rdiou_3cat.py files, there was an error in the figure below.
When I removed the code "box _ preds [..., 3:6] = torch.clamp (box _ preds [..., 3:6], max = 3.0)" , the program could train normally, but the accuracy of the validation set was still zero. I looked at the method in issue 15 which changed the clamp max value for l,w,h but it has no effect.

[hlsheng1]

Hi, sry for confusing. Fixed now, u can try it again. @jsc122

[jsc122]

Hi, I saw the new modification in the file. I'm very upset. I don't know if I have problems here, my verification result is still zero. I replace the original two files through the modified two files, the other files in the model all not modified and consistent with github, I use the four GPU, by entering the following two instructions training a class and three classes, model can normal training, but unfortunately, their validation results are still all zero.

Two instructions I entered are as follows:
bash tools/scripts/dist_train.sh 4 --cfg_file /root/autodl-tmp/RDIoU/tools/cfgs/kitti_models/rdiou_3cat.yaml
bash tools/scripts/dist_train.sh 4 --cfg_file /root/autodl-tmp/RDIoU/tools/cfgs/kitti_models/rdiou.yaml

The image of validating three classes is as follows:

[hlsheng1]

Hi, have you re-trained the model? If so, whether your loss is normal?

[jsc122]

Hi, yes, I retrained the model, and I tried to train 10 epochs on 4 GPUs today, and the verification accuracy is still zero, which is the same as the figure in my last reply.
At the same time, I replaced the Swin-transformer part of "ct _ bev _ backbone" with some MLP layers in a single GPU today, and the validation accuracy was zero, too , as shown in Figure 1 below. I went to check the loss, as shown in Figure 2 below, and I think the loss is fine. The learning rate is shown in Figure 3 below.
I am normal when using the original code to train the car class and the three classes separately on a single GPU, only with low accuracy. The first problem is: after replacing the Swin-transformer part of the car class with some MLP layers in a single GPU, the verification accuracy is all zero. The second problem is: when the original code is used to train car class and three classes on multiple GPUs, the validation accuracy is all zero.

[hlsheng1]

Hi, Never seen this before.

Can you reproduce the original SECOND network? If so, RDIoU should be okay. The differences are three paets: 1. the backbone, 2. the dence head, 3. the learning rate and training epoch should be kept as default.

I have trained our model on 1 GPU, 4 GPUs and 8 GPUs, and all the experiments exhibit normally.

[jsc122]

Ok, thanks for your advice, I'll try it more.
I found a phenomenon that when I replaced the "Swin _ transformer" part of "ct _ bev _ backbone" with the MLP structure, after training the model with a GPU, each of the "checkpoint_epoch_XX.pth" files generated was large and had about 2G. But the "checkpoint_epoch_XX.pth" files generated by the original code are only a few tens of Mb. The file size varies too much.
Do you think I can check what parts of the model to ensure if has problems?

[songuilin]

Encountered the same problem, have you solved it, eval_utils.py/ pred_dicts, ret_dict = model(batch_dict) predicts is a [ ] ,do not have some data

[hlsheng1]

Can you reproduce SECOND network? I guess it may be caused by pcdet setup version. Alternatively, you can utilize the latest OpenPCDet codebase version, then copy the rdiou code with corresponding dense_head, 2D backbone, yaml file and so on.

[jsc122]

Encountered the same problem, have you solved it, eval_utils.py/ pred_dicts, ret_dict = model(batch_dict) predicts is a [ ] ,do not have some data

Is your verification result all zero , too? I'm sorry. I haven't solved this problem yet. Have you found a solution now?

[jsc122]

Can you reproduce SECOND network? I guess it may be caused by pcdet setup version. Alternatively, you can utilize the latest OpenPCDet codebase version, then copy the rdiou code with corresponding dense_head, 2D backbone, yaml file and so on.

Thanks so much for your advice, I will try some more.

[songuilin]

Can you reproduce SECOND network? I guess it may be caused by pcdet setup version. Alternatively, you can utilize the latest OpenPCDet codebase version, then copy the rdiou code with corresponding dense_head, 2D backbone, yaml file and so on.
Thank you very much for your reply. I have found the reason. I used the pre training weight evaluation,