Using the pretrained MAE model to run the downstream task and couldn't work

[Yoooss]

I tried to use the downloaded pretrained MAE model to run the downstream object detection task , but it raised error and couldn't run.
The command I used is:
"
bash tools/benchmarks/mmdetection/mim_dist_train_c4.sh configs/benchmarks/mmdetection/voc0712/faster_rcnn_r50_c4_mstrain_24k_voc0712ls.py /home/ls/mmselfsup/checkpoints/mae_vit-base-p16_8xb512-coslr-400e-fp16_in1k_20220825-bc79e40b.pth 1
"

The error It raised is that:
"
File "/home/ls/anaconda3/envs/mmselfsupv1/lib/python3.8/site-packages/mmdet/.mim/tools/train.py", line 120, in
main()
File "/home/ls/anaconda3/envs/mmselfsupv1/lib/python3.8/site-packages/mmdet/.mim/tools/train.py", line 116, in main
runner.train()
File "/home/ls/anaconda3/envs/mmselfsupv1/lib/python3.8/site-packages/mmengine/runner/runner.py", line 1621, in train
self._init_model_weights()
File "/home/ls/anaconda3/envs/mmselfsupv1/lib/python3.8/site-packages/mmengine/runner/runner.py", line 872, in _init_model_weights
model.init_weights()
File "/home/ls/anaconda3/envs/mmselfsupv1/lib/python3.8/site-packages/mmengine/model/base_module.py", line 114, in init_weights
m.init_weights()
File "/home/ls/anaconda3/envs/mmselfsupv1/lib/python3.8/site-packages/mmengine/model/base_module.py", line 103, in init_weights
initialize(self, self.init_cfg)
File "/home/ls/anaconda3/envs/mmselfsupv1/lib/python3.8/site-packages/mmengine/model/weight_init.py", line 607, in initialize
_initialize(module, cp_cfg)
File "/home/ls/anaconda3/envs/mmselfsupv1/lib/python3.8/site-packages/mmengine/model/weight_init.py", line 515, in _initialize
func(module)
File "/home/ls/anaconda3/envs/mmselfsupv1/lib/python3.8/site-packages/mmengine/model/weight_init.py", line 497, in call
state_dict = _load_checkpoint_with_prefix(
File "/home/ls/anaconda3/envs/mmselfsupv1/lib/python3.8/site-packages/mmengine/runner/checkpoint.py", line 514, in _load_checkpoint_with_prefix
assert state_dict, f'{prefix} is not in the pretrained model'
AssertionError: backbone. is not in the pretrained model
Killing subprocess 19384
"

The config I used had been used for training the pretrained SimMIM model, and it could run successfully, I don't know why MAE couldn't work.

May I get some advice about how to fix it? Thanks.

[fangyixiao18]

After downloading the pre-train model, did you do anything else to the model? Could you also provide your download link?

[fangyixiao18]

Please try this pre-train model: https://download.openmmlab.com/mmselfsup/1.x/mae/mae_vit-base-p16_8xb512-fp16-coslr-1600e_in1k/mae_vit-base-p16_8xb512-fp16-coslr-1600e_in1k_20220825-f7569ca2.pth

The model you mentioned might have some problem, I will check it again

[Yoooss]

Ok, I'll try this model. Also, using the BYOL pretrained model got the same error.

And there is another question I wanna ask.
When I used the downloaded SimSiam pretrained model to run the downstream object detection task using mmselfsup V0.9, the result is about mAP 0.18.
However, when I runned the same config with the same pretrained SImSiam model using mmselfsup V1.0, the result is about mAP 0.09.
The same situation also occurred when I use the downloaded SimMIM pretrained model in mmselfsup V1.0, the result is about mAP 0.0936.
Both model perform bad on the downstream task , which is different from what their articles show. May I ask what caused this different result?

Thanks for your help.

[fangyixiao18]

Ok, I'll try this model. Also, using the BYOL pretrained model got the same error.

I checked BYOL pretrained model, it works. Could you provide your command and config files?

As for the other question, could you also provide your command and config files?

[Yoooss]

I run the downstream object detection task using the command as below:
"
bash tools/benchmarks/mmdetection/mim_dist_train_c4.sh configs/benchmarks/mmdetection/voc0712/faster_rcnn_r50_c4_mstrain_24k_voc0712ls.py /home/ls/mmselfsup/checkpoints/simsiam_resnet50_8xb32-coslr-200e_in1k_20220825-efe91299.pth 1
"

The config I use is "faster_rcnn_r50_c4_mstrain_24k_voc0712.py" only changed the iters number.

And the result is only about 0.09 mAP in mmselfsup v1.0 . However, the result is about 0.18 mAP in mmselfsup v0.9 in my previous try.

Thanks for your help. And I will download and try BYOL pretrained model again.

[fangyixiao18]

The number of GPU (you use 1 GPU in your command) will influence the results, because it changes the batchsize.
The first thing is the learning rate, you need to linearly scale the learning rate to 1/8.
The second thing is the about BN, as we set batch_size=2 per gpu, this will have worse results. You could check some experiments in this paper 'Group Normalization'

[tonysy]

@Yoooss Hi, have you solved this issue?