fix beit model converter

[ZhaoYi1222]

Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily get feedback. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.

Motivation

The script in tools/model_converters/beit2mmseg.py doesn't match the key of models between the official repo and MMSegmentation style (e.g. backbone.*).
The training log shows lots of keys in missing keys in source state_dict.
Trainging result after 1500 iteration:
| aAcc | mIoU | mAcc |
| 39.95 | 1.97 | 3.46 |

Modification

I modify tools/model_converters/beit2mmseg.py.
Now the warnings of missing keys related to 'backbone' disappear.
Trainging result after 1500 iteration:
| aAcc | mIoU | mAcc |
| 60.4 | 6.21 | 9.4 |

BC-breaking (Optional)

Use cases (Optional)

Checklist

[linfangjian01]

In the mmseg implementation, the pre-trained model is loaded in mmseg/models/backbones/beit.py, so the 'backbone' prefix is not needed. Are you using our tools/model_converters/beit2mmseg.py script in your own codebase? Please see the other model_converters scripts and refer to the implementation in beit.py. Also, for

mmsegmentation/mmseg/models/backbones/beit.py

Line 507 in 46326f6

self.load_state_dict(state_dict, False)

, you can set it to True, so that you can check which key values are missing. The missing keys of the current model are
. These pre-trained keys are not needed in the downstream task.

[MeowZheng]

@ ZhaoYi1222 What do you think about this？

In the mmseg implementation, the pre-trained model is loaded in mmseg/models/backbones/beit.py, so the 'backbone' prefix is not needed. Are you using our tools/model_converters/beit2mmseg.py script in your own codebase? Please see the other model_converters scripts and refer to the implementation in beit.py. Also, for

mmsegmentation/mmseg/models/backbones/beit.py

Line 507 in 46326f6

self.load_state_dict(state_dict, False)

, you can set it to True, so that you can check which key values are missing. The missing keys of the current model are
. These pre-trained keys are not needed in the downstream task.

[ZhaoYi1222]

Sorry for the delay. I try to reproduce the result shown in

configs/beit/README.md

So I download model from

https://conversationhub.blob.core.windows.net/beit-share-public/beit/beit_large_patch16_224_pt22k_ft22k.pth

Then I use

tools/model_converters/beit2mmseg.py

to convert the model.

When I train the model using --load-from, the log shows
unexpected key in source state_dict: cls_token, patch_embed.projection.weight, patch_embed.proj.weight, patch_embed.projection.bias, patch_embed.proj.bias, layers.0.gamma_1 and
missing keys in source state_dict: backbone.cls_token, backbone.patch_embed.projection.weight, backbone.patch_embed.projection.bias, backbone.layers.0.gamma_1

Maybe I make some mistakes in my training scripts, I think some guidance about training can be added to README.md?

Are you using our tools/model_converters/beit2mmseg.py script in your own codebase?

I run training based on mmsegmentation and try to follow the guidance.

Also, in

tools/model_converters/beit2mmseg.py

I think if k.startswith('patch_embed'), key would be add twice in line 17 and 29.

Thanks again for your generous help. @linfangjian01 @MeowZheng

[linfangjian01]

Thx. I think 'patch_embed' can be modified with reference to https://github.com/ZhaoYi1222/mmsegmentation/blob/0ca30ef84579ce4c62aa3aac1ab3769672b389bc/tools/model_converters/beit2mmseg.py#L23-L24 and other parts can be kept. In addition the training can be referred to https://github.com/open-mmlab/mmsegmentation/blob/master/docs/en/train.md. The pretrain path can be modified in

mmsegmentation/configs/beit/upernet_beit-base_8x2_640x640_160k_ade20k.py

Line 7 in 2dede04

pretrained='pretrain/beit_base_patch16_224_pt22k_ft22k.pth',

.

[linfangjian01]

or use ln -s pretrain_local_path pretrain to create a soft connection in the same level directory as config, mmseg.

[linfangjian01]

suggest change:

[ZhaoYi1222]

I have modified the code according to your suggestion. Thanks. @linfangjian01
I'll check my training scripts later to match parameters.

[linfangjian01]

Thanks.