"unexpected key in source state_dict:" "missing keys in source state_dict:" in training latest version of Swin Transdormer

[BenoitKAO]

Checklist

1. I have searched related issues but cannot get the expected help.
2. The bug has not been fixed in the latest version.

Describe the bug
unexpected key in source state_dict: norm.weight, norm.bias, stages.0.blocks.1.attn_mask, stages.1.blocks.1.attn_mask, stages.2.blocks.1.attn_mask, stages.2.blocks.3.attn_mask, stages.2.blocks.5.attn_mask, stages.2.blocks.7.attn_mask, stages.2.blocks.9.attn_mask, stages.2.blocks.11.attn_mask, stages.2.blocks.13.attn_mask, stages.2.blocks.15.attn_mask, stages.2.blocks.17.attn_mask

missing keys in source state_dict: norm0.weight, norm0.bias, norm1.weight, norm1.bias, norm2.weight, norm2.bias, norm3.weight, norm3.bias

Reproduction

1. What command or script did you run?

checkpoint_file = './configs/swin/swin_large_patch4_window7_224_22k_20220412-aeecf2aa.pth'

model = dict(
    backbone=dict(
        init_cfg=dict(type='Pretrained', checkpoint=checkpoint_file),
        pretrain_img_size=224,
        embed_dims=192,
        depths=[2, 2, 18, 2],
        num_heads=[6, 12, 24, 48],
        window_size=7),
    decode_head=dict(in_channels=[192, 384, 768, 1536], num_classes=4),
    auxiliary_head=dict(in_channels=768, num_classes=4))

2. Did you make any modifications on the code or config?
   No,

Did you understand what you have modified?
Yes.

3. What dataset did you use?
   custom dataset

Environment

1. Please run python mmseg/utils/collect_env.py to collect necessary environment information and paste it here.
   It is not an environment issue.

2. You may add addition that may be helpful for locating the problem, such as

   • How you installed PyTorch [e.g., pip, conda, source]
   • Other environment variables that may be related (such as $PATH, $LD_LIBRARY_PATH, $PYTHONPATH, etc.)

swin_large_patch4_window7_224_22k_20220412-aeecf2aa.pth'

Error traceback

2023-01-24 21:52:00,656 - mmseg - INFO - Set random seed to 1239451960, deterministic: False
/githubs/mmsegmentation/mmseg/models/losses/cross_entropy_loss.py:235: UserWarning: Default ``avg_non_ignore`` is False, if you would like to ignore the certain label and average loss over non-ignore labels, which is the same with PyTorch official cross_entropy, set ``avg_non_ignore=True``.
  warnings.warn(
2023-01-24 21:52:02,174 - mmseg - INFO - load checkpoint from local path: ./configs/swin/swin_large_patch4_window7_224_22k_20220412-aeecf2aa.pth
2023-01-24 21:52:02,643 - mmseg - WARNING - The model and loaded state dict do not match exactly

unexpected key in source state_dict: norm.weight, norm.bias, stages.0.blocks.1.attn_mask, stages.1.blocks.1.attn_mask, stages.2.blocks.1.attn_mask, stages.2.blocks.3.attn_mask, stages.2.blocks.5.attn_mask, stages.2.blocks.7.attn_mask, stages.2.blocks.9.attn_mask, stages.2.blocks.11.attn_mask, stages.2.blocks.13.attn_mask, stages.2.blocks.15.attn_mask, stages.2.blocks.17.attn_mask

missing keys in source state_dict: norm0.weight, norm0.bias, norm1.weight, norm1.bias, norm2.weight, norm2.bias, norm3.weight, norm3.bias

2023-01-24 21:52:02,700 - mmcv - INFO - initialize UPerHead with init_cfg {'type': 'Normal', 'std': 0.01, 'override': {'name': 'conv_seg'}}
2023-01-24 21:52:02,833 - mmcv - INFO - initialize FCNHead with init_cfg {'type': 'Normal', 'std': 0.01, 'override': {'name': 'conv_seg'}}
2023-01-24 21:52:02,834 - mmcv - INFO -

Any help is appreciated.

[AI-Tianlong]

Have you run the /tools/model_converters/swin2mmseg.py to convert swin transformer offical model weights to MMSegmentation style?
More details may be help you https://github.com/open-mmlab/mmsegmentation/tree/dev-1.x/configs/swin#usage

[BenoitKAO]

Dear @AI-Tianlong,

Thank you for getting back to me.

Yes, I have tried to convert swin transformer official model weights to MMSegmentation style.

However, they all have shown unexpected keys in source state_dict and missing keys in source state_dict.

Could you provide the successful command lines for these conversions and the successful configs for running these models in detail?

I appreciate your help and thank you again.

[MeowZheng]

we have provided the pre-trained backbone checkpoint, and you can use it directly without running any extra command

mmsegmentation/configs/swin/upernet_swin_base_patch4_window12_512x512_160k_ade20k_pretrain_384x384_1K.py

Line 5 in 5d49918

checkpoint_file = 'https://download.openmmlab.com/mmsegmentation/v0.5/pretrain/swin/swin_base_patch4_window12_384_20220317-55b0104a.pth' # noqa

[BenoitKAO]

Thank you for getting back to me so quickly.
However, it still has "unexpected key in source state_dict:" "missing keys in source state_dict:"

checkpoint_file = './configs/swin/swin_base_patch4_window12_384_20220317-55b0104a.pth'
work_dir = './work_dirs/A'
gpu_ids = [0]
auto_resume = False

2023-01-26 14:03:40,457 - mmseg - INFO - Set random seed to 1138299252, deterministic: False
/mmsegmentation/mmseg/models/losses/cross_entropy_loss.py:235: UserWarning: Default ``avg_non_ignore`` is False, if you would like to ignore the certain label and average loss over non-ignore labels, which is the same with PyTorch official cross_entropy, set ``avg_non_ignore=True``.
  warnings.warn(
2023-01-26 14:03:41,275 - mmseg - INFO - load checkpoint from local path: ./configs/swin/swin_base_patch4_window12_384_20220317-55b0104a.pth
2023-01-26 14:03:41,490 - mmseg - WARNING - The model and loaded state dict do not match exactly

unexpected key in source state_dict: norm.weight, norm.bias, stages.0.blocks.1.attn_mask, stages.1.blocks.1.attn_mask, stages.2.blocks.1.attn_mask, stages.2.blocks.3.attn_mask, stages.2.blocks.5.attn_mask, stages.2.blocks.7.attn_mask, stages.2.blocks.9.attn_mask, stages.2.blocks.11.attn_mask, stages.2.blocks.13.attn_mask, stages.2.blocks.15.attn_mask, stages.2.blocks.17.attn_mask

missing keys in source state_dict: norm0.weight, norm0.bias, norm1.weight, norm1.bias, norm2.weight, norm2.bias, norm3.weight, norm3.bias

How to train from scratch without using the checkpoint_file?

[AI-Tianlong]

I think that you changed 'num_classes=4' at your configs. However original model weights trained on other dataset, which num_classes is not 4.
This '4' decide the final feature map channels. They're unmatched, may be appear 'missing keys'.
You should find out which layers weights unmatched, if the final layers, I think that's why I said it.
If this is the case, which I think is normal, you can ignore it and continue training.
Have try another configs file with existing datasets like cityscapes?

[BenoitKAO]

I appreciate your instructions and tutorials.
However, I would like to know how to train the Swin Transformer model with custom datasets from scratch without using any checkpoint_file.
I would appreciate any help.

[AI-Tianlong]

set init_cfg=None, at configs, it will train model from scratch. And it works well to me.

You can try my configs to test your envirenment.

swin-base-patch4-window7-in1k-pre_upernet_8xb2-160k_cityscapes-512x1024.py

_base_ = [
    '../_base_/models/upernet_swin.py', '../_base_/datasets/cityscapes.py',
    '../_base_/default_runtime.py', '../_base_/schedules/schedule_160k.py'
]
crop_size = (512, 1024)
data_preprocessor = dict(size=crop_size)
# checkpoint_file = 'https://download.openmmlab.com/mmsegmentation/v0.5/pretrain/swin/swin_tiny_patch4_window7_224_20220317-1cdeb081.pth'  # noqa
model = dict(
    data_preprocessor=data_preprocessor,
    backbone=dict(
        init_cfg=None,
        embed_dims=128,
        depths=[2, 2, 18, 2],
        num_heads=[4, 8, 16, 32],
        window_size=7,
        use_abs_pos_embed=False,
        drop_path_rate=0.3,
        patch_norm=True),
    decode_head=dict(in_channels=[128, 256, 512, 1024], num_classes=19),
    auxiliary_head=dict(in_channels=512, num_classes=19))


# AdamW optimizer, no weight decay for position embedding & layer norm
# in backbone
optim_wrapper = dict(
    _delete_=True,
    type='OptimWrapper',
    optimizer=dict(
        type='AdamW', lr=0.00006, betas=(0.9, 0.999), weight_decay=0.01),
    paramwise_cfg=dict(
        custom_keys={
            'absolute_pos_embed': dict(decay_mult=0.),
            'relative_position_bias_table': dict(decay_mult=0.),
            'norm': dict(decay_mult=0.)
        }))

param_scheduler = [
    dict(
        type='LinearLR', start_factor=1e-6, by_epoch=False, begin=0, end=1500),
    dict(
        type='PolyLR',
        eta_min=0.0,
        power=1.0,
        begin=1500,
        end=160000,
        by_epoch=False,
    )
]

# By default, models are trained on 8 GPUs with 2 images per GPU
train_dataloader = dict(batch_size=2)
val_dataloader = dict(batch_size=1)
test_dataloader = val_dataloader

more details you could view mmsegmentation/mmseg/models/backbones/swin.py

[BenoitKAO]

Dear @MeowZheng and @AI-Tianlong,

I appreciate your tutorials and your explanations in detail very much.
I have tried your configs and am waiting to see the results.

Nevertheless, could you provide the documentation and tutorials that clearly illustrate in detail how to modify the deep learning model on OpenMM-series (OpenMMLab), such as MMCV, MMSegmentation, MMDetect, MMEngine, MMEval, ..., and so on?

Because I would like to build my deep learning models using your marvelous works, however, I could only find these DOCs:
mmseg-models
docs zh_cn tutorials
zh_CN latest pdf
en latest pdf

I would appreciate any help.

Thank you very much.

[xiexinch]

Checklist

1. I have searched related issues but cannot get the expected help.
2. The bug has not been fixed in the latest version.

Describe the bug unexpected key in source state_dict: norm.weight, norm.bias, stages.0.blocks.1.attn_mask, stages.1.blocks.1.attn_mask, stages.2.blocks.1.attn_mask, stages.2.blocks.3.attn_mask, stages.2.blocks.5.attn_mask, stages.2.blocks.7.attn_mask, stages.2.blocks.9.attn_mask, stages.2.blocks.11.attn_mask, stages.2.blocks.13.attn_mask, stages.2.blocks.15.attn_mask, stages.2.blocks.17.attn_mask

missing keys in source state_dict: norm0.weight, norm0.bias, norm1.weight, norm1.bias, norm2.weight, norm2.bias, norm3.weight, norm3.bias

Reproduction

1. What command or script did you run?

checkpoint_file = './configs/swin/swin_large_patch4_window7_224_22k_20220412-aeecf2aa.pth'

model = dict(
    backbone=dict(
        init_cfg=dict(type='Pretrained', checkpoint=checkpoint_file),
        pretrain_img_size=224,
        embed_dims=192,
        depths=[2, 2, 18, 2],
        num_heads=[6, 12, 24, 48],
        window_size=7),
    decode_head=dict(in_channels=[192, 384, 768, 1536], num_classes=4),
    auxiliary_head=dict(in_channels=768, num_classes=4))

2. Did you make any modifications on the code or config?
   No,

Did you understand what you have modified? Yes.

3. What dataset did you use?
   custom dataset

Environment

1. Please run python mmseg/utils/collect_env.py to collect necessary environment information and paste it here.
   It is not an environment issue.

2. You may add addition that may be helpful for locating the problem, such as

   • How you installed PyTorch [e.g., pip, conda, source]
   • Other environment variables that may be related (such as $PATH, $LD_LIBRARY_PATH, $PYTHONPATH, etc.)

swin_large_patch4_window7_224_22k_20220412-aeecf2aa.pth'

Error traceback

2023-01-24 21:52:00,656 - mmseg - INFO - Set random seed to 1239451960, deterministic: False
/githubs/mmsegmentation/mmseg/models/losses/cross_entropy_loss.py:235: UserWarning: Default ``avg_non_ignore`` is False, if you would like to ignore the certain label and average loss over non-ignore labels, which is the same with PyTorch official cross_entropy, set ``avg_non_ignore=True``.
  warnings.warn(
2023-01-24 21:52:02,174 - mmseg - INFO - load checkpoint from local path: ./configs/swin/swin_large_patch4_window7_224_22k_20220412-aeecf2aa.pth
2023-01-24 21:52:02,643 - mmseg - WARNING - The model and loaded state dict do not match exactly

unexpected key in source state_dict: norm.weight, norm.bias, stages.0.blocks.1.attn_mask, stages.1.blocks.1.attn_mask, stages.2.blocks.1.attn_mask, stages.2.blocks.3.attn_mask, stages.2.blocks.5.attn_mask, stages.2.blocks.7.attn_mask, stages.2.blocks.9.attn_mask, stages.2.blocks.11.attn_mask, stages.2.blocks.13.attn_mask, stages.2.blocks.15.attn_mask, stages.2.blocks.17.attn_mask

missing keys in source state_dict: norm0.weight, norm0.bias, norm1.weight, norm1.bias, norm2.weight, norm2.bias, norm3.weight, norm3.bias

2023-01-24 21:52:02,700 - mmcv - INFO - initialize UPerHead with init_cfg {'type': 'Normal', 'std': 0.01, 'override': {'name': 'conv_seg'}}
2023-01-24 21:52:02,833 - mmcv - INFO - initialize FCNHead with init_cfg {'type': 'Normal', 'std': 0.01, 'override': {'name': 'conv_seg'}}
2023-01-24 21:52:02,834 - mmcv - INFO -

Any help is appreciated.

Hi @BenoitKAO,
Did you use the latest version of MMSeg? We've solved it before. #2304

Or if the latest version does not work well, please no hesitate to tell us.

[xiexinch]

Closing the issue, as there is no activity for a while.
We hope your issue has been resolved.
If not, please feel free to open a new one.