Cannot load the pretrained vit model offline using timm.create_model

[dongdongtong]

I'm trying to load pretrained ViTs using timm and huggingface pretrained models offline. I download the pretrained weights from huggingface mannualy. However, when loading pretrained vit_tiny_patch16_224 and vit_small_patch16_224 weights via timm.create_model, I get the following error:
KeyError: 'embedding/kernel is not a file in the archive'.
It seems that the _load_weights function of vision_transformer.py is troublesome.

To Reproduce
Steps to reproduce the behavior:

1. Download the pretrained weights (pytorch.bin) from huggingface and put it into the proper location.
2. using timm create_model function to load the pretrained weights:
   timm.create_model( "vit_tiny_patch16_224", pretrained=True, pretrained_cfg_overlay=dict( file="/home/your_path/.cache/huggingface/hub/models--timm--vit_tiny_patch16_224.augreg_in21k_ft_in1k/pytorch_model.bin") )

Expected behavior
load the pretrained weights successfully.

Screenshots

Desktop (please complete the following information):

• OS: [Ubuntu 22.04]
• pytorch: 1.11.0
• timm: 0.9.12

[fffffgggg54]

I am able to load the model offline with timm.create_model("vit_tiny_patch16_224", pretrained=True, pretrained_cfg_overlay=dict(file="./pytorch_model.bin", custom_load=False)).

repro: https://gist.github.com/fffffgggg54/115e5721289fcc8f05200c0609b93dd5

I believe the issue has to do with whichever base vit_tiny cfg gets used having custom_load=True (all of them do) for the linked google numpy weight, which results in a model.load_pretrained call which vit uses for numpy weights.

[dongdongtong]

Thx a lot, it works and I have been waiting util now.
And is there any difference in loaded pretrained weights between custom_load=True and custom_load=False?

[fffffgggg54]

For ViT, custom_load=True causes the builder to treat the weight as a numpy weight whereas the huggingface weight is (I believe) a pytorch weight. The weight on huggingface was remapped from the numpy weight from google to a pytorch weight, so they are identical once loaded.

You can load the original numpy weights from the link as well if you want: model = timm.create_model("vit_tiny_patch16_224", pretrained=True, pretrained_cfg_overlay=dict(file="./Ti_16-i21k-300ep-lr_0.001-aug_none-wd_0.03-do_0.0-sd_0.0--imagenet2012-steps_20k-lr_0.03-res_224.npz"))

[dongdongtong]

thx a lot, I will close this issue for your kind patience.

[fffffgggg54]

For reference, timm.create_model("vit_tiny_patch16_224", checkpoint_path="/path/to/pytorch_model.bin") also works.

[watertianyi]

@fffffgggg54 @dongdongtong
model_convnextv2 = timm.create_model('convnextv2_tiny.fcmae',checkpoint_path='../FCMA/convnextv2_tiny_1k_224_fcmae.pt')

I use the official weight file to load locally, and the following error is reported. What's going on?
Traceback (most recent call last):
File "/media/CV/test_timm.py", line 46, in
model_convnextv2 = timm.create_model('convnextv2_tiny.fcmae',checkpoint_path='../nanchang/FCMA/convnextv2_tiny_1k_224_fcmae.pt')
File "/home/anaconda3/envs/image_search/lib/python3.9/site-packages/timm/models/_factory.py", line 125, in create_model
load_checkpoint(model, checkpoint_path)
File "/home/anaconda3/envs/image_search/lib/python3.9/site-packages/timm/models/_helpers.py", line 101, in load_checkpoint
incompatible_keys = model.load_state_dict(state_dict, strict=strict)
File "/home/anaconda3/envs/image_search/lib/python3.9/site-packages/torch/nn/modules/module.py", line 2215, in load_state_dict
raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for ConvNeXt:
Missing key(s) in state_dict: "stem.0.weight", "stem.0.bias", "stem.1.weight", "stem.1.bias", "stages.0.blocks.0.conv_dw.weight", "stages.0.blocks.0.conv_dw.bias", "stages.0.blocks.0.norm.weight", "stages.0.blocks.0.norm.bias", "stages.0.blocks.0.mlp.fc1.weight", "stages.0.blocks.0.mlp.fc1.bias", "stages.0.blocks.0.mlp.grn.weight", "stages.0.blocks.0.mlp.grn.bias", "stages.0.blocks.0.mlp.fc2.weight", "stages.0.blocks.0.mlp.fc2.bias", "stages.0.blocks.1.conv_dw.weight", "stages.0.blocks.1.conv_dw.bias", "stages.0.blocks.1.norm.weight", "stages.0.blocks.1.norm.bias", "stages.0.blocks.1.mlp.fc1.weight", "stages.0.blocks.1.mlp.fc1.bias", "stages.0.blocks.1.mlp.grn.weight", "stages.0.blocks.1.mlp.grn.bias", "stages.0.blocks.1.mlp.fc2.weight", "stages.0.blocks.1.mlp.fc2.bias", "stages.0.blocks.2.conv_dw.weight", "stages.0.blocks.2.conv_dw.bias", "stages.0.blocks.2.norm.weight", "stages.0.blocks.2.norm.bias", "stages.0.blocks.2.mlp.fc1.weight", "stages.0.blocks.2.mlp.fc1.bias", "stages.0.blocks.2.mlp.grn.weight", "stages.0.blocks.2.mlp.grn.bias", "stages.0.blocks.2.mlp.fc2.weight", "stages.0.blocks.2.mlp.fc2.bias", "stages.1.downsample.0.weight", "stages.1.downsample.0.bias", "stages.1.downsample.1.weight", "stages.1.downsample.1.bias", "stages.1.blocks.0.conv_dw.weight", "stages.1.blocks.0.conv_dw.bias", "stages.1.blocks.0.norm.weight", "stages.1.blocks.0.norm.bias", "stages.1.blocks.0.mlp.fc1.weight", "stages.1.blocks.0.mlp.fc1.bias", "stages.1.blocks.0.mlp.grn.weight", "stages.1.blocks.0.mlp.grn.bias", "stages.1.blocks.0.mlp.fc2.weight", "stages.1.blocks.0.mlp.fc2.bias", "stages.1.blocks.1.conv_dw.weight", "stages.1.blocks.1.conv_dw.bias", "stages.1.blocks.1.norm.weight", "stages.1.blocks.1.norm.bias", "stages.1.blocks.1.mlp.fc1.weight", "stages.1.blocks.1.mlp.fc1.bias", "stages.1.blocks.1.mlp.grn.weight", "stages.1.blocks.1.mlp.grn.bias", "stages.1.blocks.1.mlp.fc2.weight", "stages.1.blocks.1.mlp.fc2.bias", "stages.1.blocks.2.conv_dw.weight", "stages.1.blocks.2.conv_dw.bias", "stages.1.blocks.2.norm.weight", "stages.1.blocks.2.norm.bias", "stages.1.blocks.2.mlp.fc1.weight", "stages.1.blocks.2.mlp.fc1.bias", "stages.1.blocks.2.mlp.grn.weight", "stages.1.blocks.2.mlp.grn.bias", "stages.1.blocks.2.mlp.fc2.weight", "stages.1.blocks.2.mlp.fc2.bias", "stages.2.downsample.0.weight", "stages.2.downsample.0.bias", "stages.2.downsample.1.weight", "stages.2.downsample.1.bias", "stages.2.blocks.0.conv_dw.weight", "stages.2.blocks.0.conv_dw.bias", "stages.2.blocks.0.norm.weight", "stages.2.blocks.0.norm.bias", "stages.2.blocks.0.mlp.fc1.weight", "stages.2.blocks.0.mlp.fc1.bias", "stages.2.blocks.0.mlp.grn.weight", "stages.2.blocks.0.mlp.grn.bias", "stages.2.blocks.0.mlp.fc2.weight", "stages.2.blocks.0.mlp.fc2.bias", "stages.2.blocks.1.conv_dw.weight", "stages.2.blocks.1.conv_dw.bias", "stages.2.blocks.1.norm.weight", "stages.2.blocks.1.norm.bias", "stages.2.blocks.1.mlp.fc1.weight", "stages.2.blocks.1.mlp.fc1.bias", "stages.2.blocks.1.mlp.grn.weight", "stages.2.blocks.1.mlp.grn.bias", "stages.2.blocks.1.mlp.fc2.weight", "stages.2.blocks.1.mlp.fc2.bias", "stages.2.blocks.2.conv_dw.weight", "stages.2.blocks.2.conv_dw.bias", "stages.2.blocks.2.norm.weight", "stages.2.blocks.2.norm.bias", "stages.2.blocks.2.mlp.fc1.weight", "stages.2.blocks.2.mlp.fc1.bias", "stages.2.blocks.2.mlp.grn.weight", "stages.2.blocks.2.mlp.grn.bias", "stages.2.blocks.2.mlp.fc2.weight", "stages.2.blocks.2.mlp.fc2.bias", "stages.2.blocks.3.conv_dw.weight", "stages.2.blocks.3.conv_dw.bias", "stages.2.blocks.3.norm.weight", "stages.2.blocks.3.norm.bias", "stages.2.blocks.3.mlp.fc1.weight", "stages.2.blocks.3.mlp.fc1.bias", "stages.2.blocks.3.mlp.grn.weight", "stages.2.blocks.3.mlp.grn.bias", "stages.2.blocks.3.mlp.fc2.weight", "stages.2.blocks.3.mlp.fc2.bias", "stages.2.blocks.4.conv_dw.weight", "stages.2.blocks.4.conv_dw.bias", "stages.2.blocks.4.norm.weight", "stages.2.blocks.4.norm.bias", "stages.2.blocks.4.mlp.fc1.weight", "stages.2.blocks.4.mlp.fc1.bias", "stages.2.blocks.4.mlp.grn.weight", "stages.2.blocks.4.mlp.grn.bias", "stages.2.blocks.4.mlp.fc2.weight", "stages.2.blocks.4.mlp.fc2.bias", "stages.2.blocks.5.conv_dw.weight", "stages.2.blocks.5.conv_dw.bias", "stages.2.blocks.5.norm.weight", "stages.2.blocks.5.norm.bias", "stages.2.blocks.5.mlp.fc1.weight", "stages.2.blocks.5.mlp.fc1.bias", "stages.2.blocks.5.mlp.grn.weight", "stages.2.blocks.5.mlp.grn.bias", "stages.2.blocks.5.mlp.fc2.weight", "stages.2.blocks.5.mlp.fc2.bias", "stages.2.blocks.6.conv_dw.weight", "stages.2.blocks.6.conv_dw.bias", "stages.2.blocks.6.norm.weight", "stages.2.blocks.6.norm.bias", "stages.2.blocks.6.mlp.fc1.weight", "stages.2.blocks.6.mlp.fc1.bias", "stages.2.blocks.6.mlp.grn.weight", "stages.2.blocks.6.mlp.grn.bias", "stages.2.blocks.6.mlp.fc2.weight", "stages.2.blocks.6.mlp.fc2.bias", "stages.2.blocks.7.conv_dw.weight", "stages.2.blocks.7.conv_dw.bias", "stages.2.blocks.7.norm.weight", "stages.2.blocks.7.norm.bias", "stages.2.blocks.7.mlp.fc1.weight", "stages.2.blocks.7.mlp.fc1.bias", "stages.2.blocks.7.mlp.grn.weight", "stages.2.blocks.7.mlp.grn.bias", "stages.2.blocks.7.mlp.fc2.weight", "stages.2.blocks.7.mlp.fc2.bias", "stages.2.blocks.8.conv_dw.weight", "stages.2.blocks.8.conv_dw.bias", "stages.2.blocks.8.norm.weight", "stages.2.blocks.8.norm.bias", "stages.2.blocks.8.mlp.fc1.weight", "stages.2.blocks.8.mlp.fc1.bias", "stages.2.blocks.8.mlp.grn.weight", "stages.2.blocks.8.mlp.grn.bias", "stages.2.blocks.8.mlp.fc2.weight", "stages.2.blocks.8.mlp.fc2.bias", "stages.3.downsample.0.weight", "stages.3.downsample.0.bias", "stages.3.downsample.1.weight", "stages.3.downsample.1.bias", "stages.3.blocks.0.conv_dw.weight", "stages.3.blocks.0.conv_dw.bias", "stages.3.blocks.0.norm.weight", "stages.3.blocks.0.norm.bias", "stages.3.blocks.0.mlp.fc1.weight", "stages.3.blocks.0.mlp.fc1.bias", "stages.3.blocks.0.mlp.grn.weight", "stages.3.blocks.0.mlp.grn.bias", "stages.3.blocks.0.mlp.fc2.weight", "stages.3.blocks.0.mlp.fc2.bias", "stages.3.blocks.1.conv_dw.weight", "stages.3.blocks.1.conv_dw.bias", "stages.3.blocks.1.norm.weight", "stages.3.blocks.1.norm.bias", "stages.3.blocks.1.mlp.fc1.weight", "stages.3.blocks.1.mlp.fc1.bias", "stages.3.blocks.1.mlp.grn.weight", "stages.3.blocks.1.mlp.grn.bias", "stages.3.blocks.1.mlp.fc2.weight", "stages.3.blocks.1.mlp.fc2.bias", "stages.3.blocks.2.conv_dw.weight", "stages.3.blocks.2.conv_dw.bias", "stages.3.blocks.2.norm.weight", "stages.3.blocks.2.norm.bias", "stages.3.blocks.2.mlp.fc1.weight", "stages.3.blocks.2.mlp.fc1.bias", "stages.3.blocks.2.mlp.grn.weight", "stages.3.blocks.2.mlp.grn.bias", "stages.3.blocks.2.mlp.fc2.weight", "stages.3.blocks.2.mlp.fc2.bias", "head.norm.weight", "head.norm.bias".
Unexpected key(s) in state_dict: "downsample_layers.0.0.bias", "downsample_layers.0.0.weight", "downsample_layers.0.1.bias", "downsample_layers.0.1.weight", "downsample_layers.1.0.bias", "downsample_layers.1.0.weight", "downsample_layers.1.1.bias", "downsample_layers.1.1.weight", "downsample_layers.2.0.bias", "downsample_layers.2.0.weight", "downsample_layers.2.1.bias", "downsample_layers.2.1.weight", "downsample_layers.3.0.bias", "downsample_layers.3.0.weight", "downsample_layers.3.1.bias", "downsample_layers.3.1.weight", "stages.0.0.grn.beta", "stages.0.0.grn.gamma", "stages.0.0.dwconv.bias", "stages.0.0.dwconv.weight", "stages.0.0.norm.bias", "stages.0.0.norm.weight", "stages.0.0.pwconv1.bias", "stages.0.0.pwconv1.weight", "stages.0.0.pwconv2.bias", "stages.0.0.pwconv2.weight", "stages.0.1.grn.beta", "stages.0.1.grn.gamma", "stages.0.1.dwconv.bias", "stages.0.1.dwconv.weight", "stages.0.1.norm.bias", "stages.0.1.norm.weight", "stages.0.1.pwconv1.bias", "stages.0.1.pwconv1.weight", "stages.0.1.pwconv2.bias", "stages.0.1.pwconv2.weight", "stages.0.2.grn.beta", "stages.0.2.grn.gamma", "stages.0.2.dwconv.bias", "stages.0.2.dwconv.weight", "stages.0.2.norm.bias", "stages.0.2.norm.weight", "stages.0.2.pwconv1.bias", "stages.0.2.pwconv1.weight", "stages.0.2.pwconv2.bias", "stages.0.2.pwconv2.weight", "stages.1.0.grn.beta", "stages.1.0.grn.gamma", "stages.1.0.dwconv.bias", "stages.1.0.dwconv.weight", "stages.1.0.norm.bias", "stages.1.0.norm.weight", "stages.1.0.pwconv1.bias", "stages.1.0.pwconv1.weight", "stages.1.0.pwconv2.bias", "stages.1.0.pwconv2.weight", "stages.1.1.grn.beta", "stages.1.1.grn.gamma", "stages.1.1.dwconv.bias", "stages.1.1.dwconv.weight", "stages.1.1.norm.bias", "stages.1.1.norm.weight", "stages.1.1.pwconv1.bias", "stages.1.1.pwconv1.weight", "stages.1.1.pwconv2.bias", "stages.1.1.pwconv2.weight", "stages.1.2.grn.beta", "stages.1.2.grn.gamma", "stages.1.2.dwconv.bias", "stages.1.2.dwconv.weight", "stages.1.2.norm.bias", "stages.1.2.norm.weight", "stages.1.2.pwconv1.bias", "stages.1.2.pwconv1.weight", "stages.1.2.pwconv2.bias", "stages.1.2.pwconv2.weight", "stages.2.0.grn.beta", "stages.2.0.grn.gamma", "stages.2.0.dwconv.bias", "stages.2.0.dwconv.weight", "stages.2.0.norm.bias", "stages.2.0.norm.weight", "stages.2.0.pwconv1.bias", "stages.2.0.pwconv1.weight", "stages.2.0.pwconv2.bias", "stages.2.0.pwconv2.weight", "stages.2.1.grn.beta", "stages.2.1.grn.gamma", "stages.2.1.dwconv.bias", "stages.2.1.dwconv.weight", "stages.2.1.norm.bias", "stages.2.1.norm.weight", "stages.2.1.pwconv1.bias", "stages.2.1.pwconv1.weight", "stages.2.1.pwconv2.bias", "stages.2.1.pwconv2.weight", "stages.2.2.grn.beta", "stages.2.2.grn.gamma", "stages.2.2.dwconv.bias", "stages.2.2.dwconv.weight", "stages.2.2.norm.bias", "stages.2.2.norm.weight", "stages.2.2.pwconv1.bias", "stages.2.2.pwconv1.weight", "stages.2.2.pwconv2.bias", "stages.2.2.pwconv2.weight", "stages.2.3.grn.beta", "stages.2.3.grn.gamma", "stages.2.3.dwconv.bias", "stages.2.3.dwconv.weight", "stages.2.3.norm.bias", "stages.2.3.norm.weight", "stages.2.3.pwconv1.bias", "stages.2.3.pwconv1.weight", "stages.2.3.pwconv2.bias", "stages.2.3.pwconv2.weight", "stages.2.4.grn.beta", "stages.2.4.grn.gamma", "stages.2.4.dwconv.bias", "stages.2.4.dwconv.weight", "stages.2.4.norm.bias", "stages.2.4.norm.weight", "stages.2.4.pwconv1.bias", "stages.2.4.pwconv1.weight", "stages.2.4.pwconv2.bias", "stages.2.4.pwconv2.weight", "stages.2.5.grn.beta", "stages.2.5.grn.gamma", "stages.2.5.dwconv.bias", "stages.2.5.dwconv.weight", "stages.2.5.norm.bias", "stages.2.5.norm.weight", "stages.2.5.pwconv1.bias", "stages.2.5.pwconv1.weight", "stages.2.5.pwconv2.bias", "stages.2.5.pwconv2.weight", "stages.2.6.grn.beta", "stages.2.6.grn.gamma", "stages.2.6.dwconv.bias", "stages.2.6.dwconv.weight", "stages.2.6.norm.bias", "stages.2.6.norm.weight", "stages.2.6.pwconv1.bias", "stages.2.6.pwconv1.weight", "stages.2.6.pwconv2.bias", "stages.2.6.pwconv2.weight", "stages.2.7.grn.beta", "stages.2.7.grn.gamma", "stages.2.7.dwconv.bias", "stages.2.7.dwconv.weight", "stages.2.7.norm.bias", "stages.2.7.norm.weight", "stages.2.7.pwconv1.bias", "stages.2.7.pwconv1.weight", "stages.2.7.pwconv2.bias", "stages.2.7.pwconv2.weight", "stages.2.8.grn.beta", "stages.2.8.grn.gamma", "stages.2.8.dwconv.bias", "stages.2.8.dwconv.weight", "stages.2.8.norm.bias", "stages.2.8.norm.weight", "stages.2.8.pwconv1.bias", "stages.2.8.pwconv1.weight", "stages.2.8.pwconv2.bias", "stages.2.8.pwconv2.weight", "stages.3.0.grn.beta", "stages.3.0.grn.gamma", "stages.3.0.dwconv.bias", "stages.3.0.dwconv.weight", "stages.3.0.norm.bias", "stages.3.0.norm.weight", "stages.3.0.pwconv1.bias", "stages.3.0.pwconv1.weight", "stages.3.0.pwconv2.bias", "stages.3.0.pwconv2.weight", "stages.3.1.grn.beta", "stages.3.1.grn.gamma", "stages.3.1.dwconv.bias", "stages.3.1.dwconv.weight", "stages.3.1.norm.bias", "stages.3.1.norm.weight", "stages.3.1.pwconv1.bias", "stages.3.1.pwconv1.weight", "stages.3.1.pwconv2.bias", "stages.3.1.pwconv2.weight", "stages.3.2.grn.beta", "stages.3.2.grn.gamma", "stages.3.2.dwconv.bias", "stages.3.2.dwconv.weight", "stages.3.2.norm.bias", "stages.3.2.norm.weight", "stages.3.2.pwconv1.bias", "stages.3.2.pwconv1.weight", "stages.3.2.pwconv2.bias", "stages.3.2.pwconv2.weight".