model loading the checkpoint error

[TIANRENK]

RuntimeError: Error(s) in loading state_dict for BertModel:
size mismatch for embeddings.token_type_embeddings.weight: copying a param of torch.Size([16, 768]) from checkpoint, where the shape is torch.Size([2, 768]) in current model.

[TIANRENK]

But I print the model.embeddings.token_type_embeddings it was Embedding(16,768) .

[thomwolf]

which model are you loading?

[TIANRENK]

which model are you loading?

the pre-trained model chinese_L-12_H-768_A-12

[TIANRENK]

mycode:
bert_config = BertConfig.from_json_file('bert_config.json')
model=BertModel(bert_config)
model.load_state_dict(torch.load('pytorch_model.bin'))

The error:
RuntimeError: Error(s) in loading state_dict for BertModel:
size mismatch for embeddings.token_type_embeddings.weight: copying a param of torch.Size([16, 768]) from checkpoint, where the shape is torch.Size([2, 768]) in current model.

[thomwolf]

I'm testing the chinese model.
Do you use the config.json of the chinese_L-12_H-768_A-12 ?
Can you send the content of your config_json ?

[TIANRENK]

I'm testing the chinese model.
Do you use the config.json of the chinese_L-12_H-768_A-12 ?
Can you send the content of your config_json ?

In the 'config.json' of the chinese_L-12_H-768_A-12 ,the type_vocab_size=2.But I change the config.type_vocab_size=16, it still error.

[TIANRENK]

I'm testing the chinese model.
Do you use the config.json of the chinese_L-12_H-768_A-12 ?
Can you send the content of your config_json ?

{
"attention_probs_dropout_prob": 0.1,
"directionality": "bidi",
"hidden_act": "gelu",
"hidden_dropout_prob": 0.1,
"hidden_size": 768,
"initializer_range": 0.02,
"intermediate_size": 3072,
"max_position_embeddings": 512,
"num_attention_heads": 12,
"num_hidden_layers": 12,
"pooler_fc_size": 768,
"pooler_num_attention_heads": 12,
"pooler_num_fc_layers": 3,
"pooler_size_per_head": 128,
"pooler_type": "first_token_transform",
"type_vocab_size": 2,
"vocab_size": 21128
}

I change my code:
bert_config = BertConfig.from_json_file('bert_config.json')
bert_config.type_vocab_size=16
model=BertModel(bert_config)
model.load_state_dict(torch.load('pytorch_model.bin'))

it still error.

[TIANRENK]

I see you have "type_vocab_size": 2 in your config file, how is that?

Yes,but I change it in my code.

[TIANRENK]

is your pytorch_model.bin the good converted model of the chinese one (and not of an English one)?

I think it's good.

[thomwolf]

Ok, I have the models. I think type_vocab_size should be 2 also for chinese. I am wondering why it is 16 in your pytorch_model.bin

[TIANRENK]

I have no idea.Did my model make the wrong convert?

[thomwolf]

I am testing that right now. I haven't played with the multi-lingual models yet.

[TIANRENK]

I am testing that right now. I haven't played with the multi-lingual models yet.

I also use it for the first time.I am looking forward to your test results.

[TIANRENK]

I am testing that right now. I haven't played with the multi-lingual models yet.

When I was converting the model .

Traceback (most recent call last):
File "convert_tf_checkpoint_to_pytorch.py", line 95, in
convert()
File "convert_tf_checkpoint_to_pytorch.py", line 85, in convert
assert pointer.shape == array.shape
AssertionError: (torch.Size([16, 768]), (2, 768))

[thomwolf]

are you supplying a config file with "type_vocab_size": 2 to the conversion script?

[TIANRENK]

are you supplying a config file with "type_vocab_size": 2 to the conversion script?

I used the 'bert_config.json' of the chinese_L-12_H-768_A-12 when I was converting .

[thomwolf]

Ok, I think I found the issue, your BertConfig is not build from the configuration file for some reason and thus use the default value of type_vocab_size in BertConfig which is 16.

This error happen on my system when I use config = BertConfig('bert_config.json') instead of config = BertConfig.from_json_file('bert_config.json').

I will make sure these two ways of initializing the configuration file (from parameters or from json file) cannot be messed up.

[imxiaomin]

运行时错误：加载 BertModel state_dict时出错：embeddings.token_type_embeddings 的大小不匹配.weight：
复制火炬参数。大小（[16， 768]） 从检查点开始，其中形状为火炬。当前模型中的大小（[2， 768]

i have the same problem as you. did you solve the problem?