After tokenizers upgrade, the length of the token does not correspond to the length of the model

[CurtainRight]

System Info

transformers：4.48.1
tokenizers：0.2.1
python：3.9

Who can help?

@ArthurZucker @itazap

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

code snippet：

tokenizer = PegasusTokenizer.from_pretrained('IDEA-CCNL/Randeng-Pegasus-238M-Summary-Chinese')
model = AutoModelForSeq2SeqLM.from_pretrained(
    'IDEA-CCNL/Randeng-Pegasus-238M-Summary-Chinese',
    config=config
)

training_args = Seq2SeqTrainingArguments(
    output_dir=config['model_name'],
    evaluation_strategy="epoch", 
    # report_to="none",
    save_strategy="epoch",
    per_device_train_batch_size=4,
    per_device_eval_batch_size=4,
    num_train_epochs=4,
    predict_with_generate=True,
    logging_steps=0.1

)
trainer = Seq2SeqTrainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
    tokenizer=tokenizer,
    data_collator=data_collator,
    compute_metrics=compute_metrics
)

错误信息：

Trial process:
My original Trasnformers: 4.29.1 tokenizers： 0.13.3. The model is capable of reasoning and training normally.
After upgrading, the above error occurred and normal training was not possible. Therefore, I adjusted the length of the model to 'model. resice_tokec_embeddings' (len (tokenizer)). Original model length: 50000, tokenizer loading length: 50103. So the model I trained resulted in abnormal inference results.

Try again, keep tokenizers at 0.13.3, upgrade trasnformers at 4.33.3 (1. I need to upgrade because NPU only supports version 4.3.20. 2. This version is the highest compatible with tokenizers). After switching to this version, training and reasoning are normal.As long as tokenizers is greater than 0.13.3, length changes

Expected behavior

I expect tokenizer to be compatible with the original code

[Rocketknight1]

hi @CurtainRight, this is a known bug in older versions of transformers. It was fixed in this PR: #33325.

You can read an explanation of the bug here: https://nlp.stanford.edu/~johnhew/vocab-expansion.html

If you have to use an older version of transformers, then you will need to manually set the new embedding values after resizing - we can't fix older versions!

[CurtainRight]

@Rocketknight1
My version is 4.48.1, a relatively new version. After referring to the document and executing it, my reasoning is still abnormal and the result is the same as the original reasoning
`import torch
import transformers
from transformers import PegasusForConditionalGeneration

加载 Pegasus 模型

#model = PegasusForConditionalGeneration.from_pretrained('google/pegasus-large')

获取模型参数

params = model.state_dict()
print(params)

获取词嵌入层权重（Pegasus 的词嵌入层是 model.shared.weight）

embeddings = params['model.shared.weight']
print(embeddings)

计算扩展前的词嵌入统计量

pre_expansion_embeddings = embeddings[:-3, :] # 排除最后 3 个 token
mu = torch.mean(pre_expansion_embeddings, dim=0) # 均值
n = pre_expansion_embeddings.size()[0] # 词嵌入数量
sigma = ((pre_expansion_embeddings - mu).T @ (pre_expansion_embeddings - mu)) / n # 协方差矩阵

定义多元正态分布

dist = torch.distributions.multivariate_normal.MultivariateNormal(
mu, covariance_matrix=1e-5 * sigma
)

生成新的词嵌入

new_embeddings = torch.stack(tuple((dist.sample() for _ in range(3))), dim=0)
print(new_embeddings)

替换词嵌入层的最后 3 个 token

embeddings[-3:, :] = new_embeddings
params['model.shared.weight'][-3:, :] = new_embeddings

加载更新后的参数

model.load_state_dict(params)`

[CurtainRight]

@Rocketknight1 And I found that on NPU, although the token has changed, there are no errors and the reasoning is normal. Transformers and tokenizers are the same, only the torch version is different. On NPU, I am 2.1.0