Statement of the performance update in KnowEdit

[zxlzr]

Dear all,

Recently, with the help from the community (special thanks to @StarLooo), we will update the KnowEdit results (Llama2-7b-chat) in Table 4 of the paper ‘A Comprehensive Study of Knowledge Editing for Large Language Models’. Overall, the results have improved, primarily due to the following reasons:

1. AdaLoRA Optimization: we follow the FT-M instead of the FT-L, which trains the same FFN layer as FT-L using cross-entropy loss on the target answer while masking the original text. This approach not only yields better results but also highlights the optimal performance of AdaLoRA. Meanwhile, the peft version would also affect the performance.

2. ROME and MEMIT Updates: The results are updated after identifying missing components in the local version of the Llama2-7b-chat files (specifically, the legacy feature in the tokenizer_config.json). If you are using the official Llama2-7b-chat model downloaded directly from HF, this issue should not affect your results. Also, we fix a bug related to the padding_size for these two methods, which will influence the performance when you compute the results for batch inputs.

The Table below shows the updated results.

We deeply apologize for any inconvenience caused by this bug.

We will continue improving EasyEdit, updating this paper, and welcome everyone to engage in discussions and share ideas.

EasyEdit Team

[CaiJichang212]

关于这一点：

1. AdaLoRA Optimization: we follow the FT-M instead of the FT-L, which trains the same FFN layer as FT-L using cross-entropy loss on the target answer while masking the original text. This approach not only yields better results but also highlights the optimal performance of AdaLoRA. Meanwhile, the peft version would also affect the performance.

我发现lora的代码没有变化，只是调整了lora的超参数设置和FT-M一致？lora方法性能表现就提升了吗？

[zxlzr]

关于这一点：

1. AdaLoRA Optimization: we follow the FT-M instead of the FT-L, which trains the same FFN layer as FT-L using cross-entropy loss on the target answer while masking the original text. This approach not only yields better results but also highlights the optimal performance of AdaLoRA. Meanwhile, the peft version would also affect the performance.

我发现lora的代码没有变化，只是调整了lora的超参数设置和FT-M一致？lora方法性能表现就提升了吗？

您好FT-M和FT-L是两个不一样的方法，FT-M在屏蔽原始文本的情况下，对目标答案使用交叉熵损失，根据实验结果比FT-L好很多。另外peft版本会对效果产生影响。

[CaiJichang212]

关于这一点：

1. AdaLoRA Optimization: we follow the FT-M instead of the FT-L, which trains the same FFN layer as FT-L using cross-entropy loss on the target answer while masking the original text. This approach not only yields better results but also highlights the optimal performance of AdaLoRA. Meanwhile, the peft version would also affect the performance.

我发现lora的代码没有变化，只是调整了lora的超参数设置和FT-M一致？lora方法性能表现就提升了吗？

您好FT-M和FT-L是两个不一样的方法，FT-M在屏蔽原始文本的情况下，对目标答案使用交叉熵损失，根据实验结果比FT-L好很多。另外peft版本会对效果产生影响。

感谢您的回复，我想知道lora方法性能提升，是修改了代码，优化了损失的计算，还是遵循了FT-M的超参数设置。因为我观察lora的代码似乎没变，超参数文件似乎也没变。

[zxlzr]

关于这一点：

1. AdaLoRA Optimization: we follow the FT-M instead of the FT-L, which trains the same FFN layer as FT-L using cross-entropy loss on the target answer while masking the original text. This approach not only yields better results but also highlights the optimal performance of AdaLoRA. Meanwhile, the peft version would also affect the performance.

我发现lora的代码没有变化，只是调整了lora的超参数设置和FT-M一致？lora方法性能表现就提升了吗？

您好FT-M和FT-L是两个不一样的方法，FT-M在屏蔽原始文本的情况下，对目标答案使用交叉熵损失，根据实验结果比FT-L好很多。另外peft版本会对效果产生影响。

感谢您的回复，我想知道lora方法性能提升，是修改了代码，优化了损失的计算，还是遵循了FT-M的超参数设置。因为我观察lora的代码似乎没变，超参数文件似乎也没变。

您好FT-M和FT-L的损失是有区别的，代码不一样。

[CaiJichang212]

关于这一点：

1. AdaLoRA Optimization: we follow the FT-M instead of the FT-L, which trains the same FFN layer as FT-L using cross-entropy loss on the target answer while masking the original text. This approach not only yields better results but also highlights the optimal performance of AdaLoRA. Meanwhile, the peft version would also affect the performance.

我发现lora的代码没有变化，只是调整了lora的超参数设置和FT-M一致？lora方法性能表现就提升了吗？

您好FT-M和FT-L是两个不一样的方法，FT-M在屏蔽原始文本的情况下，对目标答案使用交叉熵损失，根据实验结果比FT-L好很多。另外peft版本会对效果产生影响。

感谢您的回复，我想知道lora方法性能提升，是修改了代码，优化了损失的计算，还是遵循了FT-M的超参数设置。因为我观察lora的代码似乎没变，超参数文件似乎也没变。

您好FT-M和FT-L的损失是有区别的，代码不一样。

是的，我看到了它们的损失计算方法，是不一样的。
我想问的是“AdaLoRA Optimization”，具体指的是什么优？因为我看lora方法的代码，损失函数还是之前的peft_model的计算方式，超参数也没变？

[CaiJichang212]

[zxlzr]

您好，这个是pelf版本更新和可能超参数的原因。

另外建议您关注下EasyEdit目前新支持的基于大模型的评估https://github.com/zjunlp/EasyEdit/blob/main/examples/run_LLM_evaluation.py ，目前基于teacher forcing 评估效果提升其实并不能说明什么，我们建议大家使用基于生成结果的评估，我们近期会发布一个blog来解读。

[CaiJichang212]

您好，这个是pelf版本更新和可能超参数的原因。

另外建议您关注下EasyEdit目前新支持的基于大模型的评估https://github.com/zjunlp/EasyEdit/blob/main/examples/run_LLM_evaluation.py ，目前基于teacher forcing 评估效果提升其实并不能说明什么，我们建议大家使用基于生成结果的评估，我们近期会发布一个blog来解读。

好的，感谢你的回复。

[CaiJichang212]

您好，这个是pelf版本更新和可能超参数的原因。

另外建议您关注下EasyEdit目前新支持的基于大模型的评估https://github.com/zjunlp/EasyEdit/blob/main/examples/run_LLM_evaluation.py ，目前基于teacher forcing 评估效果提升其实并不能说明什么，我们建议大家使用基于生成结果的评估，我们近期会发布一个blog来解读。

请问你们论文中LoRA的结果，使用了那个版本的peft？