The pre-trained weight file (checkpoint) does not match the model architecture defined by your current code.

[LTC232]

Thank you for your research contributions and for open-sourcing the code in this field. However, when we tried to run “python run.py --cfg configs/liger_gla.yaml” and “python run.py --cfg configs/liger_gsa.yaml” （https://huggingface.co/linear-moe-hub/Liger-GLA-8B）, we encountered the following error for both:
“
raise RuntimeError(
RuntimeError: Error(s) in loading state_dict for Embedding:
size mismatch for weight: copying a param with shape torch.Size([128256, 4096]) from checkpoint, the shape in current model is torch.Size([32000, 4096]).”

This seems to indicate a discrepancy between the model configuration provided by the project and the checkpoint weights. Could you please provide a solution?

Looking forward to your reply. Best wishes for your work!

[weigao266]

Thanks for trying our work. Could you checkout to the liger_paper branch and try it again?

[LTC232]

Thank you for your reply. We have resolved the model loading issue, but now we're encountering a new problem: When we run the command “python run.py --cfg configs/liger_gla.yaml”, the code executes until the LigerGLADecoderLayer class. Since it lacks a forward method, it proceeds to execute the forward method of its parent class LlamaDecoderLayer, which calls self_attn and causes an error. Could you provide an implementation of the forward method for the LigerGLADecoderLayer class?