[Feat]: Add support for meta llama hf model conversion

[Nick-infinity]

Description:
Llama 2 hf models have weights stored with diff name

[Nick-infinity]

Just parking it here for others. Merge is not required

[karpathy]

I'll take it because i expect a lot of finetunes will be in HF format. Ty.

[mukel]

The HF models apply a permutation to wq and wk, so the RoPE is different fro the original Llama 2 models. The permutation must be undone to export from HF to llam2a.c's simple .bin files.
It took me quite some time to figure this out, the symptom is it starts degenerating and printing gibberish after ~20 or more tokens.

See https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/convert_llama_weights_to_hf.py#L113-L115

[Nick-infinity]

Yes , I have noticed the gibberish too. That's why I left the comment in the script for RoPe encodings extraction. Thanks for validating it

…

On Tue, Aug 15, 2023, 8:18 PM Alfonso² Peterssen ***@***.***> wrote: The HF models apply a permutation to k and v, so the RoPE is different fro the original Llama 2 models. The permutation must be undone to export from HF to llam2a.c's simple .bin files. It took me quite some time to figure this out, the symptom is it starts degenerating and printing gibberish after ~20 or more tokens. — Reply to this email directly, view it on GitHub <#286 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AIHM53ZCZKX72DEMY2GWSC3XVOD5DANCNFSM6AAAAAA3PFC2BE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[karpathy]

... PR fix welcome

[calvintwr]

I don't think this is working. #314 @Nick-infinity @karpathy

[atamurad]

The HF models apply a permutation to wq and wk, so the RoPE is different fro the original Llama 2 models. The permutation must be undone to export from HF to llam2a.c's simple .bin files. It took me quite some time to figure this out, the symptom is it starts degenerating and printing gibberish after ~20 or more tokens.

See https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/convert_llama_weights_to_hf.py#L113-L115

@mukel thanks for tracking this down, fixed it here: #326