Will llama.cpp be able to use Phi-2 ?

[FiveTechSoft]

Surely we have to wait for a GGUF version, but in the meantime just curious about it

thanks

[bachittle]

Here is a related issue: #3146

It is a different model architecture so would require some work, but definitely doable.

[lostmygithubaccount]

also interested in this -- weights are on huggingface from the official microsoft account: https://huggingface.co/microsoft/phi-2/tree/main

[jadore801120]

Any help needed?

[axelpey]

@jadore801120 I think there is help needed. Back when phi-1.5 was released there was already a lot of demand for support and yet it didn't happen (see #3146)

Would be down to work on it, but I don't think I can do it alone

[ggerganov]

Unless I'm missing something, it looks relatively straightforward to support.

The one thing I didn't quite get from the python code is the use of cross-attention in some cases instead of the standard self-attention. Would be helpful if somebody can summarize what is the purpose of cross-attention here

[mory91]

@ggerganov Personally I have found this implementation more intuitive. I think the cross attention is just another way of treating cached keys and values (i.e. nothing special still decoder only).

[dnlwbb]

VLLM has this model implemented, in their conversation they mentioned this:

"I believe the "cross-attention" used in Phi-1.5 is not true cross-attention, it's just used for current token to attend to past KV-cache during autoregressive generation. From what I can see, the Phi-1.5 architecture is basically the same as a GPT-NeoX with attention/FFN in parallel, except:

In each transformer block the pre-FFN and pre-Attention layernorms are the same (GPT-NeoX doesn't share these params, as noted in their paper).
Phi-1.5 has a bias on the output linear layer, and GPT-NeoX does not.
Alternatively, Phi-1.5 has the same architecture as GPT-J, except:

GPT-J has separate q, k, v projections and Phi-1.5 has W_qkv
GPT-J does not have bias on the q, k, v layers, and Phi-1.5 does have bias on W_qkv
It shouldn't be too crazy to adapt these, given that vLLM already supports GPT-NeoX and GPT-J. The MixFormer modelling file on Hugging Face is unnecessarily complicated relative to how small the architecture changes they made are relative to GPT-NeoX/GPT-J. To me it seems like the easiest way would be:

Start with GPT-NeoX model, with the right configuration to match Phi-1.5 (including use_parallel_residual)
When loading weights in from Phi-1.5, just copy the same layernorm weight and bias from Phi into both input_layernorm and post_attn_layernorm in GPT-NeoX
Modify the output linear layer to have a bias so that the bias can be loaded in as well."

[niutech]

FWIW, Phi-2 in GGUF format is also supported by Candle.

[ebeyabraham]

I wrote an implementation based on the MLX code.
ebeyabraham@12cc80c#diff-150dc86746a90bad4fc2c3334aeb9b5887b3adad3cc1459446717638605348efR6239

Had to make some breaking changes to llama.cpp to support the last layer of Phi2 but it works!

[FiveTechSoft]

would love to test it 👍 El vie, 15 dic 2023 a las 22:02, Ebey Abraham ***@***.***>) escribió:

…

I wrote an implementation based on the MLX code <https://github.com/ml-explore/mlx-examples/tree/main/phi2>. Had to make some breaking changes to llama.cpp to support the last layer of Phi2 but it works! — Reply to this email directly, view it on GitHub <#4437 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACOC2UIBXJZK5YYN4E75NWTYJS3EXAVCNFSM6AAAAABATBICY2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNJYGQ4DEMZRG4> . You are receiving this because you authored the thread.Message ID: ***@***.***>

-- Antonio Linares www.fivetechsoft.com

[FiveTechSoft]

@mrgraycode <https://github.com/mrgraycode> is it possible then to generate a GGUF from it already ? El vie, 15 dic 2023 a las 22:34, Aman Salykov ***@***.***>) escribió:

…

hi @mrgraycode <https://github.com/mrgraycode>, I think you forgot to load biases, phi models have both weights + biases — Reply to this email directly, view it on GitHub <#4437 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACOC2UJCDRXMXQAJKSE3LUTYJS673AVCNFSM6AAAAABATBICY2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNJYGUYTMMJYHA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

-- Antonio Linares www.fivetechsoft.com

[ebeyabraham]

Yes, you can download the weights from https://huggingface.co/microsoft/phi-2 and then use convert-hf-to-gguf.py to generate gguf.