Add Support for IBM Granite

[YorkieDev]

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• [ ✅] I am running the latest code. Development is very rapid so there are no tagged versions as of now.
• [✅ ] I carefully followed the README.md.
• [ ✅] I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• [✅ ] I reviewed the Discussions, and have a new bug or useful enhancement to share.

Feature Description

IBM recently released their Granite models. A series of 3b -> 34b coding models with base and instruct finetunes.

https://huggingface.co/collections/ibm-granite/granite-code-models-6624c5cec322e4c148c8b330
https://github.com/ibm-granite

Many thanks to the llama.cpp community for their awesome work! It would be awesome to see this feature added. GGUF's can be made already, but when you try to load them you get a tokenizer error.

[sroecker]

The PR to add granite support for transformers (add MLP bias - gate, up, down) can be found here: https://github.com/huggingface/transformers/pull/30031/files

[psyv282j9d]

Based on the discussion in transformers mlp_bias PR, It's similar to Llama with just the mlp_bias added

[sroecker]

I tried to do this here: https://github.com/sroecker/llama.cpp/tree/add_mlp_bias
Just adding bias to FFN_GATE, FFN_DOWN and FFN_UP.
The tensor shapes seem to be correct but the model outputs gibberish.

./main -m ~/Downloads/granite-3b-code-base.Q8_0.gguf -p "Question: Python code to calculate the Fibonacci series\n\nAnswer:\n" with the GGUF from https://huggingface.co/NikolayKozloff/granite-3b-code-instruct-Q8_0-GGUF (

[mayank31398]

@sroecker are you tying the word embeddings? unlike llama, the input word embeddings and output projection matrix are tied for granite models

[sroecker]

Ah, not yet. Thanks! I guess then we need to define an additional ARCH (or save the mlp_bias boolean in the GGUF) and implement it like with MPT https://github.com/ggerganov/llama.cpp/blob/7e0b6a7b3ba94ff624dc27c1e0e735fded8819b8/llama.cpp#L5287 Mayank Mishra ***@***.***> schrieb am Mi., 8. Mai 2024, 10:59:

…

@sroecker <https://github.com/sroecker> are you tying the word embeddings? unlike llama, the input word embeddings and output projection matrix are tied for granite models — Reply to this email directly, view it on GitHub <#7116 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACYR3PWJGC2D4PTSLE7D6DZBHSNZAVCNFSM6AAAAABHKJDCHOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMBQGA4TQMZSG4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[JohnClaw]

(or save the mlp_bias boolean in the GGUF

Does exist a way to add mlp_bias to already made gguf? I ask about that because you mentioned my q8 gguf in one of your previous messages.

[sroecker]

(or save the mlp_bias boolean in the GGUF

Does exist a way to add mlp_bias to already made gguf? I ask about that because you mentioned my q8 gguf in one of your previous messages.

You could hack something with gguf writer https://pypi.org/project/gguf/

[sroecker]

So I've adapted build_llama to include the MLP biases as well. I've added a few FIXMEs to my branch to indicate places that might need to be adapted for the different Granite models.
Now the output is valid text but unfortunately repeats itself:
<|endoftext|>Question: Fibonacci series in Python? \n\nAnswer: Python? \n\n series in Python? \n\n series in Python? \n\n series in Python? \

[dataf3l]

I'm here to write words of support, I am interested in exploring what IBM + OLLAMA can do

[adrianpuiu]

I'm here to write words of support, I am interested in exploring what IBM + OLLAMA can do

+1 to this