Requesting Qwen-7B Support

[aiaicode]

https://huggingface.co/Qwen/Qwen-7B

https://huggingface.co/Qwen/Qwen-7B-Chat

These two models are performing better than 13B models. and on C-eval beating ChatGPT.

Requesting model support for Llama.cpp.

[vonjackustc]

Qwen is similar to Llama model.
You need add bias params for QKV, and change the tokenizer to tiktoken.

I wonder if we can convert tiktoken vocab format to sentencepiece directly.

[vonjackustc]

https://huggingface.co/JosephusCheung/Qwen-LLaMAfied-7B-Chat
This repo from JosephusCheung can be converted into ggml format. You still need to change the EOS token from ggml source code (2 for llama but other for Qwen).

I think this model is still slightly different from the original Qwen especially in QwenAttentionBlock, Qwen applied log(n) to query.

[wtarreau]

Sadly I couldn't convert it, always getting cryptic python errors.

[arch-btw]

@aiaicode @vonjackustc @wtarreau

#3337

[KerfuffleV2]

I found this script for converting from Tiktoken vocab to HF: https://gist.github.com/xenova/a452a6474428de0182b17605a98631ee (didn't test it, but it looks reasonable and seems to be from a HF person.)

To actually use it, you'll also need to use #3743 since there wasn't already support for loading from merges.txt.

@TheBloke too since I assume you're looking to support these Qwen models.

[ggerganov]

Is there a blocker to support these models in llama.cpp? Would be nice to support the new 1.8B and 72B versions

[simveit]

https://github.com/QwenLM/qwen.cpp
can this maybe be of any help to support qwen in the future?

[choyakawa]

We should first support QKVO bias in llama.
It's in hf llama config.json: "attention_bias": true

[choyakawa]

There was an implementation here but failed, can anyone figure out what's wrong with this?
#3743 (comment)
It seems correct just to add bias terms to each parts.

[ggerganov]

• The bias tensors were not offloaded via a separatecb() call
• Seems like special tokens (<|im_start|>, <|im_end|>) were not escaped in their test?

[DavidGOrtega]

should this be closed by #5037 ?