Problems about Hugging Face model to the gguf format.

[NeoFii]

Issues Converting Hugging Face Model to gguf Format

I encountered problems while attempting to convert a Hugging Face model to the gguf format on an Ubuntu system. Here is my environment information:

• PyTorch: 2.2.1
• CUDA: 12.1.1
• Python: 3.11
  I used the following commands to clone and prepare the llama.cpp project:

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
pip3 install -r requirements.txt

I successfully merged the safetensors file and saved it as a PyTorch model. However, I ran into an issue when using the convert.py script to convert the model to the GGUF format with f16 precision. The command I used was:

python3 convert.py /hy-tmp/converted_model --outfile /hy-tmp/model.gguf --outtype f16

The error that occurred is: KeyError: 'transformer.h.0.attn.c_attn.bias'
Could anyone assist me in resolving this issue?

An error occurred：KeyError: 'transformer.h.0.attn.c_attn.bias'
Could anyone help me?

[NeoFii]

The huggingface model is Qwen1.5-7B-Chat

[compilade]

Try using convert-hf-to-gguf.py instead of convert.py, which only supports legacy Llama models (and which will be renamed to reflect this in #7430).

I successfully merged the safetensors file and saved it as a PyTorch model.

Note that you don't need to do this, since convert-hf-to-gguf.py natively supports loading multi-part safetensors models.

Your command should then look like:

$ python3 convert-hf-to-gguf.py /hy-tmp/Qwen1.5-7B-Chat/ --outfile /hy-tmp/qwen1.5-7b-chat.f16.gguf --outtype f16

[NeoFii]

尝试使用convert-hf-to-gguf.py代替convert.py，它仅支持旧版 Llama 模型（并且将被重命名以反映#7430中的这一点）。

我成功合并了 safetensors 文件并将其保存为 PyTorch 模型。

请注意，您不需要这样做，因为convert-hf-to-gguf.py本机支持加载多部分安全张量模型。

您的命令应如下所示：

$ python3 convert-hf-to-gguf.py /hy-tmp/Qwen1.5-7B-Chat/ --outfile /hy-tmp/qwen1.5-7b-chat.f16.gguf --outtype f16

I have solved the problem thanks for your help.

[arnfaldur]

Please close the issue as it is resolved.