Feature Request: Support for Phi-4-mini-instruct

[MoonRide303]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

Phi-4-mini-instruct uses Phi3ForCausalLM architecture, but conversion fails:

python D:\repos-git\llama.cpp\convert_hf_to_gguf.py --outtype f16 ..\Phi-4-mini-instruct\ --outfile Phi-4-mini-instruct-F16.gguf
INFO:hf-to-gguf:Loading model: Phi-4-mini-instruct
INFO:gguf.gguf_writer:gguf: This GGUF file is for Little Endian only
INFO:hf-to-gguf:Exporting model...
Traceback (most recent call last):
  File "D:\repos-git\llama.cpp\convert_hf_to_gguf.py", line 5112, in <module>
    main()
  File "D:\repos-git\llama.cpp\convert_hf_to_gguf.py", line 5106, in main
    model_instance.write()
  File "D:\repos-git\llama.cpp\convert_hf_to_gguf.py", line 439, in write
    self.prepare_tensors()
  File "D:\repos-git\llama.cpp\convert_hf_to_gguf.py", line 280, in prepare_tensors
    for name, data_torch in chain(self.generate_extra_tensors(), self.get_tensors()):
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\repos-git\llama.cpp\convert_hf_to_gguf.py", line 2568, in generate_extra_tensors
    raise ValueError(f'The length of rope long and short factors must be {rope_dims / 2}')
ValueError: The length of rope long and short factors must be 64.0

Tested using script from b4783.

After disabling rope in config.json ("rope_scaling": null,) it fails at pre-tokenizer:

  File "D:\repos-git\llama.cpp\convert_hf_to_gguf.py", line 716, in get_vocab_base_pre
    raise NotImplementedError("BPE pre-tokenizer was not recognized - update get_vocab_base_pre()")
NotImplementedError: BPE pre-tokenizer was not recognized - update get_vocab_base_pre()

Motivation

Phi series was supported so far, and people might be interested in Phi-4-mini-instruct as well.

Possible Implementation

No response

[foldl]

FYI: Implementation in chatllm.cpp:

foldl/chatllm.cpp@887e621#diff-a68ca0be189c2d9485f5205180ba698c98de44d38cd81e8ee518cde0e9ae6d9c

[MoonRide303]

FYI: Implementation in chatllm.cpp:

foldl/chatllm.cpp@887e621#diff-a68ca0be189c2d9485f5205180ba698c98de44d38cd81e8ee518cde0e9ae6d9c

That was fast! And it seems to be working fine:

E:\chatllm.cpp\main.exe -ngl 99,prolog,epilog -l 4096 -m .\Phi-4-mini-instruct-Q8_0.bin -i
    ________          __  __    __    __  ___
   / ____/ /_  ____ _/ /_/ /   / /   /  |/  /_________  ____
  / /   / __ \/ __ `/ __/ /   / /   / /|_/ // ___/ __ \/ __ \
 / /___/ / / / /_/ / /_/ /___/ /___/ /  / // /__/ /_/ / /_/ /
 \____/_/ /_/\__,_/\__/_____/_____/_/  /_(_)___/ .___/ .___/
You are served by Phi-4,                      /_/   /_/
with 3836021760 (3.8B) parameters.

You  > hi there
A.I. > Hello! How can I assist you today?
You  > who are you?
A.I. > I am Phi, an AI developed by Microsoft. How can I help you?

[bartowski1182]

It seems that this is because we're ignoring the partial_rotary_factor (which previously was just set to 1.0 so was not changing anything)

I'm not sure if it's as simple as adding it to the check here:

llama.cpp/convert_hf_to_gguf.py

Line 2567 in b95c8af

if len(long_factors) != len(short_factors) or len(long_factors) != rope_dims / 2:

or if it also needs to be implemented elsewhere

From configuration_phi3.py:

rotary_ndims = int(self.hidden_size // self.num_attention_heads * self.partial_rotary_factor)
if not len(rope_scaling_short_factor) == rotary_ndims // 2:
    raise ValueError(
        f"`rope_scaling`'s short_factor field must have length {rotary_ndims // 2}, got {len(rope_scaling_short_factor)}"
    )

[ns3284]

I'm a bit new at digging into this, but when debugging, the configuration_phi3.py has these values:

hidden_size=3072
num_attention_heads=32

The results of the config read appear to be incorrect

        n_embd = self.find_hparam(["hidden_size", "n_embd"])
        n_head = self.find_hparam(["num_attention_heads", "n_head"])
        max_pos_embds = self.find_hparam(["n_positions", "max_position_embeddings"])
        orig_max_pos_embds = self.find_hparam(["original_max_position_embeddings"])
        rope_dims = n_embd // n_head
        print (f'n_embd = {n_embd}, n_head = {n_head}, max_pos_embds = {max_pos_embds}, orig_max_pos_embds = {orig_max_pos_embds}, rope_dims = {rope_dims}')

n_embd = 3072, n_head = 24, max_pos_embds = 131072, orig_max_pos_embds = 4096, rope_dims = 128

[bartowski1182]

configuration_phi3.py has default values set, the config.json has different ones

it has 24 for n_head and 0.75 for partial_rotary_factor

[ns3284]

Ah, I see. Looks like other models implement a partial_rotary_factor. Looks like that is the underlying issue, as you pointed out.

[bartowski1182]

Tagging @liuwei-git since he originally implemented the 128k rope changes, and @ngxson since he just seems to know things 😅

[ns3284]

Updated this in convert_hf_to_gguf.py

    def generate_extra_tensors(self) -> Iterable[tuple[str, Tensor]]:
        rot_pct = self.find_hparam(["partial_rotary_factor"])
        n_embd = self.find_hparam(["hidden_size", "n_embd"])
        n_head = self.find_hparam(["num_attention_heads", "n_head"])
        max_pos_embds = self.find_hparam(["n_positions", "max_position_embeddings"])
        orig_max_pos_embds = self.find_hparam(["original_max_position_embeddings"])
        rope_dims = int(rot_pct * n_embd) // n_head

Added this to convert_hf_to_gguf_update.py

{"name": "phi-4-mini-instruct", "tokt": TOKENIZER_TYPE.BPE, "repo": "https://huggingface.co/microsoft/Phi-4-mini-instruct", },

Seems to have worked.

INFO:hf-to-gguf:Model successfully exported to ..\models\Phi-4-mini-instruct\phi-4-mini-instruct-bf16.gguf

[bartowski1182]

Can you confirm coherency of the resulting model? Bonus points if you can show similar output to original

[ns3284]

Will give it a go in a little bit.

[ns3284]

build: 4784 (b95c8af3) with MSVC 19.42.34438.0 for x64
system info: n_threads = 8, n_threads_batch = 8, total_threads = 8

system_info: n_threads = 8 (n_threads_batch = 8) / 8 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | LLAMAFILE = 1 | OPENMP = 1 | AARCH64_REPACK = 1 |

main: HTTP server is listening, hostname: 127.0.0.1, port: 8080, http threads: 7
main: loading model
srv    load_model: loading model 'D:\LLM\models\Phi-4-mini-instruct\phi-4-mini-instruct-bf16.gguf'
llama_model_loader: loaded meta data with 36 key-value pairs and 196 tensors from D:\LLM\models\Phi-4-mini-instruct\phi-4-mini-instruct-bf16.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = phi3
llama_model_loader: - kv   1:              phi3.rope.scaling.attn_factor f32              = 1.190238
llama_model_loader: - kv   2:                               general.type str              = model
llama_model_loader: - kv   3:                               general.name str              = Phi 4 Mini Instruct
llama_model_loader: - kv   4:                           general.finetune str              = instruct
llama_model_loader: - kv   5:                           general.basename str              = Phi-4
llama_model_loader: - kv   6:                         general.size_label str              = mini
llama_model_loader: - kv   7:                            general.license str              = mit
llama_model_loader: - kv   8:                       general.license.link str              = https://huggingface.co/microsoft/Phi-...
llama_model_loader: - kv   9:                               general.tags arr[str,3]       = ["nlp", "code", "text-generation"]
llama_model_loader: - kv  10:                          general.languages arr[str,1]       = ["multilingual"]
llama_model_loader: - kv  11:                        phi3.context_length u32              = 131072
llama_model_loader: - kv  12:  phi3.rope.scaling.original_context_length u32              = 4096
llama_model_loader: - kv  13:                      phi3.embedding_length u32              = 3072
llama_model_loader: - kv  14:                   phi3.feed_forward_length u32              = 8192
llama_model_loader: - kv  15:                           phi3.block_count u32              = 32
llama_model_loader: - kv  16:                  phi3.attention.head_count u32              = 24
llama_model_loader: - kv  17:               phi3.attention.head_count_kv u32              = 8
llama_model_loader: - kv  18:      phi3.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  19:                  phi3.rope.dimension_count u32              = 128
llama_model_loader: - kv  20:                        phi3.rope.freq_base f32              = 10000.000000
llama_model_loader: - kv  21:                          general.file_type u32              = 32
llama_model_loader: - kv  22:              phi3.attention.sliding_window u32              = 262144
llama_model_loader: - kv  23:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  24:                         tokenizer.ggml.pre str              = phi-4-mini-instruct
llama_model_loader: - kv  25:                      tokenizer.ggml.tokens arr[str,200064]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  26:                  tokenizer.ggml.token_type arr[i32,200064]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  27:                      tokenizer.ggml.merges arr[str,199742]  = ["─á ─á", "─á─á ─á─á", "i n", "e r", ...
llama_model_loader: - kv  28:                tokenizer.ggml.bos_token_id u32              = 199999
llama_model_loader: - kv  29:                tokenizer.ggml.eos_token_id u32              = 199999
llama_model_loader: - kv  30:            tokenizer.ggml.unknown_token_id u32              = 199999
llama_model_loader: - kv  31:            tokenizer.ggml.padding_token_id u32              = 199999
llama_model_loader: - kv  32:               tokenizer.ggml.add_bos_token bool             = false
llama_model_loader: - kv  33:               tokenizer.ggml.add_eos_token bool             = false
llama_model_loader: - kv  34:                    tokenizer.chat_template str              = {% for message in messages %}{% if me...
llama_model_loader: - kv  35:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:   67 tensors
llama_model_loader: - type bf16:  129 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = BF16
print_info: file size   = 7.15 GiB (16.00 BPW)
llama_model_load: error loading model: error loading model vocabulary: unknown pre-tokenizer type: 'phi-4-mini-instruct'
llama_model_load_from_file_impl: failed to load model
common_init_from_params: failed to load model 'D:\LLM\models\Phi-4-mini-instruct\phi-4-mini-instruct-bf16.gguf'

[ns3284]

Seems I need to update the tokenizer name to gpt-4o and make similar changes to this:

m18coppola@7fc8505

Utilizing this tokenizer: https://huggingface.co/Xenova/gpt-4o

[ns3284]

Onto the next issue. Phi-4-mini-instruct has "tie_word_embeddings" set to true, which is why there is no output.weights.

[ngxson]

Onto the next issue. Phi-4-mini-instruct has "tie_word_embeddings" set to true, which is why there is no output.weights.

phi-3 doesn't seem to support tied word embd. However, llama does. You can copy the same code from LLM_ARCH_LLAMA:

                    // if output is NULL, init from the input tok embed
                    if (output == NULL) {
                        output = create_tensor(tn(LLM_TENSOR_TOKEN_EMBD, "weight"), {n_embd, n_vocab}, TENSOR_DUPLICATED);
                    }