llama: add initial support for Falcon-H1 model family #14534

ibrahimkhadraoui · 2025-07-04T14:21:39Z

Fixes: #13681
Summary
• Add initial support for the Falcon-H1 model family.
• Implements model loading, basic inference, and tokenizer integration for Falcon-H1.
• Updates the build scripts and relevant documentation for Falcon-H1 compatibility.

Details
• Adapted model architecture and layer mapping to match Falcon-H1.
• Integrated Falcon-H1 tokenizer with automatic fallback if tokenizer files are missing.
• Added new test cases to verify Falcon-H1 model loading and inference.
• Cleaned up redundant code from previous Falcon integration attempts.

Notes
• The Falcon-H1 integration follows the same approach as other model families (see llama and Mamba support).
• This supersedes #14238 with a cleaner and more modular implementation.
• Refer to the Falcon-H1 repo for model weights and tokenizer files.

This reverts commit 243e4d1.

…-public into add-fh1-rebased

jacekpoplawski · 2025-07-04T14:29:06Z

Is the old gguf still valid or new one must be generated?

ibrahimkhadraoui · 2025-07-04T14:33:02Z

@jacekpoplawski
You need to re convert with hf_to_gguf we did some changes on the data types

jacekpoplawski · 2025-07-04T14:38:26Z

@jacekpoplawski
You need to re convert with hf_to_gguf we did some changes on the data types

Could you upload new gguf to huggingface?

compilade · 2025-07-04T14:33:06Z

convert_hf_to_gguf.py

+    def get_tensors(self) -> Iterator[tuple[str, Tensor]]:
+        for name, tensor in super().get_tensors():


If possible, modify_tensors should be overridden instead of get_tensors to rename and/or insert tensors.

compilade · 2025-07-04T14:38:33Z

gguf-py/gguf/gguf_writer.py

+    def add_attn_head_count(self, count: int) -> None:
+        self.add_uint32(Keys.Attention.HEAD_COUNT.format(arch=self.arch), count)
+
+    def add_key_value_head_count(self, count: int) -> None:
+        self.add_uint32(Keys.Attention.HEAD_COUNT_KV.format(arch=self.arch), count)
+


Don't these already exist?

compilade · 2025-07-04T14:41:30Z

gguf-py/gguf/constants.py

@@ -172,6 +172,7 @@ class SSM:
        TIME_STEP_RANK = "{arch}.ssm.time_step_rank"
        GROUP_COUNT    = "{arch}.ssm.group_count"
        DT_B_C_RMS     = "{arch}.ssm.dt_b_c_rms"
+        HEAD_DIM       = "{arch}.ssm.head_dim"


The head dimension in Mamba-2 is also the time step rank.

I guess it could be clearer to use a more appropriate name like this, though.

I'm not against, this is only to at least let you know.

compilade · 2025-07-04T14:42:24Z

src/llama-arch.cpp

-    { LLM_KV_FINAL_LOGIT_SOFTCAPPING,           "%s.final_logit_softcapping"           },
+    { LLM_KV_FINAL_LOGIT_SOFTCAPPING,           "%s.final_logit_softcapping"            },


Spurious extra space?

compilade · 2025-07-04T14:45:30Z

src/llama-context.cpp

+    //  try {
+    auto * ctx = new llama_context(*model, params);
+    return ctx;
+    // } catch (const std::exception & err) {
+    //     LLAMA_LOG_ERROR("%s: failed to initialize the context: %s\n", __func__, err.what());
+    // }
+
+    // return nullptr;


Seems like debugging modifications made it into a commit

compilade · 2025-07-04T14:48:28Z

src/llama-arch.cpp

+    { LLM_KV_SSM_CONV_KERNEL,        "%s.ssm.conv_kernel"    },
+    { LLM_KV_SSM_INNER_SIZE,         "%s.ssm.inner_size"     },
+    { LLM_KV_SSM_STATE_SIZE,         "%s.ssm.state_size"     },
+    { LLM_KV_SSM_TIME_STEP_RANK,     "%s.ssm.time_step_rank" },
+    { LLM_KV_SSM_DT_B_C_RMS,         "%s.ssm.dt_b_c_rms"     },
+    { LLM_KV_SSM_GROUP_COUNT,        "%s.ssm.group_count"    },
+    { LLM_KV_SSM_HEAD_DIM,           "%s.ssm.head_dim"       },
+    { LLM_KV_MAMBA_D_SSM,            "%s.ssm.mamba_d_ssm"    },


I feel like these are redundant with the existing metadata used for Mamba.

Any reason to duplicate keys with the exact same meaning?

Note that the %s at the beginning will turn into the arch name anyway (e.g. falcon_h1).

Where it makes sense, try to re-use existing keys.

Yes, the LLM_KV_FALCON_H1_ prefix should be avoided from the enum values - it's too specific. Reuse existing keys and only add new generic keys if needed.

compilade · 2025-07-04T14:53:25Z

src/llama-quant.cpp

+    } else if (name.find("ssm_in.weight") != std::string::npos) {
+        // For mamba-based models it's better to not quantize the ssm-proj layers
+        if (ftype == LLAMA_FTYPE_MOSTLY_Q3_K_S) {
+            new_type = GGML_TYPE_BF16;
+        }


They can be quantized for pure Mamba models, although maybe for hybrid models it makes sense to keep this bigger in mixes.

Why only for Q3_K_S, though?

compilade · 2025-07-04T14:56:36Z

src/llama-arch.cpp

+    { LLM_KV_FALCON_H1_KEY_MULTIPLIER,           "%s.key_multiplier"           },
+    { LLM_KV_FALCON_H1_LM_HEAD_MULTIPLIER,       "%s.lm_head_multiplier"       },
+    { LLM_KV_FALCON_H1_EMBEDDING_MULTIPLIER,     "%s.embedding_multiplier"     },
+    { LLM_KV_FALCON_H1_MAMBA_CHUNK_SIZE,         "%s.ssm.mamba_chunk_size"     },


Is the mamba_chunk_size used anywhere? I thought it was meant to be a runtime parameter for the semi-structured matrices implementation of Mamba-2 (which is not yet implemented in llama.cpp).

compilade · 2025-07-04T15:03:06Z

convert_hf_to_gguf.py

            d_inner = self.find_hparam(["intermediate_size", "d_inner"], optional=True) or 2 * d_model
            n_group = self.hparams.get("n_groups", 1)
+            architectures = self.hparams.get("architectures")
+            if architectures is not None and architectures[0] == "FalconH1ForCausalLM":
+                # FalconH1F has a different d_inner
+                d_inner = self.hparams.get("mamba_d_ssm")


Another (maybe simpler) approach would be to add "mamba_d_ssm" to the find_hparams call for d_inner above.

compilade · 2025-07-04T15:05:20Z

convert_hf_to_gguf.py

        # TODO: does this really matter?
-        assert d_inner == 2 * d_model
-        assert d_inner % head_dim == 0
+        # skip the assertion for FalconH1 Model
+        architectures = self.hparams.get("architectures")
+        if architectures is None or architectures[0] != "FalconH1ForCausalLM":
+            assert d_inner == 2 * d_model
+            assert d_inner % head_dim == 0


That check isn't strictly necessary anyway, and probably should be removed altogether (and also in src/llama-model.cpp).

If you want to keep it for now, but not for Falcon-H1, does self.model_arch correspond to gguf.MODEL_ARCH.FALCON_H1 when it's that arch? (Might be simpler than reading hparams)

ibrahimkhadraoui · 2025-07-04T16:07:55Z

Many thanks @compilade for the review, tomorrow I'll address everything that was mentioned.

ggerganov · 2025-07-05T04:25:55Z

src/llama-graph.cpp

+        if (arch == LLM_ARCH_FALCON_H1) {
+            cur = ggml_scale(ctx0, cur, hparams.mlp_down_multiplier);
+        }


Make this more generic:

Suggested change

if (arch == LLM_ARCH_FALCON_H1) {

cur = ggml_scale(ctx0, cur, hparams.mlp_down_multiplier);

}

if (hparams.mlp_down_multiplier != 1.0f) {

cur = ggml_scale(ctx0, cur, hparams.mlp_down_multiplier);

}

ggerganov · 2025-07-05T04:28:15Z

src/llama-model.cpp

+            if (il == n_layer - 1) {
+                // skip computing output for unused tokens
+                ggml_tensor * inp_out_ids = build_inp_out_ids();
+                cur   = ggml_get_rows(ctx0,   cur, inp_out_ids);


Move the creation of inp_out_ids at the front before the layer loop as in other models.

ggerganov · 2025-07-05T04:30:38Z

src/llama-arch.cpp

+    { LLM_KV_SSM_CONV_KERNEL,        "%s.ssm.conv_kernel"    },
+    { LLM_KV_SSM_INNER_SIZE,         "%s.ssm.inner_size"     },
+    { LLM_KV_SSM_STATE_SIZE,         "%s.ssm.state_size"     },
+    { LLM_KV_SSM_TIME_STEP_RANK,     "%s.ssm.time_step_rank" },
+    { LLM_KV_SSM_DT_B_C_RMS,         "%s.ssm.dt_b_c_rms"     },
+    { LLM_KV_SSM_GROUP_COUNT,        "%s.ssm.group_count"    },
+    { LLM_KV_SSM_HEAD_DIM,           "%s.ssm.head_dim"       },
+    { LLM_KV_MAMBA_D_SSM,            "%s.ssm.mamba_d_ssm"    },


Yes, the LLM_KV_FALCON_H1_ prefix should be avoided from the enum values - it's too specific. Reuse existing keys and only add new generic keys if needed.

younesbelkada and others added 26 commits July 3, 2025 14:49

v1

991de6c

push more fixes

f897efd

another fix

71a6848

fix

03568c9

more fixes

0c93ef6

minor fix

fdd5cff

more cleaning on python code

14c37ec

python fixes

8bea922

changed precision for multipliers float 32->64

071f4b7

fixes

50eadc7

merge

a39a842

another fix

1415cd8

fix

243e4d1

pre-norm -> norm

cce3549

fix

22de62c

Revert "fix"

2fe057c

This reverts commit 243e4d1.

Merge branch 'add-fh1-rebased' of https://github.com/tiiuae/llama.cpp…

d22b4ea

…-public into add-fh1-rebased

fix

6c7d9e2

small fix ffn_norm

15138df

Merge branch 'add-fh1-rebased' of https://github.com/tiiuae/llama.cpp…

a6d0067

…-public into add-fh1-rebased

try

1fd0574

mix instead of max

250b4f1

fix vocab size

3ee7983

Merge branch 'add-fh1-rebased' of https://github.com/tiiuae/llama.cpp…

2aa48dd

…-public into add-fh1-rebased

conflict solve

9760c8b

fixed multipliers

7a25441

github-actions bot added the python python script changes label Jul 4, 2025

younesbelkada mentioned this pull request Jul 4, 2025

MODEL: Falcon-H1 support #14238

Closed

compilade reviewed Jul 4, 2025

View reviewed changes

ggerganov reviewed Jul 5, 2025

View reviewed changes

		def get_tensors(self) -> Iterator[tuple[str, Tensor]]:
		for name, tensor in super().get_tensors():

		{ LLM_KV_FINAL_LOGIT_SOFTCAPPING, "%s.final_logit_softcapping" },
		{ LLM_KV_FINAL_LOGIT_SOFTCAPPING, "%s.final_logit_softcapping" },

llama: add initial support for Falcon-H1 model family #14534

Are you sure you want to change the base?

llama: add initial support for Falcon-H1 model family #14534

Conversation

ibrahimkhadraoui commented Jul 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jacekpoplawski commented Jul 4, 2025

Uh oh!

ibrahimkhadraoui commented Jul 4, 2025

Uh oh!

jacekpoplawski commented Jul 4, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ibrahimkhadraoui commented Jul 4, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ibrahimkhadraoui commented Jul 4, 2025 •

edited

Loading