Feature Request: Qwen3-Next support

[jacekpoplawski]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

Qwen released two Qwen3-Next models:

https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Instruct
https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Thinking

Motivation

These are Qwen’s new SOTA models

Possible Implementation

huggingface/transformers#40771

[nomadictuba2005]

Surprised a PR has not been created yet tbh.

[theo77186]

The main innovation (and difficulty to implement in llama.cpp) is the gated DeltaNet linear attention. Otherwise it looks rather standard.
There are already provisions for hybrid models so this shouldn't be a problem.
Also see:
vllm-project/vllm#24526

edit 1: this model has (standard) attention head dimension size 256, with only 1/4 of it using RoPE (already exists in llama.cpp), the head dimension was causing trouble with flash attention and quantized KV cache (see Gemma 3 4B/12B issues), don't know if it's still the case.
edit 2: the standard attention needs work too as the attention result must then be multiplied by a gating mask, not unlike SwiGLU.
edit 3 (2025-09-27): with the refactoring of FA CUDA kernels, head size 256 KV cache quantization is no longer an issue.

[gopinath87607]

anyone trying this ?

[blart4525-design]

I feel like the Qwen team should work in advance with the llamacpp team

[gopinath87607]

I feel like the Qwen team should work in advance with the llamacpp team

i just started running directly through huggingface transformer itself but not much speedy though

[theo77186]

I feel like the Qwen team should work in advance with the llamacpp team

To be fair with them, Qwen3 had day zero support in llama.cpp. These models are much more complex to implement, mainly due to the (not yet in llama.cpp) linear attention.

Hope it gets support soon no matter what I try I can't get it to quantize with llama.cpp due to the new features the model uses

[MemoryIt]

I am trying to do this with the help of kimi，just convert the model to gguf format. I don't know if I can succeed, it's hard.

Screenshot(GGUF file)

Screenshot(conversion)

Screenshot(some code)

[kyuz0]

I am trying to do this with the help of kimi，just convert the model to gguf format. I don't know if I can succeed, it's hard.

I have never done this myself, but ping me if I can somehow help with testing.

[pwilkin]

Okay, some hints for people who want to try doing the conversion:

• start with existing similar model architectures (here, probably Qwen3MoE) and extend them if possible
• when mapping hparam (hyperparameter / config parameter) names, first search convert_hf_to_gguf.py for whether those parameters were mapped already (possibly for other models), use existing constructs first
• if a parameter is clearly new, due to some novel architectural details, you have to add it to constants.py (and later llama-arch.h/cpp, but get the conversion out of the way first)

[yairpatch]

load_tensors: tensor 'token_embd.weight' (q4_K) (and 614 others) cannot be used with preferred buffer type CUDA_Host, using CPU instead
load_tensors: offloading 0 repeating layers to GPU
load_tensors: offloaded 0/49 layers to GPU
load_tensors:   CPU_Mapped model buffer size = 42826.93 MiB
....................................................................................................
llama_context: constructing llama_context
llama_context: n_seq_max     = 1
llama_context: n_ctx         = 1024
llama_context: n_ctx_per_seq = 1024
llama_context: n_batch       = 1024
llama_context: n_ubatch      = 512
llama_context: causal_attn   = 1
llama_context: flash_attn    = auto
llama_context: kv_unified    = false
llama_context: freq_base     = 10000000.0
llama_context: freq_scale    = 1
llama_context: n_ctx_per_seq (1024) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
set_abort_callback: call
llama_context:        CPU  output buffer size =     0.58 MiB
create_memory: n_ctx = 1024 (padded)
llama_kv_cache: layer   0: dev = CPU
llama_kv_cache: layer   1: dev = CPU
llama_kv_cache: layer   2: dev = CPU
llama_kv_cache: layer   3: dev = CPU
llama_kv_cache: layer   4: dev = CPU
llama_kv_cache: layer   5: dev = CPU
llama_kv_cache: layer   6: dev = CPU
llama_kv_cache: layer   7: dev = CPU
llama_kv_cache: layer   8: dev = CPU
llama_kv_cache: layer   9: dev = CPU
llama_kv_cache: layer  10: dev = CPU
llama_kv_cache: layer  11: dev = CPU
llama_kv_cache: layer  12: dev = CPU
llama_kv_cache: layer  13: dev = CPU
llama_kv_cache: layer  14: dev = CPU
llama_kv_cache: layer  15: dev = CPU
llama_kv_cache: layer  16: dev = CPU
llama_kv_cache: layer  17: dev = CPU
llama_kv_cache: layer  18: dev = CPU
llama_kv_cache: layer  19: dev = CPU
llama_kv_cache: layer  20: dev = CPU
llama_kv_cache: layer  21: dev = CPU
llama_kv_cache: layer  22: dev = CPU
llama_kv_cache: layer  23: dev = CPU
llama_kv_cache: layer  24: dev = CPU
llama_kv_cache: layer  25: dev = CPU
llama_kv_cache: layer  26: dev = CPU
llama_kv_cache: layer  27: dev = CPU
llama_kv_cache: layer  28: dev = CPU
llama_kv_cache: layer  29: dev = CPU
llama_kv_cache: layer  30: dev = CPU
llama_kv_cache: layer  31: dev = CPU
llama_kv_cache: layer  32: dev = CPU
llama_kv_cache: layer  33: dev = CPU
llama_kv_cache: layer  34: dev = CPU
llama_kv_cache: layer  35: dev = CPU
llama_kv_cache: layer  36: dev = CPU
llama_kv_cache: layer  37: dev = CPU
llama_kv_cache: layer  38: dev = CPU
llama_kv_cache: layer  39: dev = CPU
llama_kv_cache: layer  40: dev = CPU
llama_kv_cache: layer  41: dev = CPU
llama_kv_cache: layer  42: dev = CPU
llama_kv_cache: layer  43: dev = CPU
llama_kv_cache: layer  44: dev = CPU
llama_kv_cache: layer  45: dev = CPU
llama_kv_cache: layer  46: dev = CPU
llama_kv_cache: layer  47: dev = CPU
llama_kv_cache:        CPU KV buffer size =    96.00 MiB
llama_kv_cache: size =   96.00 MiB (  1024 cells,  48 layers,  1/1 seqs), K (f16):   48.00 MiB, V (f16):   48.00 MiB
llama_context: enumerating backends
llama_context: backend_ptrs.size() = 2
llama_context: max_nodes = 4920
llama_context: reserving full memory module
llama_context: worst-case: n_tokens = 512, n_seqs = 1, n_outputs = 1
graph_reserve: reserving a graph for ubatch with n_tokens =    1, n_seqs =  1, n_outputs =    1

i was able to convert it but when i try to run it always stall at this point (last line)