fix: externally quantized 4-bit Gemma 4 (OptiQ) decodes to degenerate output that is trapped entirely in reasoning_content #467

Summary

Serving an externally quantized 4-bit Gemma 4 model (gemma-4-31B-it-qat-OptiQ-4bit) on macOS / MLX produces degenerate decode output (a tiny set of tokens repeating, e.g. //, same, 1, even at temperature: 0), and every generated token is routed into reasoning_content while content is left empty. The failure reproduces on the raw /v1/completions endpoint and with speculative decoding disabled, so it is not a chat-template or sampling problem.

Two distinct mlxcel paths combine to produce exactly these symptoms: the 4-bit quantization loader silently reconciles quantization parameters with no validation, and the Gemma 4 thinking-channel filter classifies all output as reasoning whenever the closing channel marker never appears.

Background

This triage is static. The project cannot be built or run in the triage environment (no Metal / MLX), so the analysis below comes from reading the checked-out source, not from reproducing the run.

One clarification on the original report first: <|channel> and <channel|> are not leaked gpt-oss / Harmony tokens. mlxcel deliberately uses that pair as Gemma 4's thinking-channel markers. See src/server/thinking_budget.rs:223-249 (resolves token ids for <|channel> / <channel|>), src/server/routes/chat.rs:930 (OPEN_THINKING_SUFFIXES = ["<|channel>thought\n", ...]) and src/server/chat_template.rs:186-225 (primes <|channel>thought\n for Gemma 4 when enable_thinking=true). So those tokens appearing for a Gemma 4 model is expected; the real fault is the degenerate decode plus the way unclosed thinking is handled.

Path A: degenerate decode (quantization reconciliation)

UnifiedLinear::from_weights() auto-detects the quant mode purely from which tensors are present: .biases present means "affine", otherwise it picks mxfp8 / nvfp4 / mxfp4 (src/lib/mlxcel-core/src/layers.rs:800-822). from_weights_with_mode() then re-derives bits and group_size from tensor shapes via infer_quantization_bits() (src/lib/mlxcel-core/src/layers.rs:724-770, 829-905). That function only returns an error when the inferred bit width is outside {2,3,4,5,6,8}; in every other case it silently overrides the config-declared values. There is no check that the model's declared quantization scheme / group_size actually matches its weight layout. An OptiQ packing that diverges from the affine assumption would be dequantized with the wrong parameters, collapsing the logits, which is the classic signature of repeating tokens at greedy (temperature: 0) decoding. Gemma 4 reads group_size / bits with defaults 64 / 4 (src/models/gemma4.rs:146-158) and threads them into every UnifiedLinear / UnifiedEmbedding layer.

Path B: all output trapped in reasoning_content

For Gemma 4 with enable_thinking=true, the prompt is primed with <|channel>thought\n. The non-streaming path therefore starts the stream filter already inside the Thinking state (StreamFilter::new_primed_open_thinking(), src/server/tool_calls/stream_filter.rs:292-307), and strip_unclosed_primed_thinking() returns an empty string for content whenever the raw output never contains a close marker (<channel|> or </think>) (src/server/routes/chat.rs:954, 964-975, 999-1017). When the degenerate decode never emits <channel|>, every token lands in reasoning_content and content comes back empty, which is precisely the reported shape.

Proposed Solution

1. Validate quantization metadata at load time. Compare the declared scheme, group_size, and bits against the actual tensor shapes, log the detected mode and the reconciled parameters, and fail loudly (clear warning or hard error) on a mismatch instead of silently reconciling. Unsupported external formats such as OptiQ should refuse to load with an actionable message rather than serving garbage.
2. Determine whether OptiQ uses a non-affine packing that the current autodetect misreads. If it is a supportable affine variant, add explicit handling; if not, reject it at load with a clear reason.
3. Make the reasoning routing robust to decode failures. Reconsider silently returning empty content when a primed thinking block is never closed, so a broken decode does not masquerade as a clean but empty response. At minimum, surface the unclosed-thinking condition.

Implementation Notes

• Quant autodetect and reconciliation: src/lib/mlxcel-core/src/layers.rs:724-770 (infer_quantization_bits), :800-822 (from_weights mode detection), :829-905 (from_weights_with_mode), :915-938 (forward).
• Gemma 4 quant config plumbing: src/models/gemma4.rs:52-56 (QuantizationArgs), :146-158 (group_size() / bits() defaults), :3366-3410 (quantized load path), plus the per-layer UnifiedLinear::from_weights(...) and UnifiedEmbedding::from_weights(...) call sites.
• Thinking-channel routing: src/server/routes/chat.rs:930 (OPEN_THINKING_SUFFIXES), :954 (OPEN_THINKING_CLOSE_MARKERS), :964-975 (strip_unclosed_primed_thinking), :999-1017 (extract_reasoning_content); src/server/tool_calls/stream_filter.rs:202-223 (delimiter table), :292-307 (new_primed_open_thinking).
• Channel marker resolution and priming: src/server/thinking_budget.rs:223-249, src/server/chat_template.rs:186-225.

Acceptance Criteria

• Loading a quantized model whose on-disk layout does not match its declared or auto-detected quantization scheme produces a clear warning or error, not a silent reconciliation.
• gemma-4-31B-it-qat-OptiQ-4bit either decodes coherent text or fails to load with an actionable message that names the unsupported quantization format.
• A Gemma 4 generation that never closes the thinking channel no longer silently returns empty content with all text in reasoning_content; the condition is handled or surfaced to the caller.
• Regression coverage exists for the quantization-parameter reconciliation path and for the primed-but-unclosed thinking routing.

Original Suggestion

Title: fix: QAT 4-bit Gemma produces degenerate garbage output with leaked <|channel|> tokens (mlxcel 0.3.2)

Summary

On mlxcel 0.3.2, serving a QAT 4-bit Gemma model produces degenerate garbage output for every request. The model never emits real content, output is a tiny set of repeating tokens (//, same, 1) and, tellingly, <|channel|> special tokens that do not belong to the Gemma vocabulary (they are OpenAI Harmony / gpt-oss tokens). All generated text is also misrouted into reasoning_content with content left empty.

The fault persists with MTP / speculative decoding disabled and reproduces on the raw /v1/completions endpoint (which bypasses the chat template), so it is not a prompting/template issue, it points to the inference/quant/tokenizer path inside mlxcel.

• Version: mlxcel 0.3.2
• Model: gemma-4-31B-it-qat-OptiQ-4bit
• Platform: macOS (Apple Silicon, MLX)

Reproduction

1. Chat completion, empty content, garbage reasoning_content

curl -s http://HOST:PORT/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "gemma-4-31B-it-qat-OptiQ-4bit",
  "messages": [{"role":"user","content":"What is the capital of France? Answer in one word."}],
  "max_tokens": 200, "temperature": 0
}'

Response (MTP disabled):

{"choices":[{"message":{"role":"assistant","content":"",
  "reasoning_content":"1\n\n\n//\n same////\n1\n\n\nthought//\n\n\n uma\n//\n//////\n//..."},
  "finish_reason":"length"}]}

2. Raw completion, bypasses chat template, still degenerate (key signal)

curl -s http://HOST:PORT/v1/completions -H "Content-Type: application/json" -d '{
  "model": "gemma-4-31B-it-qat-OptiQ-4bit",
  "prompt": "The capital of France is", "max_tokens": 15, "temperature": 0
}'

Response:

{"choices":[{"text":"<|channel><|channel> same<|channel> same same same<|channel> same<|channel><|channel> same<|channel><|channel><|channel>","finish_reason":"length"}]}

Reproduces at temperature: 0 and temperature: 0.7.

Why this looks like an mlxcel bug, not a bad model file

1. <|channel|> is not in Gemma's vocabulary, it is a Harmony/gpt-oss special token. Its appearance suggests the QAT 4-bit decode path or tokenizer/special-token handling is mixing up vocab/embeddings.
2. Greedy decode (temp 0) collapses to a handful of repeating token IDs, the classic signature of broken dequantization / weight mapping, not sampling.
3. All output is forced into reasoning_content with empty content, implying the reasoning-channel parser is triggering on tokens that should not be emitted.
4. Independent of MTP/speculative decoding and independent of the chat template (raw /v1/completions reproduces).

Suspected area

QAT 4-bit (OptiQ) dequantization and/or tokenizer/special-token wiring in the 0.3.2 inference path.

Possibly related

• fix(serve): mlxcel serve aborts with a nested-runtime panic when auto-downloading a missing model #463 (serve panics when downloading a model)
• fix(cli): align mlxcel-server drafter flags with mlxcel serve (--draft-model rejected) #464 (server does not understand --draft-model)

These were hit earlier in the same session while bringing the server up; this issue is the distinct garbage-generation problem that remains after those are worked around.