Repository navigation
fix(cli): mlxcel generate --draft-kind mtp prints no generated text #1200
Description
Problem
The offline MTP path generates tokens and reports the count and rate, then prints none of the text.
$ ./target/release/mlxcel generate -m models/qwen3.8-27b-4bit -n 120 --temp 0 \
--draft-model models/qwen3.8-27b-mtp-bf16 --draft-kind mtp --draft-block-size 3 \
-p "Explain in detail how a transformer neural network works, ..."
Generating...
Explain in detail how a transformer neural network works, ...Loading MTP drafter from "models/qwen3.8-27b-mtp-bf16"...
MTP drafter loaded and bound (block_size = 3).
[Generated 120 tokens in 3.34s = 35.90 tok/s]
Two blank lines where the completion should be. The same command without --draft-model prints the text normally.
The tokens are real
The round-loop diagnostics account for them, so this is a reporting gap and not a generation failure:
rounds=51 proposed_tokens=101 accepted_draft_tokens=69 acceptance_rate=0.683
emitted_from_verify_tokens=119 emitted_per_verify=2.333 decode_ms=3179.99
The text is on neither stream: stdout holds the banner and the timing line, stderr holds only the tracing output.
Scope
- Reproduced on Apple M5 Max, macOS 26.6.1,
mainat4e414bd6,models/qwen3.8-27b-4bitwith theqwen3.8-27b-mtp-bf16drafter. - Present with and without
RUST_LOG=info, and with and withoutMLXCEL_MTP_ALLOW_INEXACT=1, so it is not a logging interleave. - Pre-existing. The same behavior is in a 2026-08-17 02:12 UTC run recorded before the MLX pin moved to
9a795735(chore(mlx): bump pin to 9a795735 and rebase the four drifted overlays #1198), so neither that bump nor the perf(speculative): recover MTP throughput on Apple GPU generation 15+ without giving up temperature-0 byte-identity #1187 branch caused it. - Not checked: whether
mlxcel-serveris affected, and whether the other drafter kinds (dflash, plain speculative) share the path.
Why it matters beyond cosmetics
It removes the only way to check MTP's own contract from the CLI. MTP promises temperature-0 output byte-identical to classic decode, and the natural check is to diff generate with and without --draft-model. That check is impossible today, so every byte-identity claim on this path rests on the model-level exactness probe instead of on observed output. Both #1187 and #1189 are in that position right now.
It also means a user cannot use offline MTP for anything, since the feature's output never reaches them.
Acceptance criteria
-
mlxcel generate --draft-kind mtpstreams the generated text the same way the non-speculative path does. - A test pins it, so the path cannot go silent again without failing.
- Checked whether
mlxcel-serverand the other drafter kinds have the same gap, with the answer recorded either way. - With output restored, diff temperature-0
generatewith and without--draft-modelon a checkpoint whose gate passes, and record whether they match. That is the end-to-end confirmation the probe currently stands in for.
Activity
inureyes commentedon Aug 17, 2026
Not a defect. The text is generated and printed;
filter_reasoning_for_displaysuppresses it becauseqwen3.8-27b-4bitstays inside its thinking channel for the whole 120-token budget, so there is no post-channel answer to show.--show-reasoningprints it, and a longer budget would too.The classic-decode run in the report looked different only because its own reasoning happened to close within the budget on that sample, which made the two paths look like they disagreed about whether to print at all.
The acceptance criterion about diffing
generatewith and without--draft-modelwas worth keeping though: running it is what showed MTP's output is not byte-identical to classic decode on this pairing, independent of block width and of the #1199 switch. That is tracked separately.