Skip to content

fix(cli): mlxcel generate --draft-kind mtp prints no generated text #1200

Closed as not planned

Description

@inureyes

Problem

The offline MTP path generates tokens and reports the count and rate, then prints none of the text.

$ ./target/release/mlxcel generate -m models/qwen3.8-27b-4bit -n 120 --temp 0 \
    --draft-model models/qwen3.8-27b-mtp-bf16 --draft-kind mtp --draft-block-size 3 \
    -p "Explain in detail how a transformer neural network works, ..."
Generating...
Explain in detail how a transformer neural network works, ...Loading MTP drafter from "models/qwen3.8-27b-mtp-bf16"...
MTP drafter loaded and bound (block_size = 3).


[Generated 120 tokens in 3.34s = 35.90 tok/s]

Two blank lines where the completion should be. The same command without --draft-model prints the text normally.

The tokens are real

The round-loop diagnostics account for them, so this is a reporting gap and not a generation failure:

rounds=51 proposed_tokens=101 accepted_draft_tokens=69 acceptance_rate=0.683
emitted_from_verify_tokens=119 emitted_per_verify=2.333 decode_ms=3179.99

The text is on neither stream: stdout holds the banner and the timing line, stderr holds only the tracing output.

Scope

Why it matters beyond cosmetics

It removes the only way to check MTP's own contract from the CLI. MTP promises temperature-0 output byte-identical to classic decode, and the natural check is to diff generate with and without --draft-model. That check is impossible today, so every byte-identity claim on this path rests on the model-level exactness probe instead of on observed output. Both #1187 and #1189 are in that position right now.

It also means a user cannot use offline MTP for anything, since the feature's output never reaches them.

Acceptance criteria

  • mlxcel generate --draft-kind mtp streams the generated text the same way the non-speculative path does.
  • A test pins it, so the path cannot go silent again without failing.
  • Checked whether mlxcel-server and the other drafter kinds have the same gap, with the answer recorded either way.
  • With output restored, diff temperature-0 generate with and without --draft-model on a checkpoint whose gate passes, and record whether they match. That is the end-to-end confirmation the probe currently stands in for.

Activity

  1. added
    type:bugBug fixes, error corrections, or issue resolutions
    area:inferenceGeneration, sampling, decoding (incl. speculative, DRY)
    area:cliCommand-line interface / CLI flags
    on Aug 17, 2026
  2. inureyes commented on Aug 17, 2026

    @inureyes
    MemberAuthor

    Not a defect. The text is generated and printed; filter_reasoning_for_display suppresses it because qwen3.8-27b-4bit stays inside its thinking channel for the whole 120-token budget, so there is no post-channel answer to show. --show-reasoning prints it, and a longer budget would too.

    The classic-decode run in the report looked different only because its own reasoning happened to close within the budget on that sample, which made the two paths look like they disagreed about whether to print at all.

    The acceptance criterion about diffing generate with and without --draft-model was worth keeping though: running it is what showed MTP's output is not byte-identical to classic decode on this pairing, independent of block width and of the #1199 switch. That is tracked separately.

  3. added
    status:invalidReported behavior is not a defect; the premise did not hold
    and removed on Aug 17, 2026
  4. self-assigned this
    on Aug 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:cliCommand-line interface / CLI flagsarea:inferenceGeneration, sampling, decoding (incl. speculative, DRY)priority:highHigh prioritystatus:invalidReported behavior is not a defect; the premise did not holdtype:bugBug fixes, error corrections, or issue resolutions

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions