Repository navigation
fix(server): /v1/internal/mtp-policy reports "profiling" forever when the exactness probe vetoes MTP #1298
Description
Problem
When a server is started with an MTP drafter and the runtime exactness probe declines the pairing (fails under qmv_wide and again with it disabled), the B=1 burst never runs: mtp_capable_target returns false at every consult (src/server/batch/scheduler.rs:4311, :4801) and every request quietly serves classic decode. That fallback is the right behavior. The observability around it is not:
- The adaptive policy (perf(inference): adaptive MTP enable/decline policy from per-pair profiling #333) is still attached, but a vetoed pairing never produces a burst, so the policy never accumulates a qualifying sample and never transitions.
GET /v1/internal/mtp-policy(feat(mtp-policy): expose the settled MTP verdict through a supported interface #1257/feat(mtp-policy): expose the settled verdict at /v1/internal/mtp-policy #1268) therefore reports"state": "profiling"forever, inviting the operator to wait for a verdict that structurally cannot arrive. - The status vocabulary (
unavailable/forced/profiling/settled,src/server/batch/mtp_policy.rs) has no state for "the measured exactness gate refused this pairing", so there is nothing truthful the endpoint could even say today. - The only trace of the veto is the boot-time WARN from
mtp_exactness_gate(deliberately warmed at startup,scheduler.rs:1646), which is visible at the defaultinfofilter but scrolls away with the log.
This is exactly the ambiguity #1257 was filed to remove ("still profiling" must be distinguishable from "will never run"), reintroduced through a path that did not exist when #1257 landed.
Where this bites today
The Gemma 4 31B + bf16 assistant pairing fails the probe under both kernels on every generation 15+ host measured (M3 Ultra: #1279, 231782 of 524288 logit bytes; M5 Max: #1279 comment of 2026-08-22, 245722 of 524288, width-independent across block sizes 2 to 4). On those hosts an operator who configures --model-draft sees classic-decode throughput (a measured 2.25x left on the table on M5 Max) and an endpoint that claims the policy is still profiling.
Fix sketch
The endpoint contract explicitly allows growing the label sets within schema_version 1 ("A new state or a new unavailable reason can appear without a version bump", docs/mtp-policy-api.md).
MtpPolicyStatusgains a state for the veto (wire labelexactness_declined), andMtpPolicySnapshotgains an optional detail field carrying the probe's one-line reason (the same sentence the WARN logs, e.g. "verify block position 0 differs from the single-token chain in 245722 of 524288 logit bytes").mtp_enabledreportsfalse.models::speculative_exactnessrecords the decline reason beside the memoized verdict so the publisher can read it without re-probing.- The scheduler's attach-time publish (
with_adaptive_mtp_policy,scheduler.rs:1700) checks the memoized gate before publishing the policy snapshot: MTP dispatch present butmtp_capable_targetfalse publishes the vetoed snapshot instead of the starving policy'sprofiling. The state cannot change later in the process (the gate is memoized per block width), so attach-time is sufficient. docs/mtp-policy-api.mddocuments the new state and example body; the/healthmirror follows automatically since it renders the same snapshot.
MLXCEL_MTP_ALLOW_INEXACT=1 keeps its current behavior (gate returns true, policy runs normally); surfacing the inexact-engaged condition on the endpoint can be a follow-up if wanted.
Acceptance criteria
- A server whose exactness probe declines the pairing reports the vetoed state (not
profiling) at/v1/internal/mtp-policy, with the probe's reason string -
mtp_enabledisfalsein that state andverdictstays absent - The label addition stays within
schema_version1 per the documented growth rule, anddocs/mtp-policy-api.mdgains the state and an example - Unit coverage for the new wire label and the attach-time publish decision
- Pairings that pass the probe (or pass via the narrow retry) publish exactly what they publish today
References
- fix(speculative): the Gemma 4 31B pairing's exactness probe fails both kernels on M3 Ultra, vetoing the burst #1217 just enabled #1279: the pairing this bites, measured on M3 Ultra and M5 Max
- feat(mtp-policy): expose the settled MTP verdict through a supported interface #1257 / feat(mtp-policy): expose the settled verdict at /v1/internal/mtp-policy #1268: the endpoint whose vocabulary is missing the state
- fix(speculative): probe Gemma 4 MTP exactness instead of assuming it #1258 / feat(speculative): let the MTP gate buy back exactness by dropping qmv_wide #1199: the gate whose verdict needs surfacing
Activity
docs(comments): cut churn from the speculative-decoding comments
fix(server): report the exactness veto at /v1/internal/mtp-policy ins…