Repository navigation
feat(provider): MTP policy lives in the weights — auto activates embedded Qwen heads only - #777
Merged
Merged
Conversation
#752 (v0.8.14) made MTPMode.auto architecture-based: any qwen3_5_moe checkpoint drafted. #700 narrowed it to the single pinned EigenLabs/Qwen3.8-27B-4bit, silently dropping auto-MTP for the Qwen3.5/3.6 35B MoE fleet on upgrade — and dense qwen3_5 checkpoints (the 9B class) never had it. .auto now enables MTP for the pinned Qwen3.8 target plus, by config.json model_type, the qwen3_5 / qwen3_5_moe family — the same set SpecDecArtifactFunnel.isQwen35Target already resolves. The funnel stays the authority: a family checkpoint without an MTP artifact still falls back target-only with a recorded reason, and the kill switch is unchanged. enablesMTP now takes the model type; callers that only know an id keep the pinned-id decision. Verified live on an M4 Max, default config (no mtp_mode key): - Qwen3.5-9B inline MTP: 80.7 tok/s vs 47.6 target-only (1.7x) - Qwen3.6-35B inline MTP: 44-52 tok/s vs 26.5 kill-switch (1.9x) - Qwen3.8-27B pinned path: unchanged (canary + resolver tests)
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Gajesh2007
had a problem deploying
to
benchmarks
August 29, 2026 03:41 — with 
GitHub Actions
Failure
GitHub Actions
Failure
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
…ead path Qwen 3.8-27B moves from the separately published, revision-pinned external assistant (EigenLabs/Qwen3.8-27B-MTP-4bit) to the embedded inline artifact (mtplx_mtp in the target checkpoint), the same shape the 3.6-35B and the 9B ship. The pairing now travels inside the weights, so the machinery that existed only to pin it outside is scrap: - delete StandaloneQwen38MTPResolver (exact-revision cache staging, samePayload verification, recordingSourceRevision binding) and its branch in StandaloneServer.specDecPreparation — the funnel is the one mechanism - delete MTPMode.automaticTargetModelID and collapse enablesMTP to the model-type-only family rule (the pin gated the external pair and nothing else) - delete the 'Standalone exact Qwen3.8 MTP cache' test suite with the resolver; MTPConfigTests now pin the family rule directly Fleet (coordinator) mode is untouched: catalog spec-dec resolution keeps serving the external head until the registry switches the 27B entry to the embedded artifact. Verified live on an M4 Max, default config (no mtp_mode key), on the final binary: embedded 27B artifact drafts at 28-31 tok/s vs 20.3-20.6 with the kill switch; 9B 80.7 vs 47.9; 3.6-35B 44-52 vs 26.5.
Gajesh2007
had a problem deploying
to
benchmarks
August 29, 2026 03:53 — with
GitHub Actions
Failure
GitHub Actions
Failure
Gajesh2007
changed the title
Contributor
|
Found 1 test failure on Blacksmith runners: Failure
|
![Fix with [code]smith](https://pr-comments-assets.blacksmith.sh/codesmith/fix-with-codesmith-light.png)
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e95206d1ad
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
.auto now requires BOTH the hardcoded Qwen family gate (qwen3_5, qwen3_5_moe) AND an embedded-head declaration (mtplx_mtp.included=true in the checkpoint's config.json). A Qwen checkpoint without an embedded head is not asked about at all — no catalog lookup, no prefetch, plain config-disabled fallback. Separately published assistants and mtp_drafter_path overrides now require an explicit mtp_mode="on", and catalog prewarm runs only under "on" (auto never consults the catalog). The weights ship the policy: publishing a checkpoint with an embedded head is what turns MTP on for it, with no per-model code or registry spec-dec metadata. Verified live under a default config on the gated binary: embedded 9B drafts at 79-83 tok/s; the bare mlx-community 9B serves target-only. 74/74 MTP suite tests.
Gajesh2007
had a problem deploying
to
benchmarks
August 31, 2026 04:56 — with
GitHub Actions
Failure
GitHub Actions
Failure
Gajesh2007
changed the title
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6daf57c79f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Codex review: 'darkbloom beta list' still said automatic mode enables only the exact Qwen3.8 target. It now describes the embedded-gated family rule, and notes that catalog assistants and mtp_drafter_path overrides require the explicit on override.
Gajesh2007
had a problem deploying
to
benchmarks
August 31, 2026 05:19 — with
GitHub Actions
Failure
GitHub Actions
Failure
Member
Author
|
Codex review dispositions:
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 135dae1450
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…her MLX suite QwenVLMTargetExtractionTests evaluates MLX arrays through tiny synthetic modules but was the one MLX-touching suite without LiveInferenceFixtures.ensureMetallibColocated(); a fresh worktree needed the metallib manually copied beside the test runner. The suite init now self-heals: verified by deleting the manual copy and re-running 17/17.
Gajesh2007
had a problem deploying
to
benchmarks
August 31, 2026 05:25 — with
GitHub Actions
Failure
GitHub Actions
Failure
…iagnostics Codex review: the boolean declaresInlineArtifact applied the inspection's strict 1 MiB config cap, so a valid-but-oversized config.json that DECLARES an embedded head read as 'absent' — the auto pre-gate then reported config_disabled and the funnel never surfaced inline_artifact_invalid. SpecDecStore.inlineDeclarationProbe now distinguishes declared / absent / undeterminable with a deliberately looser probe bound (64 MiB): declared and undeterminable configs reach inspectInlineArtifact, whose strict cap rejects loudly with the named reason; only a parsed config without a declaration falls through to the catalog resolver silently, as designed. All three callers (funnel inline branch + both auto pre-gates) migrated; boolean discriminator deleted. New InlineDeclarationProbeTests cover the tri-state, the oversized-declared end-to-end inline_artifact_invalid, and the oversized-undeclared catalog fallthrough.
…olicy Codex review: beta-features.md and cli-reference.md still described MTP as a default-off Gemma-only feature that installing a release never enables. Both now document auto's embedded-Qwen activation, the explicit-off rollback, and that catalog spec_dec assistants and mtp_drafter_path overrides require the explicit on override.
Consolidated from #774 (closed in its favor): the validation record behind this PR's policy change, including the embedded-head build conventions, the measured MTP results, and the OPEN greedy-token-parity validation item that gates registry rollout of the embedded artifacts.
Gajesh2007
had a problem deploying
to
benchmarks
August 31, 2026 05:43 — with
GitHub Actions
Failure
GitHub Actions
Failure
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: da41944ab0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
1. Single-probe inline decision (TOCTOU): the funnel re-probed config.json after the auto pre-gate; a config change between the two reads could flip an embedded-only decision into a catalog resolution auto forbids. SpecDecArtifactFunnel.Request now carries the caller's one authoritative InlineDeclarationProbe and the funnel never re-probes. Pinned by SingleProbeInlineDecisionTests. 2. Inline assistants no longer charge twice: ModelScanner already counts the embedded mtp shard in the target's sizeBytes/estimatedMemoryGb, yet admission and the pending-load reservation added residentBytes on top — near the cap that misreported assistant_memory_unavailable for a fitting target+assistant. SpecDecArtifact.additionalWeightBytes is 0 for .inline and residentBytes otherwise; all three charge sites (ProviderLoop admission, pending-load reservation, standalone) use it. 3. EigenLabs/Qwen3.8-27B-4bit-mtp inherits the target's hardware invariant: ModelRuntimeRequirements now gates every concrete Qwen3.8 build id (M5 + MLX NAX), so the embedded publication cannot dodge eligibility by renaming. Pinned by EmbeddedQwen38RuntimeRequirementsTests. 107/107 across the MTP, funnel, and runtime-requirements suites.
Gajesh2007
had a problem deploying
to
benchmarks
August 31, 2026 05:59 — with
GitHub Actions
Failure
GitHub Actions
Failure
Gajesh2007
added a commit
that referenced
this pull request
Aug 31, 2026
…60822 Brings the test-suite refactor up to date with the 22 commits master gained since the last sync (#700-#790: Qwen3-VL and Qwen3.8 serving, first-content deadline admission, MTP-in-the-weights, serial-number privacy, MLX core 0.32.2, usage-checkpoint fixes). Every master-side test added or modified inside files this branch deletes, splits, or rewrites was re-homed into the refactor's successor suites and lanes rather than dropped, including: - PrefillDeadlineProductionConfigTests and the Qwen required-tool-choice tests into the slimmed EngineV2ProductionWiringTests - bridgeDeadlineRefusalReleasesRealStaging into SSDPrefixCacheReservationTests - excludesInferenceControls into WatchdogRecoveryTests - the rewritten #777 MTP config/beta suites into ConfigTests and BetaFeaturesTests (MTPConfigTests stays deleted) - vision/DeepStack/media additions into the split EngineV2Vision* and MediaIngest suites - the #704 writeDeferred provider-writer tests, first-content-budget wire test, #730 serial-privacy subtests, and #752 benchmark-control tests into their branch successors Master-added test files were audited against the lane architecture: the Qwen3.8 production canary trio moved to Live/ (with a #filePath depth fix), SchedulerPrefillDecision's real-weights suite split out to Probes/ behind its per-probe gates, and hermetic additions stay in the default lane. docs/developer/test.md updated to match. Production-code unions of note: registry.go keeps both the branch's PrivateOnly/catalog public-listing gates and master's per-model capability gating; provider_writer.go carries both determinism-seam families; LaunchAgent forwards the union of both env allowlists; postgres_model_registry.go extends nil-canonicalization to master's new required_provider_capabilities column. The branch-side prefill-keepalive tests were dropped with the feature master removed in #704. Two master-branch interaction defects fixed so the merged tree is green: the store contract harness now re-seeds the usage_totals counter row its truncate wipes (the #665 migration guard treats the bare marker as corruption), and the #730 log-report migration test restores the pooled connection's search_path before release. Submodules: mlx-swift and mlx-swift-lm test branches merged with master's 0.32.2-era pins; relocated suites verified present. Verified: go vet + full coordinator/testbed tests green; store contracts green twice consecutively against real Postgres 16; swift build --build-tests clean; default Swift lane 2,264 tests in 248 suites all passing.
Gajesh2007
added a commit
that referenced
this pull request
Sep 1, 2026
master now serves qwen3_vl_moe directly (#752, vision off M5 #781) and auto-activates embedded MTP heads (#777): the "dark fleet-wide / no CBv2 adapter" claims become a bench-side measurement gap (LLM-factory load cannot construct the wrapper; floor stays at the 5.5 default, which fits its 32 GB tier), and the qwen3.6 production route now includes MTP k=2 — the B=1-measured +0.4 GiB allowance applies to the live route (floors remain deferred for the vision reason). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UxThXGrq5VyXfUzcahTVw6
Gajesh2007
added a commit
that referenced
this pull request
Sep 2, 2026
master now serves qwen3_vl_moe directly (#752, vision off M5 #781) and auto-activates embedded MTP heads (#777): the "dark fleet-wide / no CBv2 adapter" claims become a bench-side measurement gap (LLM-factory load cannot construct the wrapper; floor stays at the 5.5 default, which fits its 32 GB tier), and the qwen3.6 production route now includes MTP k=2 — the B=1-measured +0.4 GiB allowance applies to the live route (floors remain deferred for the vision reason). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UxThXGrq5VyXfUzcahTVw6
Gajesh2007
added a commit
that referenced
this pull request
Sep 2, 2026
master now serves qwen3_vl_moe directly (#752, vision off M5 #781) and auto-activates embedded MTP heads (#777): the "dark fleet-wide / no CBv2 adapter" claims become a bench-side measurement gap (LLM-factory load cannot construct the wrapper; floor stays at the 5.5 default, which fits its 32 GB tier), and the qwen3.6 production route now includes MTP k=2 — the B=1-measured +0.4 GiB allowance applies to the live route (floors remain deferred for the vision reason). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UxThXGrq5VyXfUzcahTVw6
Gajesh2007
added a commit
that referenced
this pull request
Sep 3, 2026
…rdened, rebased for v0.8.16 (supersedes #654, addresses #653) (#791) * Resolve the activation reserve per serving set (measured per-model floors) The flat 5.5 GiB activation reserve (sized for gemma-4-26b's measured B=8 peak) is charged to every model, so gpt-oss-20b needs 20.0 GB to load — arithmetically unreachable on the catalog's own min_ram_gb=24 tier, which shows online and 429s every request (#653). Provider: add UnifiedMemoryCap.measuredActivationFloorsBytes (gpt-oss-20b: 3.5 GiB = worst measured B=8 peak, 3.20 compiled, plus slack), resolved max-over-set (advertised UNION resident), env raise-only above the resolved floor, threaded through the load gate, startup preload, doctor (serving-set verdict; per-alternative solo floors), free_for_load reporting, the measured post-load/post-bridge guards, runtime KV admission (GlobalKVCacheBudget actor setter, re-pushed on serving-set mutations), and engine KV grants plus the heartbeat budget clamp. Coordinator: mirror via servabilityModelActivationFloorsGB and a third version regime in servabilityActivationFloor(version, modelID); servabilityPerModelFloorMinVersion is a placeholder for the shipping release. needs on the 24 GB tier drops 20.0 -> 18.0 GB. Closes #653. * docs: activation reserve overhaul plan + 2026-08-30 measurement sweep The sweep re-baselines the July activation-peak measurements on the current engine: the 5.5 GiB default's basis (gemma qat-4bit B=8 = 5.05) now measures 2.28 GiB (saturated long-prompt envelope ~3.6); gpt-oss measures 2.63 (envelope ~2.7); the padded-weights estimate overshoots real residency by ~6 GiB on the 26B 8bit artifact. Plan covers per-model floors (PR #683 base), the qwen3.6 B=8 measurement, measured resident weights, and the default retune. * fix(coordinator): pin per-model floor gate to the 0.8.16 release train The cherry-picked #683 carried servabilityPerModelFloorMinVersion = "0.8.11" as a self-declared placeholder; 0.8.12–0.8.15 have since shipped holding the flat 5.5 GiB reserve, so as-is the coordinator would charge those binaries a 3.5 GiB gpt-oss floor they don't hold (fail-open, but wrong). Gate at 0.8.16 with the release-coupling contract documented at the constant, and pin 0.8.15 as the last flat-floor release in the regime test. Co-authored-by: Jeremy Joplin <jeremykjoplin@gmail.com> * docs: Phase 2 measurement results — qwen3.6 floor 4.0 candidate + 32 GB tier re-tier recommendation * fix(provider): serving-set reserve for the paged capacity decision + hybrid-safe paged cache mapping Two paged-path gaps behind the per-model floors (review findings): 1. The paged physical-capacity decision measured live headroom with the FLAT reserve while every other gate on the load path carves the serving-set floor — on a measured-only box there is a band where the flat-input liveLimit dips under the 1 GiB minimum pool and an explicit-paged slot 503s a load the relaxed gate admitted (admit-then-fail). Thread the caller's resolved reserve through makeProductionEngine/Build/ prepareProductionBackend and the slot factory; StandaloneServer keeps the flat default (nil). 2. The paged cache wiring subscripted the dense per-storage cache array with newCacheV2's MODEL layer index. Identity for Gemma/GPT-OSS; on a hybrid trunk (Qwen3.5/3.6: 40 model layers, 10 attending) it is an Index-out-of-range at engine build — latent today only because supportsPagedKV=false vetoes paged for recurrent models upstream. Map model index → storage slot so the flag can flip without a crash. Co-authored-by: Jeremy Joplin <jeremykjoplin@gmail.com> * fix(provider): close the review races in serving-set reserve resolution Four adversarial-review findings on the per-model floor machinery: 1. In-flight loads join the reserve basis (advertised ∪ resident ∪ modelsLoading): a hard-swap could drop a high-floor model from advertisedModels while its load was still running — in neither set, a relax computed in that window sized admission and grants below the reserve the model decodes under once installed (OOM direction). 2. Reserve pushes are epoch-stamped and the budget discards stale ones: cross-actor jobs from different tasks are not FIFO, so a relax computed before a raise could be DELIVERED after it and stand. Epochs increment under ProviderLoop's isolation = true mutation order. Regression test added. 3. Shrink paths regrow survivors: failed-self-test retirement now un-advertises BEFORE unloading (so unload's refresh-then-regrow runs against the shrunken set) and regrows after a cold retirement; dropAdvertisedBuild regrows after its relax. Grant clamps are min(granted, current), so nothing else ever hands the difference back (under-grant direction — capacity loss, not OOM). 4. Doctor's serving-set basis mirrors the daemon's advertise chokepoints (memory-filtered scan + supported families) instead of the raw scan, so a cached too-large or unadapted model cannot falsely FAIL a tier the daemon serves at a lower floor. The unfiltered list stays for target diagnosis. Co-authored-by: Jeremy Joplin <jeremykjoplin@gmail.com> * fix(provider): reserve re-push at load transitions + retirement tombstone Round-2 adversarial-review findings: 1. ensureModelLoaded re-pushes the reserve right after modelsLoading.insert — the insert sits after the specDecPreparation suspension, so a mid-await hard swap could drop the model from advertisedModels and push a relax while it was in neither basis set; the budget then admitted against a reserve the resuming load re-pins (OOM direction). The failed-load path symmetrically relaxes and regrows survivors — a dead high-floor id otherwise pinned the budget and grants forever (capacity-loss direction). 2. Failed-self-test retirement holds a retiringModels tombstone across its unload drain: with the slot still resident, a concurrent same-id prefetch reports .alreadyAvailable and its verified-insert re-advertised the failed build, undoing the fail-closed removal. The coordinator un-advertise also moves BEFORE the drain — leaving it after kept routing new requests at a slot fastAdmissionReject accepts, parking them behind the drain for a post-accept 404. Co-authored-by: Jeremy Joplin <jeremykjoplin@gmail.com> * fix(provider): hold the load gate through failed-load cleanup + seal the retirement/prefetch races Round-3 adversarial-review findings: 1. The failed-load catch released isLoadingAny/modelsLoading before its awaits — a new same-id loader could start mid-cleanup and place its own pending-load reservation (shared key, unconditional overwrite), which the old catch's release then DELETED, blinding KV admission to the incoming weights; the early unlock also let another model's weights go resident before the survivor regrow (sized from modelSlots only). Cleanup now runs release → clearCache → marker removal → refresh → regrow entirely under the held gate, unlocking last. 2. applyVerifiedPrefetch re-checks the retirement tombstone after each suspension (the pre-insert reserve push and the post-insert refresh) — retirement interleaving in either await could otherwise be undone by the resumed insert or the client announcement. The refused pre-insert path also re-refreshes so its phantom raise doesn't dangle. 3. Retirement waits out a drain another task owns (waitForModelUnload) before dropping its tombstone, and the forceReconnect moves back AFTER the drain: reconnecting first cancels every in-flight request on every model (cancelAllInflight on the disconnect event). The client store removal stays pre-drain so any reconnect during it registers without the failed model; the residual routed-404 window is bounded and absorbed by dispatch retries. Co-authored-by: Jeremy Joplin <jeremykjoplin@gmail.com> * feat(provider,coordinator): measured activation floor for qwen3.6-35b (4.0 GiB) Adds qwen3.6-35b-a3b-vl-mtp-mxfp8 → 4.0 GiB to both mirrored floor tables (one commit, per the contract), from the 2026-08-30 sweep on the served artifact (hub 73a03825, stock v2 = production's live route): non-KV transient saturates ~3.3 GiB across 500→8k-token B=8 cells (the raw peak-over-resident growth is contiguous-cell KV, separately accounted by the KV budget) + ~0.4 inline-MTP allowance + slack. Basis notes sit on both table entries because gpt-oss's 3.5 uses the July raw-500 convention. Effect: qwen3.6 needs drop 30.3 → 28.8 GB. That widens the 36 GB tier comfortably; the catalog's 32 GB tier remains physically marginal (<4 GiB left for macOS at the cap) — recommend min_ram_gb 32 → 36 as a separate ops/catalog change, per docs/reports/2026-08-30-activation-floor-measurements.md. Co-authored-by: Jeremy Joplin <jeremykjoplin@gmail.com> * fix(provider): ABA-proof fail-closed retirement against prefetch verification Round-4 adversarial-review finding (wrong-results direction): the prefetch's detached scan/hash suspension is long enough for an ENTIRE retirement — tombstone insert AND removal — to complete inside it, after which the resumed prefetch observed no tombstone, re-advertised the self-test-failed build, and a routed request could reload and serve it without re-passing the self-test. The durable mark is the failed WEIGHT HASH (failedSelfTestHashes, set at retirement): the same bytes that failed are refused at verification no matter how the suspensions interleave; a build with different bytes clears the record and gets its chance. The announce block also re-validates after its two client awaits and UNDOES the client-store add if retirement removed the local advertisement mid-announce — otherwise the post-drain reconnect registered a build the loop no longer serves (persistent false routing). Co-authored-by: Jeremy Joplin <jeremykjoplin@gmail.com> * fix(provider): durable self-test record at every prefetch guard + registration convergence on undo Round-5 adversarial-review findings: 1. The durable failed-self-test record was checked only at the FIRST guard — a retirement completing entirely inside the reserve-push suspension set the record after that check had passed, and the later guards consulted only the ephemeral tombstone. A shared helper (selfTestRecordRefuses) now runs at all three guards. The record is also ALWAYS set at retirement ("" sentinel when no reliable hash is known — the scanner keeps a previous hash on recompute failure, so a stale H1 could be recorded while H2 actually failed; the sentinel refuses every same-id build until restart, fail-closed), and a different-hash clear happens only at the actual advertise point. 2. The announce-undo could not retract a registration the reconnect loop had already encoded from the transiently-stale client store — the coordinator kept routing the retired id until an unrelated reconnect. The undo path now forces one more reconnect so registered inventory converges on the corrected store (rare double-race; bounded, correctness-restoring). Also aligns the measurement report's MTP allowance wording (~0.4) with the floor-table comment. Co-authored-by: Jeremy Joplin <jeremykjoplin@gmail.com> * fix(provider): slot-bound hash for the failed-self-test record Round-6 residue: the scanner maps (liveModelHashes/modelHashes) keep a previous value when recomputation fails, so retirement could record a stale H1 while H2's bytes actually loaded and failed — and a later verified H2 cleared the record and re-advertised the failed bytes. Record the slot's own verified binding (cacheEligibleWeightHash — the bytes the slot actually loaded) or the refuse-all sentinel; never the maps. Co-authored-by: Jeremy Joplin <jeremykjoplin@gmail.com> * feat(provider,coordinator): measured resident weights + qwen3.5 floor (Phase 3) Second over-reservation lever from the sweep: the scanner's disk×1.2 padding overshoots real MLX residency by ~6 GiB on the 26B 8bit artifact — enough that gemma-8bit's 36 GB tier was arithmetically unserveable (padded 31.3 + floor + 1 > 32.4 cap; measured 25.5 fits with margin). Adds the twin measured-residency tables (measuredResidentWeightsBytes / servabilityMeasuredResidentGiB, same 0.8.16 gate, same move-together mirror contract as the floors), threaded through: - provider: load gate, pending-load reservation, startup preload plan, doctor - coordinator: cold token-budget estimate AND the cold-load admit gate (reportedFreeForLoadAdmits + its four callers — without this, routing never offers the tier a cold load its providers can now take) Text-only models only (gpt-oss 11.5, gemma-4-26b/-8bit 25.5 GiB measured); vision-capable models keep the padded estimate until a vision-inclusive residency measurement exists. Also: qwen3.5-35b-a3b → 4.0 GiB activation floor in both tables (measured 2026-08-31; byte-identical B=8 profile to qwen3.6, pre-covers the family MTP default), and the plan records two deliberate resolutions — the flat default retune (5.5→~4.5) is DEFERRED until vision-inclusive measurement (its only beneficiaries are exactly the models a text-only bench cannot vouch for), and qwen3-vl-30b's real blocker is a missing CBv2 adapter (qwen3_vl_moe unsupported — the catalog entry is dark fleet-wide; ops decision, not memory arithmetic). Co-authored-by: Jeremy Joplin <jeremykjoplin@gmail.com> * fix(review): vision-gate the measured tables, padded pending-load, canonical docs Codex PR review (5 findings), all addressed: 1. qwen3.5/3.6 floor entries REMOVED (back to the 5.5 default): they are vision-capable and the tower transient rides the activation reserve — VisionMemoryGate charges decoded-media buffers + KV, and VisionTowerBudget only bounds the tower's N² allocation, so text-decode evidence alone must not lower the reserve. Their measured ~3.3 GiB text envelopes stand in the report as the baseline; entries return with a vision-inclusive measurement. 2. gemma-4-26b/-8bit residency entries REMOVED: the artifact carries vision_config (model_type gemma4), production loads the tower via VLMModelFactory, and the bench's forced-LLM 24.97 GiB under-counts it. The 36 GB tier unblock is gated on a provider-path residency measurement. Residency table = gpt-oss only (model_type gpt_oss — no VLM wrapper; the bench load IS the production path). 3. Pending-load reservation reverted to the PADDED estimate even for measured models — it guards the load transient, which steady post-load residency does not cover. 4. AGENTS.md memory-model section rewritten: the canonical instructions described the flat-reserve design and warned against exactly the per-model contract this branch ships; they now document the serving-set resolution, both mirror tables, the move-together rule, the vision gate, and the ≥4k-cell measurement convention. 5. Measurement report conclusions/follow-ups updated to final state (qwen3.6/3.5 measured; harness fix landed as mlx-swift-lm#129; qwen3-vl's blocker is the missing CBv2 adapter). Co-authored-by: Jeremy Joplin <jeremykjoplin@gmail.com> * chore(release): pin mlx-swift-lm to main (incl. bench fix #129) + bump version to 0.8.16 Submodule fe01df9 → 30da946: brings the merged bench-harness fix (mlx-swift-lm#129, paged-cache index mapping + supportsPagedKV veto) plus the engine work that landed on main since the old pin — #125 Qwen3-VL CBv2 DeepStack (engine side; provider wiring still gated off in EngineV2SupportedModels), #126 sorted-hint lane, #127 vision batch, #128 serving-correctness batch, #118 dense Qwen3.8 MTP artifacts. ProviderCore.version 0.8.13 → 0.8.16, honoring the release coupling documented at servabilityPerModelFloorMinVersion (this tree is the release train that ships the per-model tables). LatestProviderVersion moves with it — TestLatestProviderVersionMatchesProviderCore enforces the pairing; the constant only reaches operators when the coordinator deploys at release. Also hardens PagedKernelPreflightTests' hung-child deadline 0.5s → 2s: sibling tests run runtimeSmoke on real Metal kernels in parallel and the new engine's heavier compile load could pre-empt the child's startup past half a second (pid file never written → the kill assertion misread as a missing file). A rarer residue of the same timing class remains under tight isolation looping (~1-in-3 at 2s) — known-flake, same family as the KV-reclaimer wall-clock tests; full suite is green (2246/233). Co-authored-by: Jeremy Joplin <jeremykjoplin@gmail.com> * fix(review): retirement inference gate + 0.8.16 changelog + canonical citations Codex round-3 findings: 1. (P1) New inference is now rejected on BOTH resident-slot fast paths while a failed-self-test retirement drains: fastAdmissionReject returns an immediate reject (coordinator reroutes) and ensureModelLoaded throws a 503-mapped retiring error before the resident fast-return — the promised 404 only lands after the drain, and the fast paths were accepting work against a build that failed its serving-path self-test. Regression test + test seam added (RetirementGateTests). 2. (P2) CHANGELOG gains the v0.8.16 release-candidate entry: per-model floors + measured resident weights with the coordinator mirrors, the reserve race-hardening series, and the mlx-swift-lm pin advance to 30da946 with its engine PRs enumerated. 3. (P1) The plan's qwen3-vl provider-gate claims now cite the canonical code (EngineV2SupportedModels.swift:41 predicate; EngineV2Factory+Production.swift:656-region family switch). Co-authored-by: Jeremy Joplin <jeremykjoplin@gmail.com> * fix(review): retirement re-checks after every suspension + standalone measured weights + plan truth Codex round-4 findings: 1. (P1) The retirement gate is re-checked after EVERY actor suspension that precedes a resident-slot return, via a shared throwIfRetiring helper: ensureModelLoaded re-checks after the unloading-wait loop and after the loading-waiter resume (a parked request could otherwise resume past a tombstone inserted while it waited and submit to the failed build); fastAdmissionReject re-checks after both availableMemoryGb suspensions (no inference_accepted for a retiring model). 2. (P1) StandaloneServer's load gate uses the measured-residency weights figure too (loadGateWeightsGb) — darkbloom local's 24 GiB gpt-oss case was still being charged the disk×1.2 estimate. The FLOOR deliberately stays flat in standalone; the weights constant has no serving-set semantics to differ on. 3. (P2) The plan's Phase-3 status block now states the truth: both qwen floors deferred (tables ship gpt-oss only), qwen3.6's 32 GB tier does NOT fit at the 5.5 default (30.3 > 28.8 cap — re-tier to 36 REQUIRED), and gemma-8bit's 36 GB tier stays blocked pending the VLM-path residency measurement. Co-authored-by: Jeremy Joplin <jeremykjoplin@gmail.com> * fix(review): remaining retirement windows + doctor live basis + live preload requirements Codex round-5 findings: 1. (P1) throwIfRetiring at the two remaining post-suspension resident returns in ensureModelLoaded: after the specDecPreparation resume and at the load-gate (isLoadingAny) wait's resident return. 2. (P2) Doctor derives its serving-set floor basis from the daemon's OWN advertised list when the daemon is up and fresh — new optional advertised_models field in daemon-state (decode-tolerant), written from the live advertisedModels keys. CLI selection overrides (--all bypassing enabled_models, --model restricting) made any config-derived reconstruction wrong in both directions; the config-derived basis stays as the offline fallback. 3. (P2) Startup preload recomputes each candidate's requirement at its admission step (Dependencies.currentRequiredGb hook → live loadGateWeightsGb + current loadHeadroomGb): a fail-closed retirement mid-preload relaxes the serving-set floor, and a later candidate held its stale plan-time figure — skipping a model the refreshed gate admits. 4. (P2) The plan's Phase-2 step 3 is rewritten as resolved: the 4.0 qwen floor is a rejected-for-now candidate behind vision-inclusive measurement; the 32→36 re-tier is REQUIRED regardless. Co-authored-by: Jeremy Joplin <jeremykjoplin@gmail.com> * fix(review): padded admits everywhere — measured residency is post-load-only Codex round-6 findings: 1. (P1) Measured steady residency leaves EVERY admit-time gate: the load transient (shard staging) exceeds the post-load figure, which is exactly what the disk×1.2 padding was sized for — admitting a co-resident load against 11.5 GiB steady while staging peaks higher is the OOM direction. The provider-side measured-weights table/helper and all its wirings (load gate, startup preload, standalone, doctor) are REMOVED; the canonical measured values live in servabilityMeasuredResidentGiB (coordinator) and feed ONLY coldTokenBudgetEstimate — the POST-load token-budget arithmetic, where steady residency is physically correct (it converges to warm active_token_budget_max). reportedFreeForLoadAdmits reverts to the padded conversion for every binary and model, with a regression test pinning both directions (admit refuses at measured-fits/padded-doesn't; admits at padded-fits). The gemma-8bit 36 GB tier follow-up now requires a measured LOAD-PEAK through the provider's VLM path, not just steady residency. 2. (P2) fastAdmissionReject recomputes its requirement after the availableMemoryGb suspension — a concurrent verified prefetch can raise the serving-set floor during the hop (accepted-then-503 otherwise). 3+4. (P2) Measurement report finalized: the tier verdict marks the qwen floor entry deferred (shipped tables carry gpt-oss only), and the release-gate conclusion records the SHIPPED 0.8.16 state instead of the 0.8.11-placeholder/0.8.15-train caveat. CHANGELOG, AGENTS.md, and the plan reframed to the post-load-only design. Co-authored-by: Jeremy Joplin <jeremykjoplin@gmail.com> * fix(review): empty live advertised set is authoritative in doctor Codex round-7 (single P2): a fresh daemon that retired every model after failed startup self-tests legitimately advertises [] — the !isEmpty guard discarded that live value and reconstructed the serving set from config, crediting floors for models the daemon is not serving. Non-nil is now authoritative including empty. Co-authored-by: Jeremy Joplin <jeremykjoplin@gmail.com> * docs: reconcile plan and report with master post-rebase master now serves qwen3_vl_moe directly (#752, vision off M5 #781) and auto-activates embedded MTP heads (#777): the "dark fleet-wide / no CBv2 adapter" claims become a bench-side measurement gap (LLM-factory load cannot construct the wrapper; floor stays at the 5.5 default, which fits its 32 GB tier), and the qwen3.6 production route now includes MTP k=2 — the B=1-measured +0.4 GiB allowance applies to the live route (floors remain deferred for the vision reason). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UxThXGrq5VyXfUzcahTVw6 * fix(review): live headroom through eviction/retry waits, authoritative empty doctor set, capacity publish after reserve raise Codex review of the rebased head (f591ce7), six findings: 1. evictUntilAvailable takes the weights figure and re-resolves the requirement on every iteration (and after its clearCache retry) instead of comparing against a requirement captured before its suspensions — a concurrent verified prefetch can raise the serving-set floor mid-wait, and the stale lower figure admitted a load the post-load guard then rejected (accepted-then-503). The MTP admission after the loop resolves its target requirement live too. 2. fastAdmissionReject re-resolves the requirement after its retry memory sample as well (the second suspension was outside the earlier fix). 3. Doctor keeps a LIVE empty advertised set authoritative all the way into the verdict: DoctorRunner passes [] through (only the offline reconstruction maps empty to nil), and ModelFitDiagnostic resolves an empty declared set at the default floor — the target cannot be joining a set the daemon does not advertise — instead of crediting the retired target its solo floor. Regression test added. 4. applyVerifiedPrefetch re-slices resident engines and refreshes aggregate capacity after the reserve raise and BEFORE announcing the enlarged set, so the coordinator never routes against a token budget the tightened shared KV gate rejects until the next periodic tick. 5. Plan: canonical file:line citations restored for the Qwen3-VL production path (supported-family predicate, wrapper extraction, construction switch), and the Phase-1 release-coupling item marked completed (0.8.16 in-tree, pairing test). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UxThXGrq5VyXfUzcahTVw6 * fix(review): resolve the floor after every admission suspension, preflight advertise-only reserve raises under the re-slice gate - evictUntilAvailable samples memory first and resolves the requirement after that suspension: its zero-iteration exit compared against a floor captured before the await, so a concurrent raise could admit a load the post-load guard then rejects. - admitSpecDecIfMemoryAllows takes the target's weight basis and resolves the headroom after its own memory sample (same class). - applyVerifiedPrefetch preflights the serving-set reserve raise against the resident slots' serviceability floor and refuses the advertisement when the raise would strand a survivor below it, mirroring the load path's newcomer refusal. The preflight -> push -> insert -> re-slice -> capacity-publish sequence holds the re-slice gate like a load does, so a load cannot size itself against the pre-raise budget in between. - Capacity republished after the drop and cold-retire relaxes so free_for_load_gb never lags the floor by a capacity tick; retiring rejects tagged slot_state rather than memory_cap; doctor's offline serving-set basis mirrors the daemon's runtime-eligibility filter. Co-authored-by: Jeremy Joplin <jeremykjoplin@gmail.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UxThXGrq5VyXfUzcahTVw6 * fix(review): retry reserve-refused desired builds, drain the box before the retirement reconnect - A verified prefetch refused by the reserve-raise preflight now schedules a retry through the desired-build backoff policy (the re-run is a hash pass — the bytes are on disk). `.verified` still goes out for the attempt and the coordinator deduplicates unchanged desired_models pushes, so without a local retry nothing would ever re-offer the build, even after an unload frees the room. The scheduler is factored out of the transient-failure handler and shared. - The failed-self-test retirement waits for box-wide in-flight work to drain (bounded by the shutdown drain budget) before forcing the reconnect that communicates the removal: closing the socket cancels every in-flight request on the box, not just the retired model's. - Plan doc: the paged-cache subscript is fixed on this branch (cite the mapping, not the removed line); the vision-tower transient rides the activation reserve — VisionMemoryGate covers media buffers and KV, not activations — which is the reason vision-capable floors stay deferred. - Regression test: a scripted resident gpt-oss slot on a 32 GiB box where the resident floor leaves a serviceable grant but an unmeasured build's raise would not; the desired build is refused, one retry is pending, and it advertises once the box has room. Co-authored-by: Jeremy Joplin <jeremykjoplin@gmail.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UxThXGrq5VyXfUzcahTVw6 * docs(plan): gemma residency stays deferred behind the VLM-path measurement The sequencing bullet still listed gemma-qat4 as a text-only artifact ready for measured residency, contradicting the plan's own vision gate: its checkpoint declares vision_config, the provider loads it through the VLM factory, and the text-only 13.48 GiB figure undercounts the tower. Only gpt-oss is ready for Phase 3a. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UxThXGrq5VyXfUzcahTVw6 * fix(review): detach and coalesce the retirement reconnect, harden the preflight test against slow runners - The startup preloader awaits retirement inline, so the box-wide drain wait added for the reconnect would have stalled every remaining preload candidate behind a busy box for up to the drain budget. The reconnect is now a single detached task: un-advertise, unload, and the tombstone stay inline; one background re-registration per burst of retirements fires after in-flight work drains (the client store already excludes every retired id by then), skipped on shutdown. - Load-failure rejects capture the retiring flag before the cleanup awaits, so a retirement completing during them cannot misfile the reject as memory_cap. - ReserveRaisePreflightTests asserted an exact prefetch count that a slow CI runner violated (the first retry fired before the observation). Assert the refusal invariants that hold at any point of the retry cycle and use a retry budget the tight phase cannot exhaust. Co-authored-by: Jeremy Joplin <jeremykjoplin@gmail.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UxThXGrq5VyXfUzcahTVw6 * fix(review): discard capacity snapshots built across a reserve move, barrier admission across the retirement reconnect - updateAggregateCapacity captures the activation-reserve epoch at entry and discards its snapshot if a push landed across its actor hops: a refresh suspended inside the engine summary could otherwise resume after a verified prefetch raised the floor and overwrite the newer publish with pre-raise slot budgets and free_for_load_gb. - The post-retirement reconnect raises an admission barrier on the actor before closing the socket and re-checks in-flight work after the drain's last observation, so a routed request cannot be admitted in the hop only to be cancelled by the disconnect; the barrier lifts on the new session's .connected. Shutdown cancels a pending reconnect task. - A retired, un-advertised id with a standing failed-hash record is fast-rejected (slot_state) until the register lands, instead of accept-then-404 at the advertised guard; the same predicate tags both rejection sites. - Tests: the prefetch advertise test waits on the client store (the local insert now precedes the client add by the re-slice hops); the preflight test requires the reserve env override to be unset. Co-authored-by: Jeremy Joplin <jeremykjoplin@gmail.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UxThXGrq5VyXfUzcahTVw6 * fix(review): re-validate the load gate before allocation, serialize advertise-raises behind in-flight loads, consult the durable record in the shared guard - ensureModelLoaded re-runs the padded-weight gate after the last suspension before allocation (pending-load reservation, weight hashing, the pre-load hook): the serving-set floor can move across any of them and the pending reservation only fences competing KV grants. Refuse before shard staging rather than let the post-load guard unload an already-overcommitted transient. - applyVerifiedPrefetch defers an advertise-only reserve raise while any load sits between admission and install (modelsLoading is set before the gate), through the desired-build backoff — the primary defense; the allocation-time gate is the backstop. - throwIfRetiring and fastAdmissionReject's post-sample checks consult isRefusedByRetirement (tombstone OR durable failed-hash record): a suspension spanning an entire retirement could otherwise resume into the failed build with its pre-retirement modelInfo. - updateAggregateCapacity recomputes (bounded, three attempts) when the reserve epoch moves across its hops instead of returning without a publish — not every push site publishes a replacement snapshot. - Measurements report: paged-cache mapping marked fixed with its current citation; the text-only caveat states that the vision-tower transient rides the activation reserve and is not contained by VisionMemoryGate. Co-authored-by: Jeremy Joplin <jeremykjoplin@gmail.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UxThXGrq5VyXfUzcahTVw6 * fix(review): add the load's own reservation back in the allocation gate, pin pending advertisements into the reserve basis - The allocation-time backstop compared the free-for-load figure — which nets out every ledger entry, including this load's own pending reservation placed just above — against the full padded requirement, a double count that refused every load whose padded weights exceed the headroom (the 24 GB gpt-oss-20b tier: 18.0 required, ~4.6 netted). The comparison now goes through ModelLoadAdmission.fitsAtAllocation, which adds the load's own reservation back; regression test pins the tier arithmetic in both directions. - applyVerifiedPrefetch re-checks modelsLoading AFTER the survivor preflight (it awaits each bridge's grant, and a load admitted during those hops is in neither modelSlots nor the preflight) and pins the id into pendingAdvertise before the reserve push, so a load admitted during the push's own suspension already resolves its gate and fleet budget against the raised floor. Cleared when the id joins advertisedModels or on the tombstone refusal. Co-authored-by: Jeremy Joplin <jeremykjoplin@gmail.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UxThXGrq5VyXfUzcahTVw6 * fix(review): cover the retained MTP assistant in the allocation-time requirement The pending reservation added back by the allocation gate covers the padded target PLUS a separately staged assistant, but the requirement it was compared against covered the target alone, so the assistant's bytes dropped out of the check: a box where the target fits but target + assistant no longer does would pass and let the later assistant allocation cross the unified-memory cap. The requirement now uses the gate's weight basis with extraWeightBytes included — the same basis as the reservation. Regression test pins the asymmetry. Co-authored-by: Jeremy Joplin <jeremykjoplin@gmail.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UxThXGrq5VyXfUzcahTVw6 * fix(review): re-offer capacity-deferred desired builds on the capacity change itself A verified prefetch deferred for a capacity reason (the reserve raise would strand a resident slot, or a load was in flight) was retried only through the desired-build backoff — about 18 minutes — while the idle unload that typically frees the room defaults to 60 minutes; once the budget expired the provider stayed on the superseded build until the next desired_models push (the coordinator deduplicates unchanged state). Deferred ids are now remembered (reserveDeferredPrefetches) and re-offered with a fresh budget by the capacity-change events themselves: a slot unload and a load finishing (installed or failed). The backoff stays as the near-term retry; a still-tight box simply defers again. Cleared when the build advertises or leaves the desired set. Regression test: refused → re-offered by the resident slot's unload inside seconds with a 600 s backoff configured. Co-authored-by: Jeremy Joplin <jeremykjoplin@gmail.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UxThXGrq5VyXfUzcahTVw6 * fix(review): carve the per-model reserve in standalone serving, re-offer explicit prefetches deferred for capacity - StandaloneServer resolves the activation reserve over its serving set (configured models ∪ resident slots ∪ in-flight loads) and passes it to every consumer the network provider threads it through: the KV budget actor, the fleet KV budget behind engine grants, the load gate's headroom, the paged capacity decision, and both post-load probes. A tight box the coordinator path admits (24 GB gpt-oss) is admitted by `darkbloom local` too. Regression test pins the gpt-oss grant against the measured floor rather than the flat default; the existing re-slice test now expects the lone survivor's own floor after an eviction. - Capacity-deferred prefetches are remembered whether desired OR explicit (`prefetch_model`): the coordinator ignores the `.verified` status the refused attempt emits, so the capacity-change re-offer is the only path that ever advertises the build. Ids are forgotten before each re-run — a re-refusal re-remembers, an unwanted build drops out. Co-authored-by: Jeremy Joplin <jeremykjoplin@gmail.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UxThXGrq5VyXfUzcahTVw6 * fix(review): re-push the standalone reserve on setModels, pass it to the MTP fallback probe - StandaloneServer.setModels changes the serving set, which is part of the activation-reserve basis: it now re-pushes the resolved reserve into the KV budget actor and re-slices the resident grants under the new floor before a higher-floor model can be exposed (a load in flight re-slices at its own install, so the regrow is skipped then). Regression test: a gpt-oss-only server's grant shrinks to the raised-floor budget when an unmeasured model is added. - The target-only fallback probe after a failed MTP build now takes the resolved reserve like the first probe (the earlier pattern edit missed its indentation), so a tight box is not unloaded against the flat default it just cleared. Co-authored-by: Jeremy Joplin <jeremykjoplin@gmail.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UxThXGrq5VyXfUzcahTVw6 * fix(review): run the setModels re-slice when the in-flight load finishes, including on failure setModels skipped the survivors' re-slice while a load was in flight, assuming the load's install would re-slice under the updated reserve. A load that fails before install restores the PREVIOUS grants (sized under the old floor) and installs nothing, so nothing re-sized them while the KV budget actor already enforced the new floor. The re-slice is now deferred (resliceDeferredBehindLoad) and run from both completion paths of the standalone load — after a successful install (idempotent) and after the failure cleanup — via runDeferredResliceIfNeeded. Regression test drives the deferral through the load-marker seams and asserts the resident grant ends under the raised floor. Co-authored-by: Jeremy Joplin <jeremykjoplin@gmail.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UxThXGrq5VyXfUzcahTVw6 * fix(review): serialize and preflight standalone setModels, re-validate the standalone load gate before allocation - setModels is serialized behind any in-flight load: that load passed its gate against the current floor, so the whole list update waits (pendingModelsUpdate) and applies on either completion path of the load — failure included, where the restore-on-throw has just put the old grants back. - The update is preflighted against the resident slots' serviceability floor (reserveKeepsSurvivorsServiceable, the load path's newcomer check applied to a raise with no newcomer); a stranding raise is refused and the current list kept. setModels returns whether it was accepted. - The standalone load re-validates the padded gate after its last suspension before allocation, through the same fitsAtAllocation helper (own reservation added back, retained assistant included) — the backstop behind the serialization. - Tests: deferral behind a load (updated), refusal of a stranding raise on a single tight slot (new). Co-authored-by: Jeremy Joplin <jeremykjoplin@gmail.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UxThXGrq5VyXfUzcahTVw6 * fix(review): hold the standalone load gate across setModels, epoch-stamp its reserve pushes - applyModels now runs under the standalone load gate (isLoadingAny), held from the serviceability preflight through the reserve push and the re-slice: the preflight suspends on each resident bridge, and a cold load entering meanwhile would pass its own gate under the old floor and install a slot the preflight never saw. Loads and further setModels calls park on the gate exactly as they park behind a load; a list parked while an apply held the gate is applied right after it. - Every standalone reserve push carries a monotonic epoch (as the provider's do), so a stale lower-floor push can never land after a newer one at the KV budget actor even if delivery reorders. - Test seams hold and release the same gate the load path does. Co-authored-by: Jeremy Joplin <jeremykjoplin@gmail.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UxThXGrq5VyXfUzcahTVw6 * fix(review): keep a deferral whose attempt is still finishing, drain parked setModels, re-push the standalone reserve at every basis change - retryReserveDeferredPrefetches skips an id whose prefetch attempt is still in flight in the coordinator (ModelPrefetchCoordinator.isInFlight: the tail after onVerified returns, before the terminal status emits) — a re-run then would coalesce into that attempt and never reach applyVerifiedPrefetch again. The deferral and the desired backoff are kept for the next capacity change instead. - setModels drains a list parked behind its own apply (the load-completion drains cannot be relied on: no load may ever come). - The standalone eviction path pushes the relaxed reserve before the survivors grow into it; both standalone load-marker transitions push the reserve (the marker joins/leaves the basis); the standalone load re-validates its model's membership right before marking itself loading, so a stale modelInfo removed by setModels cannot re-enter the basis. Co-authored-by: Jeremy Joplin <jeremykjoplin@gmail.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UxThXGrq5VyXfUzcahTVw6 * docs(reserve): correct the 24 GB tier claim, derive the tier arithmetic in a test, fix stale standalone comments The CHANGELOG, the floors-table comment and the PR body claimed the measured gpt-oss floor makes the catalog's 24 GB tier reachable. The live gate resolves usable = free-plus-inactive − max(memory_reserve_gb, total − 0.9·total); with the shipped 4 GiB reserve default the 24 GB tier still needs ~22 GB free of 24 at the 18.0 GB requirement, and the #653 reporters measured 12.3 and 16.1 GB usable. What the floor does close is the 32 GB flap band (the gate needs ~22 GB free instead of ~24). The remaining 24 GB levers — the admit-time disk×1.2 padding and the small-box reserve default — are recorded as follow-ups. A DiagnosticsTests case now derives usable from total, the reserve default and a free-plus-inactive figure through the real gate arithmetic instead of taking usable as an input, and pins both tiers' verdicts. Two comments that still described StandaloneServer as keeping the flat default are corrected. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UxThXGrq5VyXfUzcahTVw6 * fix(review): wake a kept deferral once its prefetch attempt finishes; re-validate the gpt-oss floor on the shipped pins - A capacity-deferred prefetch whose attempt was still finishing inside the coordinator kept its deferral but had no wake-up: explicit prefetch_model builds have no backoff timer and the capacity-change callers may never fire again. A one-shot wake-up per id now waits (off the actor) for the coordinator's attempt to reach terminal cleanup and re-offers the deferred builds. - The measurements report's 32 GB conclusion assumed the 0.9×physical cap; the effective ceiling is min(0.9×physical, physical − memory_reserve_gb) = 28.0 GiB under the 4 GiB default, so a qwen3.6 floor near 3.5 GiB does not fit either — the 36 GB re-tier is required regardless of any floor change. - The gpt-oss-20b B=8 cells were re-run with BenchCBv2 built at exactly the pins this PR ships (mlx-swift-lm 30da946 incl. the #126 MoE dispatch change, mlx-swift 6b0505c): 4k-prompt peak−active 4.17 GiB (identical to August), 500-token 2.27 GiB; the non-KV envelope stays ~2.7 GiB, so the 3.5 GiB floor holds with ~0.8 GiB margin. The raw reports are committed under docs/reports/raw/ (the August raw files were lost with their worktree). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UxThXGrq5VyXfUzcahTVw6 * fix(load): refuse hopeless evictions up front; fold in the design review's pre-merge corrections - Eviction feasibility (#653's 32 GB report: "a request for a model I can't serve killed the one I could"): before the first eviction, the load path asks whether evicting EVERY idle model plus the reclaimable buffer cache could reach the requirement at all; when it cannot, the load is refused without evicting. The fast pre-accept check applies the same bound so the coordinator reroutes instead of accepting a request that would only fail after the same check. Pure decision ModelLoadAdmission.evictionCanReach, unit-tested on the report's numbers. - Wording: the floor narrows the 32 GB flap band by ~2 GB (both #653 samples get the same verdict under either floor) rather than closing it; the plan no longer claims the 24 GB tier is unblocked; the consumer hardware-requirements doc describes the reserve instead of a 2 GB headroom; an orphaned qwen basis comment (no such table entry) is removed from the coordinator floors table. - Re-validation: the 4k batch-8 cell re-run with production's prefill construction (chunk 512, solo stripe 2048) measures the same 4.17 GiB peak-over-active; raw report committed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UxThXGrq5VyXfUzcahTVw6 --------- Co-authored-by: Jeremy Joplin <jeremykjoplin@gmail.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This branch had an error being deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The weights ship the MTP policy. Eight commits (three feature, four review fixups, one consolidated report) that converge on one rule: a checkpoint drafts under default config iff it embeds its own head (
mtplx_mtpin config.json) and is Qwen-family (hardcoded for now):fix(provider): restore + extend auto MTP to the Qwen 3.5 family—.autore-broadened from feat(provider): integrate Qwen3.8-27B full VLM #700's single pinned id to theqwen3_5/qwen3_5_moefamily.refactor(provider): embedded MTP only— Qwen3.8-27B moves from the separately published revision-pinned external head to the embedded artifact (same shape as 3.6-35B and the 9B). Deleted:StandaloneQwen38MTPResolver, theautomaticTargetModelIDpin, the resolver test suite (−429/+62).docs(provider): beta mtp feature text+test(provider): extraction suite colocates the metallib— review fixups:darkbloom betanow describes the embedded-gated policy, andQwenVLMTargetExtractionTestsself-heals its metallib —QwenVLMTargetExtractionTestswas the one MLX-touching suite withoutLiveInferenceFixtures.ensureMetallibColocated()(verified by deleting the manual copy, 17/17).fix(provider): tri-state inline probe(4299d78) — review fixup:inlineDeclarationProbedistinguishes declared / absent / undeterminable so a declared-but-uninspectable head surfacesinline_artifact_invalidinstead ofconfig_disabled; operator guides updated for the embedded policy (0b8a3b5).docs(reports)(da41944) — the Qwen3.5-9B validation + MTP report consolidated from docs(reports): Qwen3.5-9B dense validation + native inline-MTP head (1.51x decode) #774 (closed in favor of this PR), including the OPEN greedy-token-parity validation item that gates registry rollout of the embedded artifacts.feat(provider): auto MTP activates only embedded Qwen heads—.autorequires the embedded declaration, not just the family:enablesMTP(forModelType:embeddedArtifactDeclared:). A Qwen checkpoint without an embedded head is never asked about (no catalog lookup, no prefetch — plain config-disabled). Separately published assistants andmtp_drafter_pathoverrides require explicitmtp_mode = "on"; catalog prewarm runs only under"on"(auto never consults the catalog).Net effect: publishing a checkpoint with an embedded head is what turns MTP on for it — no per-model code, no registry spec-dec metadata, no id pins. The funnel keeps full authority (inline inspection, kill switch); a declared-but-invalid head still falls back target-only with a recorded reason.
Verification (live, M4 Max, default config — no
mtp_modekey — on the final binary).autoUnit tests: 74/74 across the MTP suites;
targetPolicypins the full (mode × family × embedded) matrix incl. normalization and non-family exclusion.Before / After
flowchart LR subgraph Before["Before — policy in code"] A1[slot load] --> B1{"id == pinned Qwen3.8?"} B1 -->|yes| C1[external head @ exact revision<br/>StandaloneQwen38MTPResolver] B1 -->|no| D1[target-only<br/>every new model = code change] end subgraph After["After — policy in the weights"] A2[slot load] --> B2{"Qwen family AND<br/>checkpoint declares mtplx_mtp?"} B2 -->|yes| C2[funnel inspects embedded head → drafts<br/>invalid head → target-only + reason] B2 -->|no| D2[target-only, no catalog/prefetch<br/>external assistants need mtp_mode = on] endflowchart LR subgraph Code M["MTPMode.enablesMTP(modelType, embeddedArtifactDeclared)"] --> P[ProviderLoop+MTP prepare<br/>declares check via SpecDecStore] M --> S[StandaloneServer+MTP prepare<br/>funnel-only] PW[prewarmSpecDecCatalog<br/>now on-mode only] -.-> F P --> F[SpecDecArtifactFunnel<br/>inline inspection · kill switch] S --> F F --> E[CBv2 engine + drafter] endRollout
EigenLabs/Qwen3.5-9B-MLX-4bit-mtpandEigenLabs/Qwen3.8-27B-4bit-mtp(27B = target@301e9e27+ head@329261c5, byte-identical tensors, verified by round-trip download + serve). The 27B was never registered on the official darkbloom registry, so this PR lands cleanly first; register the embedded repos after.EigenLabs/Qwen3.8-27B-4bitwith the cached external head no longer auto-drafts — switch to the embedded repo, or pinmtp_drafter_path+mtp_mode = "on"..autodraft exactly for the embedded checkpoints they serve;mtp_mode = "off"untouched;"on"keeps catalog/external-assistant behavior for operators who need it.Related: #774 (9B validation + embedded-artifact build conventions).