fix: allow Swift providers without Python privacy caps by hankbobtheresearchoor · Pull Request #118 · Layr-Labs/d-inference

hankbobtheresearchoor · 2026-05-03T06:00:28Z

Summary

Fixes the Swift provider private-text routability gate uncovered during cross-device E2E profiling against #110.

Swift providers report backend == "mlx-swift" and correctly set Python-specific privacy capability fields to false because they do not embed a Python runtime:

python_runtime_locked=false
dangerous_modules_blocked=false

The coordinator was still requiring those fields for all private-text backends, which made Swift providers connect, pass runtime manifest verification, pass attestation challenge verification, and appear in /v1/providers/attestation — but remain absent from /v1/models and unroutable.

This keeps the Python runtime checks for the legacy inprocess-mlx backend and uses the Swift-relevant invariants for mlx-swift:

in-process text backend
no proxy
anti-debug enabled
core dumps disabled
env scrubbed
runtime manifest checked
coordinator-verified SIP

Validation

go test ./internal/registry -run 'SwiftProvider|LegacyMLXProviderRequiresPythonPrivacyCaps|ProviderWithoutManifestCheck|ProviderPartialPrivacyCaps' -v
go test ./internal/registry ./internal/api -v
git diff --check

Context

This is targeted at PR #110 (swift-provider). The issue reproduced on the two-machine rig with the Swift provider on the Mac Mini: /health showed a connected provider and attestation/runtime verification passed, but /v1/models was empty until the coordinator privacy gate became backend-aware.

Swift providers do not embed a Python runtime, so python_runtime_locked and dangerous_modules_blocked are not meaningful privacy capabilities for backend=mlx-swift. Preserve those checks for the legacy inprocess-mlx backend while requiring Swift providers to keep the in-process/no-proxy, anti-debug, core-dump, and env-scrubbed invariants.

vercel · 2026-05-03T06:00:33Z

@hankbobtheresearchoor is attempting to deploy a commit to the EigenLabs Team on Vercel.

A member of the Team first needs to authorize it.

…dges (#118) Re-check ready after account-cutover within the same round so max_rounds=1 works. Abort on no-progress. Concurrent disjoint allowlists + charged scope.

…dges (#118) Re-check ready after account-cutover within the same round so max_rounds=1 works. Abort on no-progress. Serialize hook tests; harden steal assertions.

…p version to 0.8.16 Submodule fe01df9 → 30da946: brings the merged bench-harness fix (mlx-swift-lm#129, paged-cache index mapping + supportsPagedKV veto) plus the engine work that landed on main since the old pin — #125 Qwen3-VL CBv2 DeepStack (engine side; provider wiring still gated off in EngineV2SupportedModels), #126 sorted-hint lane, #127 vision batch, #128 serving-correctness batch, #118 dense Qwen3.8 MTP artifacts. ProviderCore.version 0.8.13 → 0.8.16, honoring the release coupling documented at servabilityPerModelFloorMinVersion (this tree is the release train that ships the per-model tables). LatestProviderVersion moves with it — TestLatestProviderVersionMatchesProviderCore enforces the pairing; the constant only reaches operators when the coordinator deploys at release. Also hardens PagedKernelPreflightTests' hung-child deadline 0.5s → 2s: sibling tests run runtimeSmoke on real Metal kernels in parallel and the new engine's heavier compile load could pre-empt the child's startup past half a second (pid file never written → the kill assertion misread as a missing file). A rarer residue of the same timing class remains under tight isolation looping (~1-in-3 at 2s) — known-flake, same family as the KV-reclaimer wall-clock tests; full suite is green (2246/233). Co-authored-by: Jeremy Joplin <jeremykjoplin@gmail.com>

…rdened, rebased for v0.8.16 (supersedes #654, addresses #653) (#791) * Resolve the activation reserve per serving set (measured per-model floors) The flat 5.5 GiB activation reserve (sized for gemma-4-26b's measured B=8 peak) is charged to every model, so gpt-oss-20b needs 20.0 GB to load — arithmetically unreachable on the catalog's own min_ram_gb=24 tier, which shows online and 429s every request (#653). Provider: add UnifiedMemoryCap.measuredActivationFloorsBytes (gpt-oss-20b: 3.5 GiB = worst measured B=8 peak, 3.20 compiled, plus slack), resolved max-over-set (advertised UNION resident), env raise-only above the resolved floor, threaded through the load gate, startup preload, doctor (serving-set verdict; per-alternative solo floors), free_for_load reporting, the measured post-load/post-bridge guards, runtime KV admission (GlobalKVCacheBudget actor setter, re-pushed on serving-set mutations), and engine KV grants plus the heartbeat budget clamp. Coordinator: mirror via servabilityModelActivationFloorsGB and a third version regime in servabilityActivationFloor(version, modelID); servabilityPerModelFloorMinVersion is a placeholder for the shipping release. needs on the 24 GB tier drops 20.0 -> 18.0 GB. Closes #653. * docs: activation reserve overhaul plan + 2026-08-30 measurement sweep The sweep re-baselines the July activation-peak measurements on the current engine: the 5.5 GiB default's basis (gemma qat-4bit B=8 = 5.05) now measures 2.28 GiB (saturated long-prompt envelope ~3.6); gpt-oss measures 2.63 (envelope ~2.7); the padded-weights estimate overshoots real residency by ~6 GiB on the 26B 8bit artifact. Plan covers per-model floors (PR #683 base), the qwen3.6 B=8 measurement, measured resident weights, and the default retune. * fix(coordinator): pin per-model floor gate to the 0.8.16 release train The cherry-picked #683 carried servabilityPerModelFloorMinVersion = "0.8.11" as a self-declared placeholder; 0.8.12–0.8.15 have since shipped holding the flat 5.5 GiB reserve, so as-is the coordinator would charge those binaries a 3.5 GiB gpt-oss floor they don't hold (fail-open, but wrong). Gate at 0.8.16 with the release-coupling contract documented at the constant, and pin 0.8.15 as the last flat-floor release in the regime test. Co-authored-by: Jeremy Joplin <jeremykjoplin@gmail.com> * docs: Phase 2 measurement results — qwen3.6 floor 4.0 candidate + 32 GB tier re-tier recommendation * fix(provider): serving-set reserve for the paged capacity decision + hybrid-safe paged cache mapping Two paged-path gaps behind the per-model floors (review findings): 1. The paged physical-capacity decision measured live headroom with the FLAT reserve while every other gate on the load path carves the serving-set floor — on a measured-only box there is a band where the flat-input liveLimit dips under the 1 GiB minimum pool and an explicit-paged slot 503s a load the relaxed gate admitted (admit-then-fail). Thread the caller's resolved reserve through makeProductionEngine/Build/ prepareProductionBackend and the slot factory; StandaloneServer keeps the flat default (nil). 2. The paged cache wiring subscripted the dense per-storage cache array with newCacheV2's MODEL layer index. Identity for Gemma/GPT-OSS; on a hybrid trunk (Qwen3.5/3.6: 40 model layers, 10 attending) it is an Index-out-of-range at engine build — latent today only because supportsPagedKV=false vetoes paged for recurrent models upstream. Map model index → storage slot so the flag can flip without a crash. Co-authored-by: Jeremy Joplin <jeremykjoplin@gmail.com> * fix(provider): close the review races in serving-set reserve resolution Four adversarial-review findings on the per-model floor machinery: 1. In-flight loads join the reserve basis (advertised ∪ resident ∪ modelsLoading): a hard-swap could drop a high-floor model from advertisedModels while its load was still running — in neither set, a relax computed in that window sized admission and grants below the reserve the model decodes under once installed (OOM direction). 2. Reserve pushes are epoch-stamped and the budget discards stale ones: cross-actor jobs from different tasks are not FIFO, so a relax computed before a raise could be DELIVERED after it and stand. Epochs increment under ProviderLoop's isolation = true mutation order. Regression test added. 3. Shrink paths regrow survivors: failed-self-test retirement now un-advertises BEFORE unloading (so unload's refresh-then-regrow runs against the shrunken set) and regrows after a cold retirement; dropAdvertisedBuild regrows after its relax. Grant clamps are min(granted, current), so nothing else ever hands the difference back (under-grant direction — capacity loss, not OOM). 4. Doctor's serving-set basis mirrors the daemon's advertise chokepoints (memory-filtered scan + supported families) instead of the raw scan, so a cached too-large or unadapted model cannot falsely FAIL a tier the daemon serves at a lower floor. The unfiltered list stays for target diagnosis. Co-authored-by: Jeremy Joplin <jeremykjoplin@gmail.com> * fix(provider): reserve re-push at load transitions + retirement tombstone Round-2 adversarial-review findings: 1. ensureModelLoaded re-pushes the reserve right after modelsLoading.insert — the insert sits after the specDecPreparation suspension, so a mid-await hard swap could drop the model from advertisedModels and push a relax while it was in neither basis set; the budget then admitted against a reserve the resuming load re-pins (OOM direction). The failed-load path symmetrically relaxes and regrows survivors — a dead high-floor id otherwise pinned the budget and grants forever (capacity-loss direction). 2. Failed-self-test retirement holds a retiringModels tombstone across its unload drain: with the slot still resident, a concurrent same-id prefetch reports .alreadyAvailable and its verified-insert re-advertised the failed build, undoing the fail-closed removal. The coordinator un-advertise also moves BEFORE the drain — leaving it after kept routing new requests at a slot fastAdmissionReject accepts, parking them behind the drain for a post-accept 404. Co-authored-by: Jeremy Joplin <jeremykjoplin@gmail.com> * fix(provider): hold the load gate through failed-load cleanup + seal the retirement/prefetch races Round-3 adversarial-review findings: 1. The failed-load catch released isLoadingAny/modelsLoading before its awaits — a new same-id loader could start mid-cleanup and place its own pending-load reservation (shared key, unconditional overwrite), which the old catch's release then DELETED, blinding KV admission to the incoming weights; the early unlock also let another model's weights go resident before the survivor regrow (sized from modelSlots only). Cleanup now runs release → clearCache → marker removal → refresh → regrow entirely under the held gate, unlocking last. 2. applyVerifiedPrefetch re-checks the retirement tombstone after each suspension (the pre-insert reserve push and the post-insert refresh) — retirement interleaving in either await could otherwise be undone by the resumed insert or the client announcement. The refused pre-insert path also re-refreshes so its phantom raise doesn't dangle. 3. Retirement waits out a drain another task owns (waitForModelUnload) before dropping its tombstone, and the forceReconnect moves back AFTER the drain: reconnecting first cancels every in-flight request on every model (cancelAllInflight on the disconnect event). The client store removal stays pre-drain so any reconnect during it registers without the failed model; the residual routed-404 window is bounded and absorbed by dispatch retries. Co-authored-by: Jeremy Joplin <jeremykjoplin@gmail.com> * feat(provider,coordinator): measured activation floor for qwen3.6-35b (4.0 GiB) Adds qwen3.6-35b-a3b-vl-mtp-mxfp8 → 4.0 GiB to both mirrored floor tables (one commit, per the contract), from the 2026-08-30 sweep on the served artifact (hub 73a03825, stock v2 = production's live route): non-KV transient saturates ~3.3 GiB across 500→8k-token B=8 cells (the raw peak-over-resident growth is contiguous-cell KV, separately accounted by the KV budget) + ~0.4 inline-MTP allowance + slack. Basis notes sit on both table entries because gpt-oss's 3.5 uses the July raw-500 convention. Effect: qwen3.6 needs drop 30.3 → 28.8 GB. That widens the 36 GB tier comfortably; the catalog's 32 GB tier remains physically marginal (<4 GiB left for macOS at the cap) — recommend min_ram_gb 32 → 36 as a separate ops/catalog change, per docs/reports/2026-08-30-activation-floor-measurements.md. Co-authored-by: Jeremy Joplin <jeremykjoplin@gmail.com> * fix(provider): ABA-proof fail-closed retirement against prefetch verification Round-4 adversarial-review finding (wrong-results direction): the prefetch's detached scan/hash suspension is long enough for an ENTIRE retirement — tombstone insert AND removal — to complete inside it, after which the resumed prefetch observed no tombstone, re-advertised the self-test-failed build, and a routed request could reload and serve it without re-passing the self-test. The durable mark is the failed WEIGHT HASH (failedSelfTestHashes, set at retirement): the same bytes that failed are refused at verification no matter how the suspensions interleave; a build with different bytes clears the record and gets its chance. The announce block also re-validates after its two client awaits and UNDOES the client-store add if retirement removed the local advertisement mid-announce — otherwise the post-drain reconnect registered a build the loop no longer serves (persistent false routing). Co-authored-by: Jeremy Joplin <jeremykjoplin@gmail.com> * fix(provider): durable self-test record at every prefetch guard + registration convergence on undo Round-5 adversarial-review findings: 1. The durable failed-self-test record was checked only at the FIRST guard — a retirement completing entirely inside the reserve-push suspension set the record after that check had passed, and the later guards consulted only the ephemeral tombstone. A shared helper (selfTestRecordRefuses) now runs at all three guards. The record is also ALWAYS set at retirement ("" sentinel when no reliable hash is known — the scanner keeps a previous hash on recompute failure, so a stale H1 could be recorded while H2 actually failed; the sentinel refuses every same-id build until restart, fail-closed), and a different-hash clear happens only at the actual advertise point. 2. The announce-undo could not retract a registration the reconnect loop had already encoded from the transiently-stale client store — the coordinator kept routing the retired id until an unrelated reconnect. The undo path now forces one more reconnect so registered inventory converges on the corrected store (rare double-race; bounded, correctness-restoring). Also aligns the measurement report's MTP allowance wording (~0.4) with the floor-table comment. Co-authored-by: Jeremy Joplin <jeremykjoplin@gmail.com> * fix(provider): slot-bound hash for the failed-self-test record Round-6 residue: the scanner maps (liveModelHashes/modelHashes) keep a previous value when recomputation fails, so retirement could record a stale H1 while H2's bytes actually loaded and failed — and a later verified H2 cleared the record and re-advertised the failed bytes. Record the slot's own verified binding (cacheEligibleWeightHash — the bytes the slot actually loaded) or the refuse-all sentinel; never the maps. Co-authored-by: Jeremy Joplin <jeremykjoplin@gmail.com> * feat(provider,coordinator): measured resident weights + qwen3.5 floor (Phase 3) Second over-reservation lever from the sweep: the scanner's disk×1.2 padding overshoots real MLX residency by ~6 GiB on the 26B 8bit artifact — enough that gemma-8bit's 36 GB tier was arithmetically unserveable (padded 31.3 + floor + 1 > 32.4 cap; measured 25.5 fits with margin). Adds the twin measured-residency tables (measuredResidentWeightsBytes / servabilityMeasuredResidentGiB, same 0.8.16 gate, same move-together mirror contract as the floors), threaded through: - provider: load gate, pending-load reservation, startup preload plan, doctor - coordinator: cold token-budget estimate AND the cold-load admit gate (reportedFreeForLoadAdmits + its four callers — without this, routing never offers the tier a cold load its providers can now take) Text-only models only (gpt-oss 11.5, gemma-4-26b/-8bit 25.5 GiB measured); vision-capable models keep the padded estimate until a vision-inclusive residency measurement exists. Also: qwen3.5-35b-a3b → 4.0 GiB activation floor in both tables (measured 2026-08-31; byte-identical B=8 profile to qwen3.6, pre-covers the family MTP default), and the plan records two deliberate resolutions — the flat default retune (5.5→~4.5) is DEFERRED until vision-inclusive measurement (its only beneficiaries are exactly the models a text-only bench cannot vouch for), and qwen3-vl-30b's real blocker is a missing CBv2 adapter (qwen3_vl_moe unsupported — the catalog entry is dark fleet-wide; ops decision, not memory arithmetic). Co-authored-by: Jeremy Joplin <jeremykjoplin@gmail.com> * fix(review): vision-gate the measured tables, padded pending-load, canonical docs Codex PR review (5 findings), all addressed: 1. qwen3.5/3.6 floor entries REMOVED (back to the 5.5 default): they are vision-capable and the tower transient rides the activation reserve — VisionMemoryGate charges decoded-media buffers + KV, and VisionTowerBudget only bounds the tower's N² allocation, so text-decode evidence alone must not lower the reserve. Their measured ~3.3 GiB text envelopes stand in the report as the baseline; entries return with a vision-inclusive measurement. 2. gemma-4-26b/-8bit residency entries REMOVED: the artifact carries vision_config (model_type gemma4), production loads the tower via VLMModelFactory, and the bench's forced-LLM 24.97 GiB under-counts it. The 36 GB tier unblock is gated on a provider-path residency measurement. Residency table = gpt-oss only (model_type gpt_oss — no VLM wrapper; the bench load IS the production path). 3. Pending-load reservation reverted to the PADDED estimate even for measured models — it guards the load transient, which steady post-load residency does not cover. 4. AGENTS.md memory-model section rewritten: the canonical instructions described the flat-reserve design and warned against exactly the per-model contract this branch ships; they now document the serving-set resolution, both mirror tables, the move-together rule, the vision gate, and the ≥4k-cell measurement convention. 5. Measurement report conclusions/follow-ups updated to final state (qwen3.6/3.5 measured; harness fix landed as mlx-swift-lm#129; qwen3-vl's blocker is the missing CBv2 adapter). Co-authored-by: Jeremy Joplin <jeremykjoplin@gmail.com> * chore(release): pin mlx-swift-lm to main (incl. bench fix #129) + bump version to 0.8.16 Submodule fe01df9 → 30da946: brings the merged bench-harness fix (mlx-swift-lm#129, paged-cache index mapping + supportsPagedKV veto) plus the engine work that landed on main since the old pin — #125 Qwen3-VL CBv2 DeepStack (engine side; provider wiring still gated off in EngineV2SupportedModels), #126 sorted-hint lane, #127 vision batch, #128 serving-correctness batch, #118 dense Qwen3.8 MTP artifacts. ProviderCore.version 0.8.13 → 0.8.16, honoring the release coupling documented at servabilityPerModelFloorMinVersion (this tree is the release train that ships the per-model tables). LatestProviderVersion moves with it — TestLatestProviderVersionMatchesProviderCore enforces the pairing; the constant only reaches operators when the coordinator deploys at release. Also hardens PagedKernelPreflightTests' hung-child deadline 0.5s → 2s: sibling tests run runtimeSmoke on real Metal kernels in parallel and the new engine's heavier compile load could pre-empt the child's startup past half a second (pid file never written → the kill assertion misread as a missing file). A rarer residue of the same timing class remains under tight isolation looping (~1-in-3 at 2s) — known-flake, same family as the KV-reclaimer wall-clock tests; full suite is green (2246/233). Co-authored-by: Jeremy Joplin <jeremykjoplin@gmail.com> * fix(review): retirement inference gate + 0.8.16 changelog + canonical citations Codex round-3 findings: 1. (P1) New inference is now rejected on BOTH resident-slot fast paths while a failed-self-test retirement drains: fastAdmissionReject returns an immediate reject (coordinator reroutes) and ensureModelLoaded throws a 503-mapped retiring error before the resident fast-return — the promised 404 only lands after the drain, and the fast paths were accepting work against a build that failed its serving-path self-test. Regression test + test seam added (RetirementGateTests). 2. (P2) CHANGELOG gains the v0.8.16 release-candidate entry: per-model floors + measured resident weights with the coordinator mirrors, the reserve race-hardening series, and the mlx-swift-lm pin advance to 30da946 with its engine PRs enumerated. 3. (P1) The plan's qwen3-vl provider-gate claims now cite the canonical code (EngineV2SupportedModels.swift:41 predicate; EngineV2Factory+Production.swift:656-region family switch). Co-authored-by: Jeremy Joplin <jeremykjoplin@gmail.com> * fix(review): retirement re-checks after every suspension + standalone measured weights + plan truth Codex round-4 findings: 1. (P1) The retirement gate is re-checked after EVERY actor suspension that precedes a resident-slot return, via a shared throwIfRetiring helper: ensureModelLoaded re-checks after the unloading-wait loop and after the loading-waiter resume (a parked request could otherwise resume past a tombstone inserted while it waited and submit to the failed build); fastAdmissionReject re-checks after both availableMemoryGb suspensions (no inference_accepted for a retiring model). 2. (P1) StandaloneServer's load gate uses the measured-residency weights figure too (loadGateWeightsGb) — darkbloom local's 24 GiB gpt-oss case was still being charged the disk×1.2 estimate. The FLOOR deliberately stays flat in standalone; the weights constant has no serving-set semantics to differ on. 3. (P2) The plan's Phase-3 status block now states the truth: both qwen floors deferred (tables ship gpt-oss only), qwen3.6's 32 GB tier does NOT fit at the 5.5 default (30.3 > 28.8 cap — re-tier to 36 REQUIRED), and gemma-8bit's 36 GB tier stays blocked pending the VLM-path residency measurement. Co-authored-by: Jeremy Joplin <jeremykjoplin@gmail.com> * fix(review): remaining retirement windows + doctor live basis + live preload requirements Codex round-5 findings: 1. (P1) throwIfRetiring at the two remaining post-suspension resident returns in ensureModelLoaded: after the specDecPreparation resume and at the load-gate (isLoadingAny) wait's resident return. 2. (P2) Doctor derives its serving-set floor basis from the daemon's OWN advertised list when the daemon is up and fresh — new optional advertised_models field in daemon-state (decode-tolerant), written from the live advertisedModels keys. CLI selection overrides (--all bypassing enabled_models, --model restricting) made any config-derived reconstruction wrong in both directions; the config-derived basis stays as the offline fallback. 3. (P2) Startup preload recomputes each candidate's requirement at its admission step (Dependencies.currentRequiredGb hook → live loadGateWeightsGb + current loadHeadroomGb): a fail-closed retirement mid-preload relaxes the serving-set floor, and a later candidate held its stale plan-time figure — skipping a model the refreshed gate admits. 4. (P2) The plan's Phase-2 step 3 is rewritten as resolved: the 4.0 qwen floor is a rejected-for-now candidate behind vision-inclusive measurement; the 32→36 re-tier is REQUIRED regardless. Co-authored-by: Jeremy Joplin <jeremykjoplin@gmail.com> * fix(review): padded admits everywhere — measured residency is post-load-only Codex round-6 findings: 1. (P1) Measured steady residency leaves EVERY admit-time gate: the load transient (shard staging) exceeds the post-load figure, which is exactly what the disk×1.2 padding was sized for — admitting a co-resident load against 11.5 GiB steady while staging peaks higher is the OOM direction. The provider-side measured-weights table/helper and all its wirings (load gate, startup preload, standalone, doctor) are REMOVED; the canonical measured values live in servabilityMeasuredResidentGiB (coordinator) and feed ONLY coldTokenBudgetEstimate — the POST-load token-budget arithmetic, where steady residency is physically correct (it converges to warm active_token_budget_max). reportedFreeForLoadAdmits reverts to the padded conversion for every binary and model, with a regression test pinning both directions (admit refuses at measured-fits/padded-doesn't; admits at padded-fits). The gemma-8bit 36 GB tier follow-up now requires a measured LOAD-PEAK through the provider's VLM path, not just steady residency. 2. (P2) fastAdmissionReject recomputes its requirement after the availableMemoryGb suspension — a concurrent verified prefetch can raise the serving-set floor during the hop (accepted-then-503 otherwise). 3+4. (P2) Measurement report finalized: the tier verdict marks the qwen floor entry deferred (shipped tables carry gpt-oss only), and the release-gate conclusion records the SHIPPED 0.8.16 state instead of the 0.8.11-placeholder/0.8.15-train caveat. CHANGELOG, AGENTS.md, and the plan reframed to the post-load-only design. Co-authored-by: Jeremy Joplin <jeremykjoplin@gmail.com> * fix(review): empty live advertised set is authoritative in doctor Codex round-7 (single P2): a fresh daemon that retired every model after failed startup self-tests legitimately advertises [] — the !isEmpty guard discarded that live value and reconstructed the serving set from config, crediting floors for models the daemon is not serving. Non-nil is now authoritative including empty. Co-authored-by: Jeremy Joplin <jeremykjoplin@gmail.com> * docs: reconcile plan and report with master post-rebase master now serves qwen3_vl_moe directly (#752, vision off M5 #781) and auto-activates embedded MTP heads (#777): the "dark fleet-wide / no CBv2 adapter" claims become a bench-side measurement gap (LLM-factory load cannot construct the wrapper; floor stays at the 5.5 default, which fits its 32 GB tier), and the qwen3.6 production route now includes MTP k=2 — the B=1-measured +0.4 GiB allowance applies to the live route (floors remain deferred for the vision reason). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UxThXGrq5VyXfUzcahTVw6 * fix(review): live headroom through eviction/retry waits, authoritative empty doctor set, capacity publish after reserve raise Codex review of the rebased head (f591ce7), six findings: 1. evictUntilAvailable takes the weights figure and re-resolves the requirement on every iteration (and after its clearCache retry) instead of comparing against a requirement captured before its suspensions — a concurrent verified prefetch can raise the serving-set floor mid-wait, and the stale lower figure admitted a load the post-load guard then rejected (accepted-then-503). The MTP admission after the loop resolves its target requirement live too. 2. fastAdmissionReject re-resolves the requirement after its retry memory sample as well (the second suspension was outside the earlier fix). 3. Doctor keeps a LIVE empty advertised set authoritative all the way into the verdict: DoctorRunner passes [] through (only the offline reconstruction maps empty to nil), and ModelFitDiagnostic resolves an empty declared set at the default floor — the target cannot be joining a set the daemon does not advertise — instead of crediting the retired target its solo floor. Regression test added. 4. applyVerifiedPrefetch re-slices resident engines and refreshes aggregate capacity after the reserve raise and BEFORE announcing the enlarged set, so the coordinator never routes against a token budget the tightened shared KV gate rejects until the next periodic tick. 5. Plan: canonical file:line citations restored for the Qwen3-VL production path (supported-family predicate, wrapper extraction, construction switch), and the Phase-1 release-coupling item marked completed (0.8.16 in-tree, pairing test). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UxThXGrq5VyXfUzcahTVw6 * fix(review): resolve the floor after every admission suspension, preflight advertise-only reserve raises under the re-slice gate - evictUntilAvailable samples memory first and resolves the requirement after that suspension: its zero-iteration exit compared against a floor captured before the await, so a concurrent raise could admit a load the post-load guard then rejects. - admitSpecDecIfMemoryAllows takes the target's weight basis and resolves the headroom after its own memory sample (same class). - applyVerifiedPrefetch preflights the serving-set reserve raise against the resident slots' serviceability floor and refuses the advertisement when the raise would strand a survivor below it, mirroring the load path's newcomer refusal. The preflight -> push -> insert -> re-slice -> capacity-publish sequence holds the re-slice gate like a load does, so a load cannot size itself against the pre-raise budget in between. - Capacity republished after the drop and cold-retire relaxes so free_for_load_gb never lags the floor by a capacity tick; retiring rejects tagged slot_state rather than memory_cap; doctor's offline serving-set basis mirrors the daemon's runtime-eligibility filter. Co-authored-by: Jeremy Joplin <jeremykjoplin@gmail.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UxThXGrq5VyXfUzcahTVw6 * fix(review): retry reserve-refused desired builds, drain the box before the retirement reconnect - A verified prefetch refused by the reserve-raise preflight now schedules a retry through the desired-build backoff policy (the re-run is a hash pass — the bytes are on disk). `.verified` still goes out for the attempt and the coordinator deduplicates unchanged desired_models pushes, so without a local retry nothing would ever re-offer the build, even after an unload frees the room. The scheduler is factored out of the transient-failure handler and shared. - The failed-self-test retirement waits for box-wide in-flight work to drain (bounded by the shutdown drain budget) before forcing the reconnect that communicates the removal: closing the socket cancels every in-flight request on the box, not just the retired model's. - Plan doc: the paged-cache subscript is fixed on this branch (cite the mapping, not the removed line); the vision-tower transient rides the activation reserve — VisionMemoryGate covers media buffers and KV, not activations — which is the reason vision-capable floors stay deferred. - Regression test: a scripted resident gpt-oss slot on a 32 GiB box where the resident floor leaves a serviceable grant but an unmeasured build's raise would not; the desired build is refused, one retry is pending, and it advertises once the box has room. Co-authored-by: Jeremy Joplin <jeremykjoplin@gmail.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UxThXGrq5VyXfUzcahTVw6 * docs(plan): gemma residency stays deferred behind the VLM-path measurement The sequencing bullet still listed gemma-qat4 as a text-only artifact ready for measured residency, contradicting the plan's own vision gate: its checkpoint declares vision_config, the provider loads it through the VLM factory, and the text-only 13.48 GiB figure undercounts the tower. Only gpt-oss is ready for Phase 3a. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UxThXGrq5VyXfUzcahTVw6 * fix(review): detach and coalesce the retirement reconnect, harden the preflight test against slow runners - The startup preloader awaits retirement inline, so the box-wide drain wait added for the reconnect would have stalled every remaining preload candidate behind a busy box for up to the drain budget. The reconnect is now a single detached task: un-advertise, unload, and the tombstone stay inline; one background re-registration per burst of retirements fires after in-flight work drains (the client store already excludes every retired id by then), skipped on shutdown. - Load-failure rejects capture the retiring flag before the cleanup awaits, so a retirement completing during them cannot misfile the reject as memory_cap. - ReserveRaisePreflightTests asserted an exact prefetch count that a slow CI runner violated (the first retry fired before the observation). Assert the refusal invariants that hold at any point of the retry cycle and use a retry budget the tight phase cannot exhaust. Co-authored-by: Jeremy Joplin <jeremykjoplin@gmail.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UxThXGrq5VyXfUzcahTVw6 * fix(review): discard capacity snapshots built across a reserve move, barrier admission across the retirement reconnect - updateAggregateCapacity captures the activation-reserve epoch at entry and discards its snapshot if a push landed across its actor hops: a refresh suspended inside the engine summary could otherwise resume after a verified prefetch raised the floor and overwrite the newer publish with pre-raise slot budgets and free_for_load_gb. - The post-retirement reconnect raises an admission barrier on the actor before closing the socket and re-checks in-flight work after the drain's last observation, so a routed request cannot be admitted in the hop only to be cancelled by the disconnect; the barrier lifts on the new session's .connected. Shutdown cancels a pending reconnect task. - A retired, un-advertised id with a standing failed-hash record is fast-rejected (slot_state) until the register lands, instead of accept-then-404 at the advertised guard; the same predicate tags both rejection sites. - Tests: the prefetch advertise test waits on the client store (the local insert now precedes the client add by the re-slice hops); the preflight test requires the reserve env override to be unset. Co-authored-by: Jeremy Joplin <jeremykjoplin@gmail.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UxThXGrq5VyXfUzcahTVw6 * fix(review): re-validate the load gate before allocation, serialize advertise-raises behind in-flight loads, consult the durable record in the shared guard - ensureModelLoaded re-runs the padded-weight gate after the last suspension before allocation (pending-load reservation, weight hashing, the pre-load hook): the serving-set floor can move across any of them and the pending reservation only fences competing KV grants. Refuse before shard staging rather than let the post-load guard unload an already-overcommitted transient. - applyVerifiedPrefetch defers an advertise-only reserve raise while any load sits between admission and install (modelsLoading is set before the gate), through the desired-build backoff — the primary defense; the allocation-time gate is the backstop. - throwIfRetiring and fastAdmissionReject's post-sample checks consult isRefusedByRetirement (tombstone OR durable failed-hash record): a suspension spanning an entire retirement could otherwise resume into the failed build with its pre-retirement modelInfo. - updateAggregateCapacity recomputes (bounded, three attempts) when the reserve epoch moves across its hops instead of returning without a publish — not every push site publishes a replacement snapshot. - Measurements report: paged-cache mapping marked fixed with its current citation; the text-only caveat states that the vision-tower transient rides the activation reserve and is not contained by VisionMemoryGate. Co-authored-by: Jeremy Joplin <jeremykjoplin@gmail.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UxThXGrq5VyXfUzcahTVw6 * fix(review): add the load's own reservation back in the allocation gate, pin pending advertisements into the reserve basis - The allocation-time backstop compared the free-for-load figure — which nets out every ledger entry, including this load's own pending reservation placed just above — against the full padded requirement, a double count that refused every load whose padded weights exceed the headroom (the 24 GB gpt-oss-20b tier: 18.0 required, ~4.6 netted). The comparison now goes through ModelLoadAdmission.fitsAtAllocation, which adds the load's own reservation back; regression test pins the tier arithmetic in both directions. - applyVerifiedPrefetch re-checks modelsLoading AFTER the survivor preflight (it awaits each bridge's grant, and a load admitted during those hops is in neither modelSlots nor the preflight) and pins the id into pendingAdvertise before the reserve push, so a load admitted during the push's own suspension already resolves its gate and fleet budget against the raised floor. Cleared when the id joins advertisedModels or on the tombstone refusal. Co-authored-by: Jeremy Joplin <jeremykjoplin@gmail.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UxThXGrq5VyXfUzcahTVw6 * fix(review): cover the retained MTP assistant in the allocation-time requirement The pending reservation added back by the allocation gate covers the padded target PLUS a separately staged assistant, but the requirement it was compared against covered the target alone, so the assistant's bytes dropped out of the check: a box where the target fits but target + assistant no longer does would pass and let the later assistant allocation cross the unified-memory cap. The requirement now uses the gate's weight basis with extraWeightBytes included — the same basis as the reservation. Regression test pins the asymmetry. Co-authored-by: Jeremy Joplin <jeremykjoplin@gmail.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UxThXGrq5VyXfUzcahTVw6 * fix(review): re-offer capacity-deferred desired builds on the capacity change itself A verified prefetch deferred for a capacity reason (the reserve raise would strand a resident slot, or a load was in flight) was retried only through the desired-build backoff — about 18 minutes — while the idle unload that typically frees the room defaults to 60 minutes; once the budget expired the provider stayed on the superseded build until the next desired_models push (the coordinator deduplicates unchanged state). Deferred ids are now remembered (reserveDeferredPrefetches) and re-offered with a fresh budget by the capacity-change events themselves: a slot unload and a load finishing (installed or failed). The backoff stays as the near-term retry; a still-tight box simply defers again. Cleared when the build advertises or leaves the desired set. Regression test: refused → re-offered by the resident slot's unload inside seconds with a 600 s backoff configured. Co-authored-by: Jeremy Joplin <jeremykjoplin@gmail.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UxThXGrq5VyXfUzcahTVw6 * fix(review): carve the per-model reserve in standalone serving, re-offer explicit prefetches deferred for capacity - StandaloneServer resolves the activation reserve over its serving set (configured models ∪ resident slots ∪ in-flight loads) and passes it to every consumer the network provider threads it through: the KV budget actor, the fleet KV budget behind engine grants, the load gate's headroom, the paged capacity decision, and both post-load probes. A tight box the coordinator path admits (24 GB gpt-oss) is admitted by `darkbloom local` too. Regression test pins the gpt-oss grant against the measured floor rather than the flat default; the existing re-slice test now expects the lone survivor's own floor after an eviction. - Capacity-deferred prefetches are remembered whether desired OR explicit (`prefetch_model`): the coordinator ignores the `.verified` status the refused attempt emits, so the capacity-change re-offer is the only path that ever advertises the build. Ids are forgotten before each re-run — a re-refusal re-remembers, an unwanted build drops out. Co-authored-by: Jeremy Joplin <jeremykjoplin@gmail.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UxThXGrq5VyXfUzcahTVw6 * fix(review): re-push the standalone reserve on setModels, pass it to the MTP fallback probe - StandaloneServer.setModels changes the serving set, which is part of the activation-reserve basis: it now re-pushes the resolved reserve into the KV budget actor and re-slices the resident grants under the new floor before a higher-floor model can be exposed (a load in flight re-slices at its own install, so the regrow is skipped then). Regression test: a gpt-oss-only server's grant shrinks to the raised-floor budget when an unmeasured model is added. - The target-only fallback probe after a failed MTP build now takes the resolved reserve like the first probe (the earlier pattern edit missed its indentation), so a tight box is not unloaded against the flat default it just cleared. Co-authored-by: Jeremy Joplin <jeremykjoplin@gmail.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UxThXGrq5VyXfUzcahTVw6 * fix(review): run the setModels re-slice when the in-flight load finishes, including on failure setModels skipped the survivors' re-slice while a load was in flight, assuming the load's install would re-slice under the updated reserve. A load that fails before install restores the PREVIOUS grants (sized under the old floor) and installs nothing, so nothing re-sized them while the KV budget actor already enforced the new floor. The re-slice is now deferred (resliceDeferredBehindLoad) and run from both completion paths of the standalone load — after a successful install (idempotent) and after the failure cleanup — via runDeferredResliceIfNeeded. Regression test drives the deferral through the load-marker seams and asserts the resident grant ends under the raised floor. Co-authored-by: Jeremy Joplin <jeremykjoplin@gmail.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UxThXGrq5VyXfUzcahTVw6 * fix(review): serialize and preflight standalone setModels, re-validate the standalone load gate before allocation - setModels is serialized behind any in-flight load: that load passed its gate against the current floor, so the whole list update waits (pendingModelsUpdate) and applies on either completion path of the load — failure included, where the restore-on-throw has just put the old grants back. - The update is preflighted against the resident slots' serviceability floor (reserveKeepsSurvivorsServiceable, the load path's newcomer check applied to a raise with no newcomer); a stranding raise is refused and the current list kept. setModels returns whether it was accepted. - The standalone load re-validates the padded gate after its last suspension before allocation, through the same fitsAtAllocation helper (own reservation added back, retained assistant included) — the backstop behind the serialization. - Tests: deferral behind a load (updated), refusal of a stranding raise on a single tight slot (new). Co-authored-by: Jeremy Joplin <jeremykjoplin@gmail.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UxThXGrq5VyXfUzcahTVw6 * fix(review): hold the standalone load gate across setModels, epoch-stamp its reserve pushes - applyModels now runs under the standalone load gate (isLoadingAny), held from the serviceability preflight through the reserve push and the re-slice: the preflight suspends on each resident bridge, and a cold load entering meanwhile would pass its own gate under the old floor and install a slot the preflight never saw. Loads and further setModels calls park on the gate exactly as they park behind a load; a list parked while an apply held the gate is applied right after it. - Every standalone reserve push carries a monotonic epoch (as the provider's do), so a stale lower-floor push can never land after a newer one at the KV budget actor even if delivery reorders. - Test seams hold and release the same gate the load path does. Co-authored-by: Jeremy Joplin <jeremykjoplin@gmail.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UxThXGrq5VyXfUzcahTVw6 * fix(review): keep a deferral whose attempt is still finishing, drain parked setModels, re-push the standalone reserve at every basis change - retryReserveDeferredPrefetches skips an id whose prefetch attempt is still in flight in the coordinator (ModelPrefetchCoordinator.isInFlight: the tail after onVerified returns, before the terminal status emits) — a re-run then would coalesce into that attempt and never reach applyVerifiedPrefetch again. The deferral and the desired backoff are kept for the next capacity change instead. - setModels drains a list parked behind its own apply (the load-completion drains cannot be relied on: no load may ever come). - The standalone eviction path pushes the relaxed reserve before the survivors grow into it; both standalone load-marker transitions push the reserve (the marker joins/leaves the basis); the standalone load re-validates its model's membership right before marking itself loading, so a stale modelInfo removed by setModels cannot re-enter the basis. Co-authored-by: Jeremy Joplin <jeremykjoplin@gmail.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UxThXGrq5VyXfUzcahTVw6 * docs(reserve): correct the 24 GB tier claim, derive the tier arithmetic in a test, fix stale standalone comments The CHANGELOG, the floors-table comment and the PR body claimed the measured gpt-oss floor makes the catalog's 24 GB tier reachable. The live gate resolves usable = free-plus-inactive − max(memory_reserve_gb, total − 0.9·total); with the shipped 4 GiB reserve default the 24 GB tier still needs ~22 GB free of 24 at the 18.0 GB requirement, and the #653 reporters measured 12.3 and 16.1 GB usable. What the floor does close is the 32 GB flap band (the gate needs ~22 GB free instead of ~24). The remaining 24 GB levers — the admit-time disk×1.2 padding and the small-box reserve default — are recorded as follow-ups. A DiagnosticsTests case now derives usable from total, the reserve default and a free-plus-inactive figure through the real gate arithmetic instead of taking usable as an input, and pins both tiers' verdicts. Two comments that still described StandaloneServer as keeping the flat default are corrected. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UxThXGrq5VyXfUzcahTVw6 * fix(review): wake a kept deferral once its prefetch attempt finishes; re-validate the gpt-oss floor on the shipped pins - A capacity-deferred prefetch whose attempt was still finishing inside the coordinator kept its deferral but had no wake-up: explicit prefetch_model builds have no backoff timer and the capacity-change callers may never fire again. A one-shot wake-up per id now waits (off the actor) for the coordinator's attempt to reach terminal cleanup and re-offers the deferred builds. - The measurements report's 32 GB conclusion assumed the 0.9×physical cap; the effective ceiling is min(0.9×physical, physical − memory_reserve_gb) = 28.0 GiB under the 4 GiB default, so a qwen3.6 floor near 3.5 GiB does not fit either — the 36 GB re-tier is required regardless of any floor change. - The gpt-oss-20b B=8 cells were re-run with BenchCBv2 built at exactly the pins this PR ships (mlx-swift-lm 30da946 incl. the #126 MoE dispatch change, mlx-swift 6b0505c): 4k-prompt peak−active 4.17 GiB (identical to August), 500-token 2.27 GiB; the non-KV envelope stays ~2.7 GiB, so the 3.5 GiB floor holds with ~0.8 GiB margin. The raw reports are committed under docs/reports/raw/ (the August raw files were lost with their worktree). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UxThXGrq5VyXfUzcahTVw6 * fix(load): refuse hopeless evictions up front; fold in the design review's pre-merge corrections - Eviction feasibility (#653's 32 GB report: "a request for a model I can't serve killed the one I could"): before the first eviction, the load path asks whether evicting EVERY idle model plus the reclaimable buffer cache could reach the requirement at all; when it cannot, the load is refused without evicting. The fast pre-accept check applies the same bound so the coordinator reroutes instead of accepting a request that would only fail after the same check. Pure decision ModelLoadAdmission.evictionCanReach, unit-tested on the report's numbers. - Wording: the floor narrows the 32 GB flap band by ~2 GB (both #653 samples get the same verdict under either floor) rather than closing it; the plan no longer claims the 24 GB tier is unblocked; the consumer hardware-requirements doc describes the reserve instead of a 2 GB headroom; an orphaned qwen basis comment (no such table entry) is removed from the coordinator floors table. - Re-validation: the 4k batch-8 cell re-run with production's prefill construction (chunk 512, solo stripe 2048) measures the same 4.17 GiB peak-over-active; raw report committed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UxThXGrq5VyXfUzcahTVw6 --------- Co-authored-by: Jeremy Joplin <jeremykjoplin@gmail.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

glg2672 mentioned this pull request Aug 24, 2026

feat(provider): integrate Qwen3.8-27B full VLM #700

Merged

2 of 5 tasks

Gajesh2007 mentioned this pull request Aug 31, 2026

fix(provider,coordinator): per-model activation floors — measured, hardened, rebased for v0.8.16 (supersedes #654, addresses #653) #791

Merged

Gajesh2007 closed this Sep 14, 2026

Navigation Menu

Repository navigation

fix: allow Swift providers without Python privacy caps - #118

fix: allow Swift providers without Python privacy caps#118
hankbobtheresearchoor wants to merge 1 commit into
Layr-Labs:swift-providerfrom
hankbobtheresearchoor:fix/swift-provider-privacy-caps

hankbobtheresearchoor commented May 3, 2026

Uh oh!

vercel Bot commented May 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

hankbobtheresearchoor commented May 3, 2026

Summary

Validation

Context

Uh oh!

vercel Bot commented May 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants