release(provider): v0.7.5 — one engine: full legacy deletion, vision+video+prefix on CBv2, KV re-slicing, fail-loud#522
Merged
Merged
Conversation
…o main (+ Codex P2 fixes, fused-cache base)
…s (v0.7.4) Image requests on a v2-bridged Gemma 4 slot now prefill through the CBv2 engine instead of the legacy non-batched wrapper path — vision joins the continuous batch: - EngineV2VisionPrefill: decode media exactly as legacy (buildUserInput, same caps/errors), run the SAME Gemma4Processor.prepare (resize/ normalization/templating/placeholder expansion), run the wrapper's vision tower + projector via the new per-image seam and eval under container isolation (engine step loop never pays the tower cost), carve one CBv2ImageSpan per image out of the placeholder runs (ascending/non-overlapping/in-prompt by construction; adjacent images carve into adjacent spans the engine coalesces into one block). - MultiModelBatchSchedulerEngine: media branch attempts v2 when the slot has a bridge and the request has no video (hasVideo gate — video stays wholly on legacy, deliberately not half-supported); ANY construction failure -> WARN engine_v2_vision_fallback + unchanged legacy path (never a dropped request); CancellationError releases and rethrows instead (no WARN, no legacy re-serve). Vision reservation released once the bridge owns the request (its shared-budget gate covers prompt incl. soft tokens + max_tokens). Shared event-translation pump extracted (makeEventStream) and reused by both text-v2 and vision-v2 paths. Injectable EngineV2VisionPlumbing so unit tests drive the routing without weights. - EngineV2Bridge.submitTokenized(multimodal:) threads the spans + precomputed embeddings into CBv2Request; INFO engine_v2_vision telemetry per accepted vision request (filter-at-source). - Submit-time CBv2MultimodalError -> canonical multimodal_rejected: message -> .multimodalRejected -> HTTP 400 (deterministic rejections, never retry-signaled as capacity). - Telemetry: multimodal boolean field allowlisted in all three mirrors (Swift filter, Go coordinator, console-ui TS). Fallback WARN carries the human-readable error only for the content-safe EngineV2VisionPrefillError; foreign throws get error_class only. Unit tests (scripted CBv2Engine, stub container, injected preparer): span carving incl. adjacency/at-start/mismatches, bridge passthrough of spans+embeddings+telemetry, routing success, vision-reservation released exactly once (real GlobalKVCacheBudget), construction-failure fallback + WARN, non-bridged/video/text gates, multimodal_rejected->400.
…kpoints New self-contained GemmaVLMVisionEngineV2LiveTests (deliberately independent of GemmaVLMEngineV2LiveTests — that file is owned by the concurrent 0.7.3 hotfix and is untouched here): - a real image request end-to-end through v2 (real vision-tower embeddings at real spans), asserted to actually take the v2 path (bridge admit count + zero fallback WARNs); - greedy determinism (same image + prompt twice => identical) and embedding steering (red vs blue image, same prompt => different completions — the key 'spans/embeddings actually reach the model' regression); - legacy-vs-v2 output for the same input logged, asserted only qualitatively (v0.7.2 RoPE-correctness divergence between extracted model and wrapper makes token-exactness a non-invariant, mirroring the parity-gate reasoning); - interleave hygiene: legacy-vision and v2-vision requests between v2 text decodes leave the text output bit-identical (mrope-class regression pattern); - structured teardown: unload/shutdown awaited on every exit path.
…er a mapped 500 Codex quality-gate finding on the v0.7.4 range: a CancellationError thrown while STARTING the stream (consumer cancel during prompt templating or vision-feature construction — handleCancellation cancels the detached request task, and the vision seam rethrows instead of falling back to legacy) was caught by the generic pre-stream catch and mapped via mapInferenceErrorToStatus, which fell through to 500 — a client's own cancel counted as a provider .inferenceError and could trip the (provider, model) 5xx routing cooldown. - ProviderLoop+InferenceHandler: the pre-stream catch special-cases CancellationError / token.isCancelled and mirrors the established cancelled-with-nothing-delivered terminal exactly: 499 + "request cancelled" + the cancellations-before-output stat. - ProviderLoop+ErrorMapping: CancellationError -> 499 at the top of mapInferenceErrorToStatus as defense-in-depth for every other call site (the standalone --local HTTP path). - Test (routing harness): cancel during vision construction -> CancellationError propagates (no fallback WARN, engine untouched), model reservation released exactly once, vision KV reservation fully released, and the mapper reports 499.
…rVersion (0.7.3 is the shared-KV-gate hotfix release, shipped separately.)
…ugh v2 in 0.7.4, video stays legacy
… base — pin stays bc1c0ee6, version renumbered
…fixCacheV2; salt scoping live
… carved from engine KV ceiling - PrefixCachePolicy (scheduler-free): DARKBLOOM_PREFIX_CACHE gate (default ON, 0/false/off/no opt-out), DARKBLOOM_PREFIX_CACHE_MAX_GB budget (default physical/8 — legacy numbers verbatim), and the carve that splits a slot's fleet-sized KV grant between engine admission ceiling and cache budget (1 GiB serviceable floor: live KV wins, the prefix budget shrinks; sub-one- block budgets degrade to off). Legacy gate/budget/stats resolvers delegate here — one source of truth until legacy deletion. - makeEngineV2BridgeForSlot is THE single carve point: the engine, its AdmissionV2 ledger, the bridge capacity snapshot, and the heartbeat activeTokenBudgetMax all read the REDUCED capacity (engine truth); fleet sizing and the heartbeat clamp subtract co-resident slots' TOTAL claims (engine ceiling + cache budget, EngineV2Bridge.slotKVBytesClaim) plus the slot's own cache budget, so the coordinator is never told about bytes the cache consumes. - makeProductionEngine(prefixCache:): non-nil ⇒ enablePrefixCache — TB-007 comment replaced with the updated T-041 posture; cacheScope → CBv2Request .cacheSalt plumbing is now live. - Per-request usage detail: bridge records engine prefixCacheHitTokens into EngineV2RequestUsageSignal (logprobs-channel pattern); the inference handler splices OpenAI-standard usage.prompt_tokens_details.cached_tokens into the trailing SSE usage chunk (injectCachedTokens, mirror of the reasoning_tokens splice). - Periodic stats: bridge-owned logger mirrors the legacy checkpoint-tier line with an engine=v2 tag (PrefixCacheV2.stats(): hits/misses/tokensSaved/ entries/bytes), DARKBLOOM_PREFIX_CACHE_STATS_INTERVAL_SECS cadence, cancelled on bridge shutdown.
…carve, capacity agreement, salt plumbing, cached_tokens splice - PrefixCachePolicyTests: env-gate semantics (default ON, 0/false/off/no), MAX_GB budget override + physical/8 fallback, legacy-resolver delegation pins (no drift), carve conservation/floor-guard/one-block viability, makePrefixCache config (blockSize 256, model namespace, byte budget). - EngineV2PrefixCacheWiringTests: gate-off ⇒ raw grant + zero budget; gate-on ⇒ engine handed grant−budget; capacity-reduction AGREEMENT test (engine snapshot == bridge claim math == heartbeat activeTokenBudgetMax); co-residency test pinning that a later slot subtracts the earlier slot's TOTAL claim (engine ceiling + cache budget); cacheScope→cacheSalt bridge plumbing (distinct/stable/empty); EngineV2RequestUsageSignal terminal recording; injectCachedTokens splice/merge/passthrough. - slotFactoryUsesFleetResidency opts out of the (default-on) carve so it keeps pinning pure fleet sizing; heartbeatBudgetShrinksWhenSecondSlotLoads now runs WITH the carve — passing unchanged proves the own-budget clamp accounting.
…tch-invariance on gpt-oss-20b + gemma-4-26b-qat Multi-turn conversation per model, turn-1 prefix sized past the model's own recompute bound (windowCount × maxWindow from cbv2LayerKinds — 1,536 for gpt-oss, 25,600 for gemma-4): (a) turn-2 greedy output byte-identical cache-on vs cache-off (two makeProductionEngine constructions, hybrid-window recompute exercised), (b) prefixCacheHitTokens > 0 with a shared-prefix-block lower bound, read through EngineV2RequestUsageSignal, (c) cached row + cold row batched together match their solo outputs. TTFT deltas logged. Gates: DARKBLOOM_LIVE_MLX_TESTS + _GPTOSS / _GEMMA; clean skip when weights absent.
…readers (heartbeat, tests) need no actor hop
…city + Gemma4 video seam Local pin for the v0.7.5 integration branch; re-pointed to the merged mlx-swift-lm commit at ship time.
… overshoot blew the engine's 120s request deadline The first sizing loop assumed ≥8 tokens/line and rendered 104k-token prompts (target 26.4k); every gemma request then died at the CBv2 loop's 120s per-request deadline. Probe 64 lines for the real per-line rate, extrapolate, top up by measured deficit (~2% slack). Also relax the shared-prefix assertion: turn 2 shares turn 1's prompt minus the template's generation suffix (4 tokens on gemma), which is irrelevant to whole-block chain matching — require shared ≥ the sized minimum instead. Result on M4 Max: turn1=27,936 tok, hit=2,304 (=109 shared blocks − 25,600 recompute bound, exact), byte-identical turn-2, batch invariance green, 129.7s total.
Scheduler-free homes for everything the v2 slot lifecycle used to parasitize from the always-constructed legacy BatchScheduler: - SlotSizingSnapshot: weight bytes + fp16 kvBytesPerToken derived from the model's own cbv2LayerKinds (engine truth — the long-context marginal rate of AdmissionV2.estimatedBytes; VLM checkpoints decode text_config with the same decoder the extraction uses) + context window + default max tokens. KVEstimation's config.json parse stays as the pre-load estimate/fallback only. - ModelEOSPolicy: BatchScheduler.effectiveEOSTokenIds re-homed unchanged (scheduler static now forwards). - KVHeadroomProbe: measuredLiveKVHeadroomBytes/hasServeableKVHeadroom re-homed (pure UnifiedMemoryCap math; scheduler members forward). - VisionMemoryGate: reserveVisionRequest/releaseVisionRequest re-homed as a per-slot value gating media-decode RAM + generation KV against the shared GlobalKVCacheBudget. - EngineV2SupportedModels: architecture-derived supported set (mirror of the makeProductionEngine switch keyed on config model_type: gpt_oss + gemma4*). - EngineV2KVSizing.resliceGrants: pure fair-share re-slice policy for multi-model co-residency — single model keeps the FULL fleet budget, >=2 share proportional to fp16 rate x min(context, 131072), equal split degenerate, sum(grants) <= budget invariant, and the 1 GiB serviceability floor check that refuses loads which would thrash every slot below serviceability.
…whose adoption bound exceeds real prompt lengths keep their full KV grant The engine adopts a cached prefix only past windowCount × maxWindow matched tokens (cbv2RequiredRecompute; EngineV2.makeAdoption declines less). For gemma-4-26B that bound is 25 × 1024 = 25,600 tokens against a production p90 prompt of ~4k — hits are unreachable, yet the carve was still taking physical/8 (16 GiB on 128 GB; ~4.5 GB on 36 GB gemma boxes) out of the engine's live-KV ceiling for zero benefit, cutting gemma concurrency. - PrefixCachePolicy: adoptionBoundTokens(layerKinds:) (the exact bound term of the engine's recompute derivation), maxAdoptionBoundTokens env (DARKBLOOM_PREFIX_CACHE_MAX_ADOPTION_BOUND_TOKENS: absent/malformed => 4,096 default; 0 => never fund), shouldFund(adoptionBoundTokens:) — gpt-oss (1,536) funds, gemma-4 (25,600) does not; bound-unknown treated as 0/fund (pure-full-attention semantics; documented). Rationale in the file header. - Slot factory: model snapshot moved ahead of the carve; the gate resolves the bound from the loaded model (text slots, via the factory's family switch: EngineV2Factory.adoptionBoundTokens(model:)) or from the checkpoint's text_config alone for VLM slots (EngineV2VLMTextExtraction.adoptionBoundTokens(modelDirectory:) — no extraction run). Unfunded => requested budget 0 => carve passthrough: full grant to live KV, no cache instance, no stats logger; serving log names the unfunded reason (bound + cap). - Tests: policy units (bound derivation incl. prod shapes + mixed windows, threshold env semantics, funding matrix incl. boundary + never-fund), wiring (threshold-0 unfunded through the slot factory: raw grant, zero bridge budget, claim == capacity; unknown-family resolver => 0), live (gemma pins the UNFUNDED default verdict end-to-end — incl. config-only bound == loaded-model bound — then runs the funded hybrid-recompute scenario as the explicit operator override; gpt-oss pins funded-by- default). gpt-oss 15.9s (hit 768 exact, TTFT 2.78s->1.65s), gemma 147.8s (hit 2,304 exact under the override), full suite 1,391 green.
…ngine_v2, fail-loud refusals Generalize EngineV2VisionPrefill from images-only to all media on a v2-bridged Gemma 4 VLM slot: - Span carving walks the prompt once, pairing placeholder runs of BOTH token ids (image + video) with per-kind tower features strictly in prompt order — one CBv2ImageSpan per image and per sampled video frame (perVideoFrameVisionFeatures, ≤32 uniformly sampled frames, per-frame data-driven soft-token counts). Frame-count/span-count mismatches throw in both directions; the defensive videoUnsupported throw is gone. - Routing: the hasVideo → legacy peel-off in MultiModelBatchSchedulerEngine is removed; video and mixed requests prefill through the engine like images. Byte/pixel/duration caps are unchanged (same validateMedia + buildUserInput path; inline-video temp files cleaned up inside prepare). - FAIL LOUD: v2 media construction failure no longer falls back to the legacy VLM path — it emits an engine_v2_vision_refusal ERROR (tagged media_kind image/video/mixed) and returns a retriable 503 (.requestRejected) so the coordinator's pre-content failover reroutes. CancellationError still propagates as 499; MediaError keeps its deterministic 4xx (400 / videoWriteFailed 500). The legacy wrapper path remains reachable only on slots without a v2 bridge. - Telemetry: engine_v2_vision success INFO + the new refusal ERROR both carry media_kind; the field is allowlisted in all three mirrors (Swift TelemetryFieldFilter, coordinator telemetry_handlers.go, console-ui telemetry-types.ts). Tests: EngineV2VisionRoutingTests extended (video routes to v2, mixed carving order, frame-count mismatch → refusal, construction failure → refusal with media-kind tag + budget release, MediaError keeps 4xx, garbage video still 400s pre-preparer); new GemmaVLMVideoEngineV2LiveTests generate deterministic solid-color H.264 clips via AVFoundation at test time (no committed binaries) and verify construction introspection, v2 serve + greedy determinism + steering, token parity vs the legacy VLMRequestInference.stream reference (token-exact on qat-4bit), the 32-frame sampling cap end-to-end, and TTFT logging. Vision live suite re-run green as regression.
… media, media-kind tag alignment Independent-review findings on the video-through-v2 commit: - A media request whose parts all sit on non-user roles (buildUserInput drops them, so the processor yields no pixels) previously became a retriable 503 refusal that fails identically on every provider — pre-content failover would burn retries for nothing. noProcessedMedia now maps to .multimodalRejected (400, deterministic client fault, no refusal telemetry), mirroring the MediaError stance. - EngineV2VisionPrefill.mediaKind(of:) now scans user messages only, so refusal media_kind tags match the success path's lmInput-derived kind. - ProviderLoop+ErrorMapping.swift: the .multimodalRejected comment still claimed construction failures fall back to the legacy VLM path — rewritten for the fail-loud contract. - New tests: noProcessedMedia → 400 with no refusal event; refusal on a video request tags media_kind=video; mediaKind ignores non-user-role media.
…fusal, supported-set gate, v2-only heartbeat The one-engine core of v0.7.5. Every model slot serves through the CBv2 engine; the legacy BatchScheduler is no longer constructed anywhere in the ProviderLoop serving path (it remains only for the standalone 'darkbloom local' server, a separate workstream). KV re-slicing (root fix for the gemma postmortem layer 5): - ensureModelLoaded final sequence: loadContainer -> Metal/MLX guards (moved from scheduler.loadModel) -> SlotSizingSnapshot.build -> clearCache + KVHeadroomProbe (unload+503 on fail, v0.7.3 guard re-homed) -> reslice: SHRINK existing engines via their bridges' updateKVBytesCapacity -> makeBridge with the newcomer's grant (on throw: RESTORE previous grants exactly, unload, 503) -> register -> post-bridge headroom re-guard -> install slot + record model_load_time_ms on the bridge. - unloadModel: unregister -> bridge.shutdown -> clearCache -> reslice-GROW survivors (lone survivor gets the FULL budget back). - Serviceability floor: a slice leaving any slot under 1 GiB refuses the load (engine_v2_refusal reason=reslice_floor, 503) instead of thrashing all slots below serviceability. - The heartbeat clamp (liveEngineKVBytesBudget) stays as the between-reslice safety net and now reads each engine's CURRENT grant per heartbeat (engine capacity() reflects resizes). Fail loud, never fall back: - EngineV2Factory.makeBridgeIfSelected (nil-on-failure) -> throwing makeBridge; WARN engine_v2_fallback becomes ERROR engine_v2_refusal with a machine-classifiable reason (no_kv_headroom / unsupported_model / vlm_extraction_failed / reslice_floor / engine_init_failed). All fields already on the telemetry allowlist. - The selection gate dies: EngineV2Config.selection, the model allowlist, and DARKBLOOM_ENGINE_V2(_MODELS) consumption are deleted. The engine_v2 config key stays parsed-but-ignored; startup WARNs when it or the retired env vars are set. - Scan/advertise-time supported set (EngineV2SupportedModels — mirror of the makeProductionEngine switch keyed on config model_type: gpt_oss + gemma4*): unsupported families are dropped from the advertised set at ProviderLoop init (the single chokepoint feeding coordinator registration and the local /v1/models) and at prefetch-verify time. A stale-catalog load request 404s via the existing mapping; coordinator load_model pushes get load_model_status: failed. - Request-time backstop: a TEXT request reaching a registry entry with no engine at all is a hard 500 internal error. The media peel-off branch is untouched (video still runs the legacy VLM container path in this branch state — owned by the video workstream); its memory reservations now flow through the scheduler-free VisionMemoryGate. - ModelSlot: engineV2 non-optional, scheduler field DELETED, sizing snapshot + vision gate attached. Heartbeat/capacity + concurrency config + prefill: - updateAggregateCapacity: the legacy backendCapacity() fold is deleted; EngineV2Runtime.capacitySummary(fleetKV:) is the only slot source. Same BackendSlotCapacity wire shape, same state strings; no protocol changes. Fleet weights come from slot sizing snapshots. - model_load_time_ms moves to slot-level bookkeeping on the bridge (measured across the full load: weights + sizing + engine build). - productionMaxConcurrentRequests = 4 -> config: [backend] engine_v2_max_concurrent (default 4) + per-model map engine_v2_max_concurrent_by_model, clamped to [1, 8]; heartbeat max_concurrency reports the effective per-slot value. - observed_prefill_tps is now populated by the v2 bridge: prefill window = first-token time - submit, rate = prompt tokens / window, EWMA alpha 0.3, with PR #454's plausibility bounds (1 ms floor, raised 20k tok/s ceiling). This ABSORBS and supersedes #454's intent for the v2 engine — that PR's engine-side prefill-start marker was built for the legacy engine; the v2 bridge owns both timestamps natively, so no engine change is needed. Existing suites updated from legacy-fallback semantics to refusal semantics (EngineV2BridgeTests, EngineV2ProductionWiringTests) and to the bridge-required ModelSlot (LoadedModelsStoreTests, StartupPreloadTests, shared EngineV2TestStubs).
…efill sampling suites - EngineV2ResliceTests: pure resliceGrants policy — proportional shares (rate x min(context, 131072)), single-model full budget, equal-split degenerate, context cap, sum<=budget invariant, 1 GiB serviceability floor, load/unload round trip. - SlotSizingDriftTests: pins the cbv2-layer-kinds fp16 rate against the KVEstimation config.json parse for BOTH prod families (gpt-oss 24,576 B/tok; gemma-4 qat 20,480 B/tok with K-eq-V full layers), proves estimatedKVBytes mirrors AdmissionV2.estimatedBytes exactly (engine oracle, window plateaus included), and re-checks against the REAL checkpoint config.json when present in the HF cache. - EngineV2SupportedSetGateTests: init drops CBv2-adapterless families from the advertised set; stale-id loads map to 404; reslice-floor errors map to 503; coordinator load_model push for an unsupported id emits load_model_status: failed. - EngineV2PrefillSamplingTests: classifier bounds (1 ms floor, 20k ceiling, PR #454's fast-cold band accepted) and the submit->first- token EWMA population on success / non-population on cancel.
…t, real weights) The v0.7.5 §1.1 acceptance test, gated on DARKBLOOM_LIVE_MLX_TESTS + DARKBLOOM_LIVE_MLX_GEMMA with skip-if-absent per checkpoint: 1. load gpt-oss -> lone engine holds the FULL fleet KV budget (exact UnifiedMemoryCap figure); 2. worst-case admission probe (2.5M max_tokens ~ 61 GiB KV) ADMITS pre-shrink, then is cancelled; 3. a real greedy generation streams on gpt-oss WHILE gemma-qat loads: the stream completes untouched across the shrink; both slots hold live v2 bridges; grants equal the pure resliceGrants targets and sum <= the recomputed fleet budget; 4. the SAME probe is now REFUSED (token_budget_exhausted) — the next admission respects the shrunk ceiling; 5. both models serve a real decode through the production path (runStartupSelfTestDecode); unloading gemma grows gpt-oss back to the exact full budget and it still serves. Adds two read-only seams (slot sizing + slot bridge) and a bridge active-request-ids seam for the probe cancel.
…; self-scaling live probe - ModelPrefetchCoordinatorTests + StartupPreloadTests fixtures declare a supported model_type (gpt_oss / gemma4): the machinery under test (prefetch verify, hard swap, slot caps, preload gate) is orthogonal to the supported-set gate, which now drops adapterless families from the advertised set at init/verify time. - Live co-residency probe sizes itself dynamically between the expected post-shrink share (+15%) and the current full grant (−15%), under a test-scoped memory_reserve_gb that pins the cap at 64 GiB on a 128 GiB box — keeps the worst-case probe ~20 GiB so it is robust under concurrent-suite memory contention.
… factory/extraction headers
…ger directly The bridge-path probe reserved its worst case in the shared GlobalKVCacheBudget too, making the pre-shrink admit dependent on free system RAM — flaky when another suite holds tens of GiB on this box (observed: load gate refused at 14.2 GB free under a 64 GiB test reserve). The probe now submits straight to the engine (CBv2Engine.submit -> AdmissionV2.canEverFit): a pure grant-ledger verdict, drift-test-pinned to the same estimatedBytes arithmetic, immune to memory contention. The test reserve drops to a modest 8 GiB so the model LOAD gate breathes; the streaming request still exercises the full bridge + shared-budget path. PASSED on the M4 Max 128 GB (26.97 s): grantA0 = 100 GiB (gpt-oss alone, full budget) -> load gemma-qat while gpt-oss streams -> stream completed uninterrupted; grants re-sliced to 47 GiB (gpt-oss) + 39 GiB (gemma) = the 86 GiB fleet budget exactly; the ~53 GiB worst-case probe (2,355,631 tokens) admitted pre-shrink and REFUSED post-shrink; both models served real decodes; unload gemma -> gpt-oss back to 100 GiB exactly.
… (reviewer finding) Independent review found the one hole in the re-slice invariants: loads are serialized by isLoadingAny, but the idle monitor's unloadModel -> resliceGrowSurvivors runs from its own task and could interleave at the load path's suspension points — including the gap AFTER the newcomer's grant is carved out but BEFORE its slot is installed, where a regrow recomputes over the survivors alone and re-inflates them past the fleet budget (not an OOM — the shared GlobalKVCacheBudget still gates real reservations — but it broke the claimed 'sum(grants) <= budget at all times' and 'restore EXACTLY' invariants until the next re-slice). Fix: a re-slice gate (same waiter idiom as loadGateWaiters) serializes every grant-mutating section. ensureModelLoaded holds it across the WHOLE shrink -> build -> guard -> install-slot stretch (released on each failure path; the post-bridge-guard failure regrows survivors via the new resliceGrowSurvivorsLocked while still holding it); resliceGrowSurvivors (idle unload) is the gated wrapper. Deadlock-free: neither gated section calls the other — eviction unloads inside ensureModelLoaded run before the gated stretch begins. Regression test (fails without the fix, verified: sum(grants) 58.7 GB vs 29.6 GB budget): hold the gate as an in-flight load would, park a regrow behind it, install the newcomer, release — the regrow must mutate nothing while parked and must recompute over BOTH slots after. Also fixes the reviewer's doc nits: stale makeBridgeIfSelected/ selection-era headers in EngineV2Bridge.swift and EngineV2Factory+Production.swift.
… survivor KV grants (Codex finding) Codex review of the v0.7.5 re-slice found the failure-unwind ordering inverted: on a failed load of model B, ensureModelLoaded had already released the pending-load reservation (B's weights resident, counted only by live MLX usage), and the unwind then restored/regrew survivor grants while B's container was STILL retained by the load scope until the outer catch ran. Window: sum(engine grants) > the true fleet budget while B's weights were resident -> heartbeat/admission could transiently over-advertise capacity the shared GlobalKVCacheBudget would then reject (the gray-box class; never an OOM - the shared gate still bounded real reservations). Fix - both failure paths now unwind in this order, with the re-slice gate held across the entire sequence: shrink -> build fails -> RELEASE newcomer weights (EngineV2NewcomerBox) -> MLX clearCache -> restore/regrow survivor grants -> gate release - EngineV2NewcomerBox: ownership box for the loading container; ensureModelLoaded routes every access through it (borrow() is evaluated inline, never bound to a long-lived local) so release() provably drops the LAST strong reference to the weights. - resliceAndBuildEngineV2Slot takes the box; its catch releases + clears the cache BEFORE growing shrunk engines back to their exact previous grants. The pre-shrink floor refusal also releases promptly. - unwindBuiltSlotAndRegrow: the post-bridge-headroom failure path (unregister -> shutdown -> release weights -> clearCache -> regrow), shared by ensureModelLoaded and the test seam. - All existing cleanup semantics preserved: recordModelLoadError, loading-waiter wakeups, 404/503 mapping, load-gate release ordering, and the reslice-gate span through slot install (with a defensive gate release if the install-time container read could ever fail). Regression tests (EngineV2ResliceUnwindTests, verified FAILING on the old ordering - newcomerAlive was true at restore/regrow time - and passing on the new): a weak reference to B's container is checked at the exact instant each survivor-grant update lands on the engine; both the build-failure restore and the post-bridge-guard regrow must observe the newcomer's weights already dead, so the restored grants are <= the true fleet budget at every observable point of the unwind. Full suite: 1,386 tests / 127 suites green.
…ner loading Pulls the production bridge assembly (container EOS snapshot -> ModelEOSPolicy -> VLM text extraction -> makeProductionEngine -> fail-loud makeBridge + the kv_quant WARN) out of ProviderLoop's makeEngineV2BridgeForSlot into EngineV2SlotFactory.makeProductionBridge, and the VLM-aware checkpoint->container factory pick into ModelContainerLoading. Behavior-preserving: the ProviderLoop production branch now delegates to the factory (test-hooks branch unchanged), and the loop keeps registration + logging at its call site. Prepares the standalone server rewire (both slot owners construct through ONE path so assembly can never drift).
…local chat body ceiling (v0.7.5 §1.9) The last legacy BatchScheduler construction site outside tests is gone: darkbloom start --local now builds the SAME v2 slots the ProviderLoop does — sizing snapshot -> KV re-slice over the shared pure policy (EngineV2KVSizing.resliceGrants: single slot = full budget, >=2 share by rate x capped context, shrink-first / restore-on-throw / grow-after, serviceability-floor refusal) -> EngineV2SlotFactory.makeProductionBridge against the server's own GlobalKVCacheBudget. Grant mutations are serialized by the existing isLoadingAny load gate (standalone evictions only run from the load path, so no separate re-slice gate is needed). LRU eviction drains the bridge and regrows survivors; VLM checkpoints now load via the VLM factory so media requests work locally too. Fail loud: models without a CBv2 adapter are dropped from the catalog at construction (WARN) and the CLI prints a per-model error + exits non-zero when nothing serveable remains; engine construction failure surfaces as 503 (tokenBudgetExhausted shape) on the local endpoint; requests for dropped ids 404. engine_v2_max_concurrent (+ per-model map) now reaches standalone engines. The AcquiredModel/RegistryEntry scheduler leg is documented DEAD (removal belongs to the deletion pass). Also fixes the known 413 backlog item: the upstream router's BasicRequestContext pins Hummingbird's 2 MiB decode default with no config knob (protocol-witness default, not overridable retroactively), so the chat-completions POSTs are now served by LocalChatUploadResponder — collects the body under a 32 MiB ceiling (matches the coordinator WS path), decodes with the identical JSON configuration, and calls the same public MLXOpenAIService entry points with byte-identical response framing. Both --local and --local-endpoint get the fix; other routes keep the upstream decode path. Tests: v2-entry acquire (bridge populated, NO scheduler), catalog gate (construction + setModels + request-time 404), re-slice across loads + eviction regrow + restore-on-throw pinned against the pure policy, 3 MiB body clears the ceiling (404 unknown-model proves it reached the engine, streaming + buffered), >32 MiB -> 413 envelope, non-chat routes unchanged.
…r the retained container, swap (v0.7.5 §1.10)
Ports the legacy BatchScheduler+Liveness selfRestartForRecovery to the
one-engine slot shape. The v2 engine detected wedges (WedgeMonitor +
stepsExecuted flatline -> heartbeat 'crashed') but could not HEAL; the
capacity-refresh tick now drives ProviderLoop+EngineV2Liveness:
* CONFIRMED wedge (EngineV2Bridge+Liveness): >=1 hanging admit stalled
>= 120s (the legacy BackendLivenessPolicy.defaultWedgeStallSeconds,
one definition) AND the engine step counter frozen >= 120s — the
flatline term keeps a starved-but-alive engine from triggering a
rebuild (stricter than legacy .wedged; documented).
* Recovery: legacy-shaped engine_health telemetry
(engine_v2_self_restart ERROR / _complete INFO with duration_ms /
_cooldown_unload / _failed, all allowlisted fields) -> old bridge
reports 'reloading' (legacy isReloadingForRecovery precedence over
'crashed') -> drain (pump cancel + bounded engine drain) -> rebuild
engine+bridge over the RETAINED container with the slot's CURRENT
grant (NOT a re-slice — co-resident grants untouched, sum(grants)
<= budget preserved) via the existing factory path (re-registers in
EngineV2Runtime) -> atomic ModelSlot swap.
* Serialization: the drain->rebuild->swap stretch holds the re-slice
gate; a synthetic requestToModel pin keeps the idle monitor + every
eviction filter off the recovering slot; explicit unloads win —
every suspension point re-checks slot identity and aborts the swap.
* 120s cooldown (legacy livenessRestartCooldown): a second confirmed
wedge inside it UNLOADS the slot (fail loud — heartbeats drop it,
coordinator routes around) instead of thrashing rebuilds; rebuild
failure also unloads (refusal already fired).
* Config/env parity: none to mirror — the legacy watchdog had no
gate (always-on, 120s/120s); this port keeps that and adds no
knobs. The legacy .pinned verdict is deliberately not ported (the
collapsing-token-budget failure mode does not exist on re-sliced
v2 byte grants).
Tests (scripted engines, injectable clocks, no weights): verdict
thresholds (stall + flatline both required; first token clears; moving
steps never confirm), 'reloading' state precedence, full state machine
(drain -> rebuild with the EXACT prior grant -> re-register -> swap ->
serving again with a real streamed token), the two-model grant
regression (recovering slot keeps ITS grant, co-resident engine sees
zero capacity updates, sum <= budget), cooldown unload + re-wedge
after cooldown recovers again, rebuild-failure unload, and
shutdown/unloading skips.
…kpoints
The standalone live tests served tiny Qwen models, which have no CBv2
adapter — under one-engine v0.7.5 the standalone server (correctly)
refuses them, so the tests are re-pointed at the production pair:
* two-models-through-one-process now serves gpt-oss + gemma-qat (the
EngineV2CoResidencyLiveTests fixtures; additionally gated on
DARKBLOOM_LIVE_MLX_GEMMA) and doubles as the standalone
co-residency check (both slots hold live engine KV grants);
* socket-disconnect cleanup runs on gpt-oss with the bridge-level
debug surface (debugActiveRequestCount / debugSlotReservationCount
replacing the legacy SchedulerCapacity shape) and gpt-oss-sized
timeouts.
liveModelInfo gains modelType/estimatedMemoryGb parameters (defaults
preserve the remaining legacy call sites, which the deletion pass owns
— the tiny-qwen ProviderLoop live tests have been unservable since the
v2-only slot lifecycle landed and are on the §1.11 prune list).
…tract note (fd5baca) The per-request tool_call_parser validator now pins exactly what the stream parses (nil loaded format ⇒ .json), closing the reviewer-flagged divergence window where a matching 'json' override could be wrongly rejected under an operator --model-type. Doc-only contract note on the jinja fallback.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Preserve 400s for planner validation rejects
When the v2 scheduler emits one of these deterministic planner validation errors, this branch wraps it as .requestRejected, but ProviderLoop.mapInferenceErrorToStatus maps .requestRejected to retryable 503 because the same case is also used for provider-side refusals. Requests with an invalid token count, duplicate request id, or per-batch token-budget violation therefore get a capacity/failover response instead of the intended 400 and can be retried even though they cannot succeed; use a distinct 400-mapped case for these validation failures.
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
#68 (one-engine: resizable KV, video seam, v1 deletion) squash-merged to mlx-swift-lm main as ffede00; main additionally carries #65 (Qwen3.5-VLM ropeDeltas batched-decode crash fix — the broadcast_shapes E2E flake). Tree-verified: main == the release engine branch (fd5baca) byte-for-byte except #65's Qwen35 files; all CLI/template/ChatSession fixes intact. Provider builds clean against the new pin.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
release(provider): v0.7.5 — one engine: full legacy deletion, vision+video+prefix on CBv2, KV re-slicing, fail-loud
What this release is
Since v0.7.0 the provider has run two engines: CBv2 for three allowlisted model ids (text only), and the legacy scheduler for everything else — including as a silent fallback when CBv2 couldn't construct. That fallback (cap 24, eager decode, 2–3 tok/s at batch ≥4) carried 54% of open-pool gemma traffic during the Jul 4–6 collapse (
docs/reports/2026-07-06-gemma4-serving-failure-postmortem.md, layer 5). This release ends the duality: every request serves on CBv2 or fails loudly; the legacy engine is deleted from the tree.Net diff: 199 provider files, +12.5k/−35.2k (99 files deleted) plus −16.2k lines in mlx-swift-lm (the whole v1
ContinuousBatching/engine), plus the encrypted SSD KV-offload tier (+5.0k, provider-only). The submodule is pinned to the currentmlx-swift-lm/maintip,1a0d40ae79dbda987c6fb06235b9caf2a3b3f397, which includes runtime-resizable KV capacity and the Gemma4 per-frame video seam. The SSD tier needed no additional engine API: it conforms to the frozenCBv2PrefixCacheexistential.Behavior: before / after
flowchart LR subgraph Before["Before (v0.7.4)"] A1[request] --> B1{model allowlisted\n& v2 buildable?} B1 -- yes, text --> C1[CBv2] B1 -- "no / noKVHeadroom\n(2nd model on box)" --> D1[SILENT legacy fallback\ncap 24 → 2-3 tok/s @ batch≥4] A1 -- image --> C1 A1 -- video --> E1[legacy VLM path] F1[first engine claims ALL KV\nforever - co-residency broken] end subgraph After["After (v0.7.5)"] A2[request: text / image / video] --> C2[CBv2 - the only engine] C2 -- "can't build / floor" --> G2[503 + engine_v2_refusal ERROR\ncoordinator reroutes invisibly] H2[KV re-sliced on every load/unload:\nsingle model = full budget,\nco-residents = fair shares,\nin-flight never disturbed] endCode: before / after
flowchart LR subgraph Before2["Before"] S1[ModelSlot] --> Sch[BatchScheduler ~5.5k lines\nlifecycle+admission+heartbeat+VLM+prefix] S1 -.optional.-> Br1[EngineV2Bridge] Sch --> V1[VLMRequestInference video/image] Sch --> CP[Checkpoint/SSD prefix tier] end subgraph After2["After"] S2[ModelSlot] --> Br2[EngineV2Bridge non-optional] Br2 --> F[EngineV2SlotFactory\nsizing → reslice → carve → SSD tier → engine] F --> E[CBv2 engine: compiled decode,\nvision+video spans, CBv2PrefixCache seam] F --> SSD[SSDPrefixCache kv2/\nencrypted, HMAC names, default-on\nwrite-behind + read-through staging] S2 --> VG[VisionMemoryGate] S2 --> LV[EngineV2Bridge+Liveness\nwedge → rebuild over retained weights] endMain changes
AdmissionV2.updateBytesCapacity— new engine API). Load shrinks co-residents to fair shares before granting the newcomer; unload regrows; single-model boxes keep the full budget; failed loads restore grants only after the newcomer's weights are provably released. Live-proven on M4 Max 128GB: 100 GiB solo → 47+39=86 GiB (= recomputed budget exactly) with an uninterrupted in-flight stream → back to 100 GiB on unload.engine_v2_refusalERROR telemetry (reasons: no_kv_headroom / unsupported_model / vlm_extraction_failed / reslice_floor); 503 → coordinator cooldown. The allowlist is deleted — support is architecture-derived (Gemma4 + GPT-OSS families); unsupported models are never advertised.feat/v073-multimodal(vision, renumbered) and adds video — one image-span per sampled frame via the new engine seam; live greedy parity vs the old path was token-exact; mixed image+video, 32-frame cap verified. Media construction failure = loud refusal, kind-tagged.PrefixCacheV2, budget carved out of the slot grant, per-model funding gate) ships dormant/opt-in (DARKBLOOM_PREFIX_CACHE=1with the SSD tier killed); the encrypted SSD offload tier ships DEFAULT-ON — see the section below.SlotSizingSnapshot(engine-truth fp16 KV rates, drift-tested vs config.json),ModelEOSPolicy,KVHeadroomProbe,VisionMemoryGate,EngineV2SlotFactory. Standalone server (darkbloom local) wired to v2 (+32 MiB upload fix). Wedge self-recovery ported: drain → rebuild engine over retained weights →"reloading"heartbeat, 120s cooldown, second wedge ⇒ unload.max_concurrencyreports the real engine cap (configengine_v2_max_concurrent, default 4, clamp [1,8]);observed_prefill_tpspopulated (absorbs the intent of PR fix(provider): populate observed_prefill_tps (engine prefill-start marker + raised ceiling) #454 for v2); no wire-type changes.kv_quantrejected-with-WARN (CBv2 KV-quant is a planned fast-follow; the engine's quantized-row machinery and sizing seam are preserved).Merge-readiness fixes after the coordinator stack
master, including PRs fix(routing): gray-box budget clamp + capacity-503 rate penalty (postmortem layers 3-4) #523, fix(routing): correct capacity-rate window accounting #527, and fix(registry): per-model solo-TPS quality caps + pooled-KV admission (postmortem layer 6) #524, so this PR is based on the coordinator code it will actually ship beside.darkbloom start --localreserves an incoming model's estimated weight footprint before awaiting the load, then releases it after live MLX sizing reflects the weights or on every failure path.ChunkSendercoalescing test deterministic by queuing behind a blocked primer delivery; production batching behavior is unchanged. The old order-biased 1.5x synthetic timing comparison is now diagnostic-only after fresh CI reproduced its existingmasterflake; normal CI still requires both paths to drain under pressure and retains every functional ordering, exactly-once, reconnect, and coalescing assertion.This remains a merge-only step. Do not deploy the updated coordinator, publish or activate v0.7.5, or change the fleet version until the complete signed and notarized provider release has passed its final artifact gates. The intended release order is: merge the complete code, produce and verify the v0.7.5 artifacts, deploy the compatible coordinator, activate v0.7.5 for a canary, then expand the rollout.
Encrypted SSD KV-offload tier (merged from
feat/v075-ssd)ProviderCore/KVCacheSSD/— caching without occupying serving RAM. Reviewed and shipped under threat-model T-041 (the finding's sign-off text ships updated in this PR).GlobalKVCacheBudgetreservation per staged entry, with same-prefix requests attaching to it; the write-behind pipeline uses bounded host buffers and drops on pressure. Refused staging or dropped donation falls back to normal recompute without reducing live-serving capacity.~/Library/Caches/darkbloom/kv2/<modelKey>— a sibling of the legacykv/root the upgrade sweeper sheds; survival against the REALLegacyKVCacheSweeperis pinned by test.DARKBLOOM_PREFIX_CACHE_SSD=0kills the tier (and survives launchd restarts);DARKBLOOM_PREFIX_CACHE=0remains the master kill for every tier.EngineV2SlotFactory.makeProductionBridge): reslice grant → RAM-tier funding-gated carve (dormant ⇒ passthrough) → SSD tier construction → engine construction. The standalonedarkbloom localserver gets the tier through the same path.Live numbers (M4 Max, real weights, merged head):
Release gates (blocking; refreshed on the current merge head)
mlx-swift-lm/maintip; 728 tests green with compiled-decode cases excluded. The compiled-decode XCTest lane stalls on this host after loading the metallib, with no test failure emitted; this remains a documented local verification limitation. The tip change itself is isolated to Qwen3.5 VLM, outside the v0.7.5 served model familiesgo test ./... -race -count=1,go vet ./..., and coordinator build green · eslint 0 errors · vitest 456/456 · Next build green1e149580Release builds crashed ~2s after startup (SIGABRT in
swift_task_dealloc) — a Swift 6.3/macOS 26 toolchain issue with inlined genericTask.sleep(for:). Debug builds never crash; only the release-binary E2E gate caught it. This would have crash-looped the fleet on adoption. Fixed via a non-generictaskSleep()shim (all 21 call sites;TaskSleep.swiftdocuments it — revisit on the next toolchain bump).Review ledger
Every workstream passed an independent Claude review + Codex review (or refuted the Codex finding with a pinning test). Findings fixed along the way include: an unload-regrow race that could exceed the KV budget (regression-tested), a failed-load unwind that restored grants before the newcomer's weights died (regression-tested via weak-reference sampling), and a version-floor gap on self-route (coordinator side).
Notes for reviewers / ops
mlx-community/gpt-oss-20b-MXFP4-Q8in its HF cache.docsrunbook; deploy A must be live before the gemma shed lifts.feat/v073-multimodal) — no separate vision release.