Add hypervisor memory isolation for inference workloads#6
Merged
Conversation
Uses Apple Hypervisor.framework to create a lightweight VM with Stage 2 page tables that make inference memory invisible to RDMA DMA transfers. This upgrades the security model from software-enforced (PT_DENY_ATTACH, Hardened Runtime) to hardware-enforced (hypervisor page table isolation). Provider changes: - New hypervisor.rs module: VM lifecycle, buffer mapping, overlap detection with page-aligned guest physical addresses - security.rs: RDMA policy updated — RDMA is now allowed if hypervisor is active (enables future multi-node inference over Thunderbolt 5) - main.rs: VM created at startup before security posture check - coordinator.rs: Fresh hypervisor_active status in challenge-response - protocol.rs: New hypervisor_active field in AttestationResponse Coordinator changes: - provider.go: RDMA check now accepts RDMA+hypervisor combination - attestation.go: HypervisorActive field in attestation blob (alphabetical ordering preserved for SE signature compatibility) - messages.go: HypervisorActive in AttestationResponseMessage Validated experimentally: - Zero performance overhead (50.1 GB model, 100% memory coverage) - Rust+MLX FFI path produces correct output with hypervisor active - AES-256-GCM encryption at 42 GB/s (0.37ms for 16MB activation tensor) - Pipelined encryption hides crypto behind GPU compute (2.9% overhead) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
macOS 26 Hypervisor.framework requires hv_vm_map addresses and sizes to be 16 MB-aligned. The previous approach (mapping individual MLX buffers after allocation) failed for Metal Heap suballocations and small tensors, giving only ~56% coverage on quantized models. New approach: pre-allocate a large memory pool via mmap, VM-map it in 16 MB chunks at startup, then serve all Metal buffer allocations from this pool using makeBuffer(bytesNoCopy:). This gives 100% coverage for ALL model types — BF16, FP16, and quantized (4-bit, 8-bit). Validated experimentally: - 8 GB pool: 511 x 16MB chunks mapped in 0.2ms - Metal makeBuffer(bytesNoCopy:) works for all sizes (256B to 68MB) - GPU compute verified on pool memory - All quantization formats covered (uint32 packed weights, float16 scales) The pool exposes alloc_buffer(size) which returns page-aligned pointers within the VM-mapped region. MLX's inference engine creates Metal buffers on these pointers, ensuring all model weights, activations, and KV cache live in hardware-isolated memory. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The pool was previously sized to 90% of physical RAM — too aggressive and not model-aware. Now: - VM creation and pool allocation are separate steps: create_vm() at startup, allocate_pool() after model selection - Pool sized to 2x model file size (covers weights + activations + KV cache with headroom) - Added pool_has_capacity() for fail-closed enforcement: if the pool can't fit an allocation, the inference request must be refused rather than silently falling back to unprotected memory visible to RDMA This ensures no inference data ever leaks outside the VM-mapped pool, regardless of model size or quantization format. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Gajesh2007
added a commit
that referenced
this pull request
Apr 21, 2026
SSRF (pentest #6): Server-side proxy routes no longer accept x-coordinator-url from client requests. Coordinator URL is resolved exclusively from NEXT_PUBLIC_COORDINATOR_URL env var. Removes the attack vector where an attacker-controlled header redirected requests (including API keys) to arbitrary URLs. API key self-replication (pentest #5): Key creation (POST /v1/auth/keys) and device approval (POST /v1/device/approve) now require Privy JWT session auth via new requirePrivyAuth middleware — API keys are rejected. A leaked key can no longer create derivative keys or self-approve device links. Added DELETE /v1/auth/keys for key revocation. Session poisoning (pentest #8): Logout now clears darkbloom_api_key and darkbloom_coordinator_url from localStorage. Login resets coordinator URL to prevent pre-auth poisoning. CORS (pentest #17): Replaced wildcard Access-Control-Allow-Origin: * with configurable origin via CORS_ORIGIN env var, defaulting to https://console.darkbloom.dev.
Gajesh2007
added a commit
that referenced
this pull request
Jun 17, 2026
…dex #6) A changed APNs device token already forces a real re-challenge and drops the in-memory reuse record, but the PERSISTED row survived. With the Postgres store (prod), a coordinator restart before the forced re-challenge completed could reseed and reuse the pre-rotation proof. Delete the persisted row too, off the read loop, so the "token change forces a real re-challenge" invariant is durable across restarts. Adds Store.DeleteCodeAttestation (memory + postgres) and Server.invalidatePersistedCodeAttestation, wired into maybeRearmCodeAttest. Also corrects stale in-memory-store-in-prod comments (AGENTS.md, main.go). Tested: TestRearmChangedTokenDeletesPersistedReuse; full coordinator go test ./....
Gajesh2007
added a commit
that referenced
this pull request
Jun 17, 2026
The #6 fix deletes the persisted reuse row when a token rotation is OBSERVED via heartbeat on a live connection. But a token that rotates while the provider is DISCONNECTED (or between a blue/green restart and the next registration) is never seen, so after a restart reseed reuseAttestation(seKey, version) could mark the new connection code-attested without ever challenging the newly registered token. Bind the proof to the token: persist apns_token with the reuse row (additive migration), thread the token through recordAttested/reuseAttestation/seed, and require it to match the current registration token before reusing. A rotated token now falls through to a real challenge even across a restart. Records with no recorded token (legacy rows from before this change) still reuse, so the deploy does not trigger a fleet-wide re-push; those rows become token-bound on their next attestation and expire within the reuse window regardless. Not a fail-closed bypass (reuse is SE-key-bound, and the SE key is attested at registration, so only the genuine device reaches the reuse path) — defense in depth that makes token rotation always force re-verification. Tests: TestCodeAttestThrottleTokenBinding, TestSeededRowWithRotatedTokenForcesRealChallenge, + APNsToken round-trip in the store contract test. Full coordinator go test ./... green.
Gajesh2007
added a commit
that referenced
this pull request
Jun 17, 2026
…poch, late-SecurityInfo caching, MDM-config gate, 10m window, security docs FIX A (P1, durable hard-untrust — close the write-after-delete race; Codex trust_reuse.go:366 + Threat-Model #6): the persisted reuse WRITE was still fire-and-forget, so a hard untrust right after a grant could delete first and the async write could land a stale `hardware` row that a restart reseeds → fast-skip resurrects untrusted hardware. Added a per-provider hard-untrust EPOCH: registry.Provider.untrustEpoch (atomic.Uint64) + HardUntrustEpoch(), bumped in markUntrusted's HARD path BEFORE the delete hook fires. recordTrustReuse now captures the epoch at grant time, persists SYNCHRONOUSLY (no saferun.Go), and immediately before the upsert re-checks the provider is not hard-untrusted AND the epoch is unchanged — else it skips the persist and drops the in-memory entry. markUntrusted's invalidate stays synchronous. Deterministic tests (registry epoch counter; record-after-untrust skips persist; durable-across-restart both orderings). FIX B (P2, cache late grants; Codex provider.go:2755): ApplyLateSecurityInfo now mirrors the synchronous path with the same epoch-checked write-through, so a self_signed→hardware upgrade from a late SecurityInfo also gets restart-survivable fast-skip. Test added. FIX C (P2, gate on MDM configured; Codex trust_reuse.go:438): tryTrustReuseFastSkip returns false when s.mdmClient == nil — a no-MDM/misconfigured deploy must not grant hardware from a (possibly stale) cache with no live MDM fallback. Test added. FIX D (Threat-Model #3, window): default reuse window 30m→10m so hardware-trust reuse cannot span a SIP-disable reboot cycle; still configurable via EIGENINFERENCE_TRUST_REUSE_WINDOW. Comment updated. FIX E (Threat-Model #5, table doc): annotated the provider_trust_reuse migration as security-sensitive — a row grants a hardware fast-skip, so write access must be guarded like the payment ledger; SeedTrustReuseCache trusts DB contents, bounded by the always-run live SE challenge + future-date rejection. No schema change. FIX F (Threat-Model SEC-004, doc): noted near recordTrustReuse/SeedTrustReuseCache that a forged (localhost-bound, unauthenticated) MDM webhook driving a grant would be durably persisted + reseeded → amplified across restarts; bounded by the localhost-only webhook, full mitigation is authenticating the webhook (tracked separately). coordinator/: gofmt clean, go build ./... ok, go test ./... all pass (postgres tests skipped w/o DATABASE_URL), -race clean on the new tests.
Gajesh2007
added a commit
that referenced
this pull request
Jun 17, 2026
…on coordinator restart/swap) (#391) * feat(attestation): DAR-326 Phase 0 — provider trust-reuse cache (skip MDM re-verify herd on restart) A planned coordinator restart/blue-green swap reconnects the whole fleet at once; today each provider's per-connection mdmVerificationLoop fires a live MDM SecurityInfo round-trip (+ APNs push) almost immediately, so the fleet hammers MicroMDM/APNs in a herd. This adds a coordinator-side, RDS-persisted "trust-reuse" cache (mirroring the existing code-identity reuse cache) that lets a reconnecting, recently-fully-verified provider be granted hardware from its record once a FRESH live SE challenge re-proves identity + posture — skipping the live MDM round-trip. Store layer (mirrors code_attestations): - store.ProviderTrustReuse + List/Upsert/Delete in interface.go, postgres.go (provider_trust_reuse table), memory.go. Cache (api/trust_reuse.go, mirrors code_attest_throttle.go): - trustReuseCache with reuseTrust / hasFreshRecord / recordTrust / invalidateReuse / seed; window via EIGENINFERENCE_TRUST_REUSE_WINDOW (default 30m). Server wiring: SeedTrustReuseCache (seed + write-through + hard-untrust hook), recordTrustReuse, persistTrustReuse, invalidateTrustReuse, tryTrustReuseFastSkip, awaitTrustReuseGrant. Seeded at startup in main.go. Write-through: verifyProviderViaMDM records a reuse entry after a live MDM pass + hardware grant. Read-path fast-skip: after the live SE challenge passes (verifyChallengeResponse), grant hardware iff ALL gates hold: (a) SE-key+serial match, (b) fresh signed binary_hash == cached, (c) fresh SIP+SecureBoot good && status fields signed, (d) within window, (e) not hard-untrusted. Any miss falls through to the unchanged full live MDM verify. The mdmVerificationLoop briefly defers to this fast-skip only for candidates (hasFreshRecord), so the live round-trip is actually skipped instead of racing ahead. Invalidation: a registry hard-untrust hook (SetHardUntrustHook, fired in markUntrusted only for non-recoverable deroutes) drops the record in-memory + persisted, keeping "hard untrust always takes effect" durable across restarts. Security invariants: the live SE challenge ALWAYS runs first; the skip is a gated optimization, never a trust shortcut; first-ever verification (no record) always does full live MDM; persisted rows are re-validated on every read. Tests: cache gates + window + seed + env (mirrors code_attest_throttle_test), store round-trip (mirrors code_attest_test), fast-skip grant + every fall-through case, and the registry hard/transient untrust hook. * fix(dar-326): address review findings — durable invalidation, clock-skew guard, fast-skip latency, test gaps, hook decoupling FIX 1 (should-fix) — durable hard-untrust invalidation: invalidateTrustReuse now does a synchronous in-memory invalidateReuse plus an INLINE bounded-retry DeleteProviderTrustReuse (3 attempts, 5s timeout each, short backoff, log only after the final failure) instead of a fire-and-forget saferun.Go. Safe because the hard-untrust hook fires off all registry locks (registry.markUntrusted) and a hard untrust is rare; this keeps "hard untrust always takes effect" durable across a coordinator restart even through a transient DB blip. (Parity for invalidatePersistedCodeAttestation deliberately NOT applied: it runs on the WebSocket read loop via maybeRearmCodeAttest, where an inline blocking delete would stall reads — only the trust-reuse hook is guaranteed off-locks.) FIX 2 (nit) — clock-skew guard: reuseTrust/hasFreshRecord/seed now require age in [-clockSkewTolerance, reuseWindow) (clockSkewTolerance = 2m), rejecting implausibly future-dated (corrupt/forged) records that would otherwise stay "fresh" indefinitely. FIX 3 (nit) — fast-skip latency: added a per-connection buffered "challenge settled without fast-skip grant" signal on registry.Provider (SignalChallengeSettled/ChallengeSettledChan, created fresh per Register). It is fired in verifyChallengeResponse right after tryTrustReuseFastSkip returns false, and awaitTrustReuseGrant now selects on it — so a non-fast-skip candidate proceeds to the full live MDM verify immediately instead of stalling up to 10s. FIX 4 (nit) — tests: added serial-mismatch and SE-key-mismatch fall-through cases to TestTrustReuseFastSkipFallsThrough. FIX 5 (nit) — hook decoupling: SeedTrustReuseCache now wires the hard-untrust hook UNCONDITIONALLY (independent of store presence); store write-through + startup seeding stay behind the store check. Doc updated. (5b) a successful fast-skip grant now clears any stashed pending-ACME result, mirroring a normal upgrade. Tests: extended TestInvalidateTrustReuseDeletesPersisted (now asserts synchronous delete) and added TestInvalidateTrustReuseRetriesPersistedDelete (flaky store, retry-then-succeed) + TestInvalidateTrustReuseDurableAcrossRestart (record → hard untrust → fresh cache seeded from same store finds nothing); future-dated cases in the cache + seed tests; await-signal returns-promptly + returns-true-on- hardware; hook-wired-without-store. gofmt clean, go build ./... ok, go test ./... all pass (postgres tests skipped w/o DATABASE_URL), -race clean on new tests. * fix(dar-326): address Codex + threat-model review — durable untrust epoch, late-SecurityInfo caching, MDM-config gate, 10m window, security docs FIX A (P1, durable hard-untrust — close the write-after-delete race; Codex trust_reuse.go:366 + Threat-Model #6): the persisted reuse WRITE was still fire-and-forget, so a hard untrust right after a grant could delete first and the async write could land a stale `hardware` row that a restart reseeds → fast-skip resurrects untrusted hardware. Added a per-provider hard-untrust EPOCH: registry.Provider.untrustEpoch (atomic.Uint64) + HardUntrustEpoch(), bumped in markUntrusted's HARD path BEFORE the delete hook fires. recordTrustReuse now captures the epoch at grant time, persists SYNCHRONOUSLY (no saferun.Go), and immediately before the upsert re-checks the provider is not hard-untrusted AND the epoch is unchanged — else it skips the persist and drops the in-memory entry. markUntrusted's invalidate stays synchronous. Deterministic tests (registry epoch counter; record-after-untrust skips persist; durable-across-restart both orderings). FIX B (P2, cache late grants; Codex provider.go:2755): ApplyLateSecurityInfo now mirrors the synchronous path with the same epoch-checked write-through, so a self_signed→hardware upgrade from a late SecurityInfo also gets restart-survivable fast-skip. Test added. FIX C (P2, gate on MDM configured; Codex trust_reuse.go:438): tryTrustReuseFastSkip returns false when s.mdmClient == nil — a no-MDM/misconfigured deploy must not grant hardware from a (possibly stale) cache with no live MDM fallback. Test added. FIX D (Threat-Model #3, window): default reuse window 30m→10m so hardware-trust reuse cannot span a SIP-disable reboot cycle; still configurable via EIGENINFERENCE_TRUST_REUSE_WINDOW. Comment updated. FIX E (Threat-Model #5, table doc): annotated the provider_trust_reuse migration as security-sensitive — a row grants a hardware fast-skip, so write access must be guarded like the payment ledger; SeedTrustReuseCache trusts DB contents, bounded by the always-run live SE challenge + future-date rejection. No schema change. FIX F (Threat-Model SEC-004, doc): noted near recordTrustReuse/SeedTrustReuseCache that a forged (localhost-bound, unauthenticated) MDM webhook driving a grant would be durably persisted + reseeded → amplified across restarts; bounded by the localhost-only webhook, full mitigation is authenticating the webhook (tracked separately). coordinator/: gofmt clean, go build ./... ok, go test ./... all pass (postgres tests skipped w/o DATABASE_URL), -race clean on the new tests. * fix(dar-326): address review round 2 — empty-seKey guard, post-write untrust recheck, queue drain, ACME handling, MDA-freshness doc FIX 1 (MUST — empty-seKey poisoning, a bug FIX B introduced): recordTrustReuse is now a hard no-op for an empty seKey OR empty binaryHash (no in-memory entry, no persisted row) — an unkeyed row would poison the cache/store and an empty binary hash can never satisfy the read gate. ApplyLateSecurityInfo nil-guards the derived values and skips recordTrustReuse when AttestationResult is absent or the SE key / binary hash is empty. FIX 2 (MUST — write-delete TOCTOU, Codex P1 trust_reuse.go:393): the pre-write epoch check left a check-then-write window where a hard untrust landing between the check and the upsert committing could resurrect a stale row. recordTrustReuse now adds a POST-write epoch recheck: after a successful upsert, if the provider is hard-untrusted or the epoch changed, it deletes the just-written row (bounded-retry idempotent helper, now shared with invalidateTrustReuse) and drops the in-memory entry. With markUntrusted's own synchronous delete this fully linearizes write vs delete: untrust-before-precheck is dropped, untrust-during-write is compensated, untrust-after is removed by its own delete. FIX 3 (SHOULD — drain queue after fast-skip, Codex P2): on a fast-skip hardware grant, verifyChallengeResponse now drains the provider's queued requests (DrainQueuedRequestsForProvider, off the challenge goroutine) — mirroring the code-attest / MDM grant path — so freshly-routable capacity picks up queued work instead of waiting for the next heartbeat / 120s timeout. FIX 4 (SHOULD — ACME interaction, Codex P2 + Threat Model): (a) reconcileACMEAfterFastSkip — after an MDM-reuse grant, if a connect-time ACME cert is stashed but never bound, try the binding once; if it still doesn't bind, clear the optimistic ACMEVerified flag and discard the stale pending result so the attestation report never claims acme_verified for an unproven binding. (b) on a fast-skip MISS, retryACMETrust runs BEFORE SignalChallengeSettled, and the settled signal is fired only if ACME did not promote to hardware — so a bound device cert can promote without forcing a live MDM round-trip. (c) connection-scope the challengeSettled signal: Provider.ResetChallengeSettled() drains it, called from Register, so no stale buffered signal can carry into a new connection. FIX 5 (DOC — Threat Model MDA freshness): documented in docs/threat-model.yaml (T-036, ref TB-005) and in a code comment at the fast-skip grant that the fast-skip skips live MDA cert-chain re-verification within the reuse window — an accepted trade-off for the restart-herd problem, bounded by the short window, the always-run live SE challenge, the hard-untrust epoch, and the MDM-configured gate. coordinator/: gofmt clean, go build ./... ok, go test ./... all pass (postgres tests skipped w/o DATABASE_URL), -race clean on api + registry.
Gajesh2007
added a commit
that referenced
this pull request
Jul 2, 2026
…probs buffer, sanity guards Security/hardening findings from PR #499 review (ethenotethan): #4 request-id validation: a caller-supplied request-id is validated (non-empty, ≤256 chars, no ASCII control chars) before it becomes a dictionary key / cancel-correlation handle; nil/invalid ids normalize to a fresh req-<uuid>. Static normalizedRequestId/isValidRequestId are pure and unit-tested. This also subsumes #11 (duplicate-id): prod ids are provider-minted UUIDs, and any malformed id is now normalized to a fresh UUID before the existing duplicate guard, so an attacker cannot induce a collision — no separate change needed. #5 nextRawId overflow: += → &+= (wrapping) with a comment; 2^64 is unreachable in practice, this makes the theoretical overflow explicitly defined rather than a trap. #6 engine.submit on the actor: verified O(1) non-blocking enqueue against the CBv2Engine contract (admission check + register + enqueue, no forward pass; tokenization already off-actor) and documented it at the call site. No code change. #7 pump task lifecycle: per-request pump Tasks are now tracked in pumpTasks and cancelled by shutdown() (before the engine drain), so none outlives the bridge; AsyncStream cancellation unwinds each pump through its teardown path, releasing per-request KV. Tested: shutdown cancels a live pump and its reservation returns to 0. #8 bounded logprobs buffer: EngineV2LogprobsChannel caps at 4096 entries with drop-oldest eviction + a droppedCount and a one-line WARN on first drop, so a stalled/never-draining consumer can't grow it unboundedly. Dropping logprobs never affects content/billing/hash (out-of-band). #9 dropped logit_bias keys: parseLogitBiasCountingDropped returns the count of dropped (non-numeric/negative) keys; samplingParams emits a count-only WARN (never the keys — request content) instead of dropping silently. #10 kvBytesCapacity sanity: clampKVBytesCapacity caps the v2 admission ceiling at physical RAM so a mis-derived budget degrades to all-of-memory rather than an absurd ceiling. Tests: id validation matrix + normalization-in-submit, wrapping id, shutdown-cancels-live-pumps, logprobs cap drop-oldest + droppedCount, dropped-key count, and the capacity clamp. Full suite green (1245).
Gajesh2007
added a commit
that referenced
this pull request
Jul 2, 2026
…probs buffer, sanity guards Security/hardening findings from PR #499 review (ethenotethan): #4 request-id validation: a caller-supplied request-id is validated (non-empty, ≤256 chars, no ASCII control chars) before it becomes a dictionary key / cancel-correlation handle; nil/invalid ids normalize to a fresh req-<uuid>. Static normalizedRequestId/isValidRequestId are pure and unit-tested. This also subsumes #11 (duplicate-id): prod ids are provider-minted UUIDs, and any malformed id is now normalized to a fresh UUID before the existing duplicate guard, so an attacker cannot induce a collision — no separate change needed. #5 nextRawId overflow: += → &+= (wrapping) with a comment; 2^64 is unreachable in practice, this makes the theoretical overflow explicitly defined rather than a trap. #6 engine.submit on the actor: verified O(1) non-blocking enqueue against the CBv2Engine contract (admission check + register + enqueue, no forward pass; tokenization already off-actor) and documented it at the call site. No code change. #7 pump task lifecycle: per-request pump Tasks are now tracked in pumpTasks and cancelled by shutdown() (before the engine drain), so none outlives the bridge; AsyncStream cancellation unwinds each pump through its teardown path, releasing per-request KV. Tested: shutdown cancels a live pump and its reservation returns to 0. #8 bounded logprobs buffer: EngineV2LogprobsChannel caps at 4096 entries with drop-oldest eviction + a droppedCount and a one-line WARN on first drop, so a stalled/never-draining consumer can't grow it unboundedly. Dropping logprobs never affects content/billing/hash (out-of-band). #9 dropped logit_bias keys: parseLogitBiasCountingDropped returns the count of dropped (non-numeric/negative) keys; samplingParams emits a count-only WARN (never the keys — request content) instead of dropping silently. #10 kvBytesCapacity sanity: clampKVBytesCapacity caps the v2 admission ceiling at physical RAM so a mis-derived budget degrades to all-of-memory rather than an absurd ceiling. Tests: id validation matrix + normalization-in-submit, wrapping id, shutdown-cancels-live-pumps, logprobs cap drop-oldest + droppedCount, dropped-key count, and the capacity clamp. Full suite green (1245).
Gajesh2007
added a commit
that referenced
this pull request
Jul 3, 2026
…ma 4 + GPT-OSS 20B) (#499) * feat(engine-v2): provider bridge for ContinuousBatchingV2 behind DARKBLOOM_ENGINE_V2 - EngineV2Config: env DARKBLOOM_ENGINE_V2 / [backend] engine_v2 key selection, gemma-4*/gpt-oss* per-model allowlist (DARKBLOOM_ENGINE_V2_MODELS override), EngineV2Factory with safe legacy fallback + WARN engine_health telemetry (operation=engine_v2_fallback) on v2 init failure - EngineV2Bridge: adapts CBv2Engine to the existing GenerationEvent surface; event framing mirrors BatchScheduler+EngineBridge (chunk/info/error, abort usage-before-error, teardown sentinel); billing-zero max() defense and the legacy decode-TPS methodology/EWMA - EngineV2Bridge+Translation: pure ChatRequest -> CBv2Request translation (temperature ?? 0.0 legacy default, logit_bias string-key id parsing, logprobs/top_logprobs mapping, seed passthrough, buildStopTokenIds-semantics stop resolution, max_tokens defaulting); CBv2KVError.capacityExhausted -> canonical token_budget_exhausted: string (429/503 classification unchanged) - EngineV2Bridge+Capacity: CBv2CapacitySnapshot -> existing BackendSlotCapacity fields (no protocol change), truthful bytes-derived token budgets, WedgeMonitor reuse + engine_v2 step_wedge transition telemetry - EngineV2Runtime: process-wide bridge registry; ProviderLoop capacity heartbeat folds v2 slots, handleCancellation fans request-id out to CBv2Engine.cancel; both no-ops when the flag is off - ChatCompletionRequest: additive optional logit_bias/logprobs/top_logprobs (v2 path only; legacy ignores them) - docs/engine-v2/CONTRACT-ISSUES-H-provider.md: contract gaps + chosen shapes - submodule: pin libs/mlx-swift-lm at the CBv2 contracts commit (fe60e17) * test(engine-v2): bridge unit suite — scripted CBv2Engine stub, no models/network 34 tests in 7 suites (swift test --filter EngineV2): - translation: field-by-field CBv2Request mapping, legacy greedy defaults, logit_bias string-key parsing, logprobs/top_logprobs mapping, buildStopTokenIds-semantics stop resolution + construction-time stamping - event framing: fixture-compare against the recorded legacy BatchScheduler+EngineBridge stream shape (chunks -> single info -> finish; abort usage-before-error; teardown sentinel; billing-zero max() defense; tokenize failure short-circuits before the engine) - error mapping: capacityExhausted -> token_budget_exhausted (retryable .tokenBudgetExhausted class), backendIneligible/unknown -> generationFailed - cancellation: provider id -> minted CBv2RequestID, unknown-id no-op, EngineV2Runtime fan-out (ProviderLoop hook path) - capacity: snapshot -> BackendSlotCapacity (bytes-derived budgets, idle/ running states, wedge counters), runtime aggregate summary - config gating: env/config flag matrix, allowlist + env override, factory legacy selection without invoking the builder, init-failure fallback with engine_v2_fallback telemetry, engine_v2 TOML key decode * fix(engine-v2): duplicate request-id guard + deterministic fan-out test - EngineV2Bridge.submitTokenized: reject a second submit under an active request id with the legacy planner's canonical message ('token_budget_exhausted: duplicate request ID' -> .requestRejected, 400) before minting an engine id, so two pumps can never share bookkeeping - test: duplicate-id rejection classification + engine isolation - test: runtimeFanOut holds the submitted stream alive for the whole test; dropping it fired onTermination(.cancelled) and raced a second (idempotent) engine cancel into the count assertions * feat(engine-v2): wire ContinuousBatchingV2 into production serving paths Closes the P1 integration gap: the EngineV2 bridge (DARKBLOOM_ENGINE_V2 / engine_v2) was implemented and tested but never instantiated outside tests — model loading always built only the legacy BatchScheduler, both request paths captured slot.scheduler directly, and the EngineV2Runtime registry stayed permanently empty. - Model load (ProviderLoop+ModelLoading -> new ProviderLoop+EngineV2): when EngineV2Config selects v2 for a (non-VLM) model, assemble the real CBv2 engine over the already-loaded container via the new EngineV2Factory.makeProductionEngine (Gemma4/GPT-OSS cbv2LayerKinds + newCacheV2, contiguous KV backend sized from the unified-memory KV budget, text detokenizer over the model tokenizer), register the bridge with EngineV2Runtime, and store it on ModelSlot alongside (never replacing) the legacy scheduler. - Routing: MultiModelBatchSchedulerEngine registry entries and AcquiredModel carry the optional bridge; streamChatCompletion submits the same tokenized prompt through it when present (translated sampling/ stop params — v2 additionally honors penalties + stop strings the legacy submit drops) and cancels through the owning engine. The coordinator inference handler and the unified local endpoint both pass slot.engineV2; standalone --local mode intentionally stays legacy. - Zero-overhead flag-off: capacity heartbeats and cancellation fan-out consult the runtime only when at least one v2 slot exists (previously an unconditional empty-registry actor hop). A v2-served slot reports capacity through its bridge ONLY (the dormant legacy scheduler is skipped so the model is never advertised twice). unloadModel unregisters and drains the bridge. - Failure semantics now exercised in production: v2 init throw -> WARN engine_health (engine_v2_fallback) + legacy fallback; VLM slots skip selection silently (no per-load WARN for permanently-unsupported shapes). - Pin libs/mlx-swift-lm to b2b6d53 (integrated CBv2 engine) — the public EngineV2 assembly the production factory constructs did not exist at the previously pinned SHA. Tests (EngineV2ProductionWiringTests, live-isolated, no weights/network): slot-factory flag-off/allowlist/VLM-gate/init-failure + runtime registration, both request-routing shapes through a scripted CBv2Engine, cancellation propagation to the engine-minted id, runtime-guard zero-consult, heartbeat fold-in with the no-double-report regression, unload retirement, and the production factory throw paths. 50 EngineV2 tests green; full suite 1214 tests / 96 suites green. * chore(submodule): bump mlx-swift-lm to engine-v2 final tip 8662159 Pulls the CBv2 contract updates the provider bridge adopts next: - CBv2Engine is now Sendable (contract-issue H§1 fix) - CBv2CapacitySnapshot.stepsExecuted monotonic step counter (H§3 fix) - CBv2Event.delta logprobs populated when sampling.topLogprobs > 0 (E§1) - CBv2Request.cacheSalt per-request prefix-cache scope (TB-007) - prefix-cache donation skip for quantized KV; shutdown timeout hardening * feat(engine-v2): adopt final CBv2 contract — drop engine box, real step counter, logprobs + cacheSalt plumbing Follow-ups on the engine-v2 bridge against mlx-swift-lm tip 8662159: - Drop EngineV2EngineBox: CBv2Engine is Sendable in the contract now, so the bridge actor holds and calls the engine directly (H§1 resolved). - Wedge signal from the engine's own counter: WedgeMonitor now samples CBv2CapacitySnapshot.stepsExecuted (published every step) instead of the bridge's event-count proxy (H§3 resolved). New capacity test pins the passthrough plus the flatline-under-hanging-admits wedge verdict. - Logprobs passthrough (coordinator path): engine delta logprobs are converted to the OpenAI streaming entry shape and ride out-of-band via a per-request EngineV2LogprobsChannel (bridge pump appends before the chunk yield); the inference handler splices them into the next content-bearing SSE chunk as choices[0].logprobs = {content: [...]} before encryption. The legacy engine NEVER emitted logprobs, so this is the OpenAI-standard shape, not a legacy match; the standalone --local path (frames served inside the upstream router) still emits none. logprobs/top_logprobs are decoded from the sealed body (the upstream request type does not model them) like reasoning_effort. - cacheSalt forward plumbing (inert): the per-tenant cache scope (SHA256(prompt_cache_key)/SHA256(user), the same value the legacy scheduler threads into the checkpoint cache) now maps onto CBv2Request.cacheSalt; "" maps to nil (cache-level salt fallback). Production still constructs the v2 engine with prefixCache: nil. Tests: scripted-stub coverage for channel ordering/conversion, SSE splice shape + skip rules, extractLogprobsSpec, cacheSalt mapping on both bridge entry points, and engine wiring threading scope+plumbing. * chore(submodule): bump mlx-swift-lm to engine-v2 final tip a0ae73b (paged eligibility guard + bench driver) * chore(submodule): bump mlx-swift-lm to engine-v2 composed tip 255b223 (compiled decode + kernel-opt + profiler) * fix(engine-v2): thread sealed-body logit_bias + seed through the coordinator v2 translation The upstream OpenAIChatCompletionRequest models neither logit_bias nor seed, so the coordinator path's internal ChatCompletionRequest always carried nil for both and EngineV2Translation silently dropped them (PR #499 review, Codex P2 #1). Recover them the same way logprobs / reasoning_effort / cache scope are recovered: decode straight from the E2E-sealed body (ProviderLoop.extractSamplingOverrides, field-independent probes) and overlay onto the v2 translation via a new EngineV2SamplingOverrides plumbing struct. v2 engine path only — the legacy engine never honored either knob and its submit call is byte-identical; the --local path still drops both (no provider seam, documented in the narrowed KNOWN DEVIATION). Tests: sealed-body extraction (both fields, absence, malformed-field independence), translate overlay → CBv2SamplingParams, and a coordinator-path wiring test proving the parsed bias + seed reach the stub CBv2Engine. * fix(engine-v2): truthful shared-KV accounting + fp16 sizing for kv_quant models Two coupled memory-accounting gaps from PR #499 review (Codex P2 #2 + #3): Fix #2 (shared-budget KV accounting): the v2 bridge bypassed the shared GlobalKVCacheBudget, so model-load admission (availableMemoryGb minus outstanding reservations) and the legacy live-KV gate under-counted when a v2 slot was active alongside a legacy slot — a concurrent legacy load could over-commit unified memory. The v2 engine runs its own optimistic + preemption admission against a private byte ledger, so a per-request GATING reservation would fight that design (and double-count against the engine's ledger). Instead each in-flight v2 request records its worst-case KV footprint (prompt + maxTokens, fp16 rate) in the shared ledger via a new bookkeeping-only recordEngineKV (never rejects — the v2 engine stays authoritative for v2 acceptance), released on every terminal/teardown path in the pump. outstandingReservedBytes() now truthfully includes v2 in-flight KV; the v2 engine's own ledger is unchanged. Fix #3 (kv_quant fp16 sizing): engine_v2 builds UNQUANTIZED CBv2LayerCache regardless of kv_quant (makeProductionEngine never builds a quantized cache), but the slot factory sized the bridge from the scheduler's live kvBytesPerToken — the QUANTIZED rate when kv_quant=true — overstating v2 token budgets/heartbeats 2–4x and under-sizing the fix-#2 reservation. EngineV2KVSizing.resolve now picks the fp16 rate (scheduler fp16KVBytesPerToken) for v2 sizing and emits a WARN engine_health telemetry (engine_v2_kv_quant_unsupported) once per load when kv_quant actually engaged for the model. Tests: sizing decision (both kv_quant configs), record+release invariant (finish and no-terminal teardown), no-op when rate unknown, and the slot factory's fp16 sizing + WARN end-to-end via a new BatchScheduler rate seam. * harden(engine-v2): request-id validation, pump lifecycle, bounded logprobs buffer, sanity guards Security/hardening findings from PR #499 review (ethenotethan): #4 request-id validation: a caller-supplied request-id is validated (non-empty, ≤256 chars, no ASCII control chars) before it becomes a dictionary key / cancel-correlation handle; nil/invalid ids normalize to a fresh req-<uuid>. Static normalizedRequestId/isValidRequestId are pure and unit-tested. This also subsumes #11 (duplicate-id): prod ids are provider-minted UUIDs, and any malformed id is now normalized to a fresh UUID before the existing duplicate guard, so an attacker cannot induce a collision — no separate change needed. #5 nextRawId overflow: += → &+= (wrapping) with a comment; 2^64 is unreachable in practice, this makes the theoretical overflow explicitly defined rather than a trap. #6 engine.submit on the actor: verified O(1) non-blocking enqueue against the CBv2Engine contract (admission check + register + enqueue, no forward pass; tokenization already off-actor) and documented it at the call site. No code change. #7 pump task lifecycle: per-request pump Tasks are now tracked in pumpTasks and cancelled by shutdown() (before the engine drain), so none outlives the bridge; AsyncStream cancellation unwinds each pump through its teardown path, releasing per-request KV. Tested: shutdown cancels a live pump and its reservation returns to 0. #8 bounded logprobs buffer: EngineV2LogprobsChannel caps at 4096 entries with drop-oldest eviction + a droppedCount and a one-line WARN on first drop, so a stalled/never-draining consumer can't grow it unboundedly. Dropping logprobs never affects content/billing/hash (out-of-band). #9 dropped logit_bias keys: parseLogitBiasCountingDropped returns the count of dropped (non-numeric/negative) keys; samplingParams emits a count-only WARN (never the keys — request content) instead of dropping silently. #10 kvBytesCapacity sanity: clampKVBytesCapacity caps the v2 admission ceiling at physical RAM so a mis-derived budget degrades to all-of-memory rather than an absurd ceiling. Tests: id validation matrix + normalization-in-submit, wrapping id, shutdown-cancels-live-pumps, logprobs cap drop-oldest + droppedCount, dropped-key count, and the capacity clamp. Full suite green (1245). * fix(engine-v2): record shared-KV reservation atomically with admission; truthful kv_quant WARN Addresses both reviewers of the PR #499 fixes: Codex (race): the shared-budget KV reservation was recorded in the detached pump task, which runs AFTER submitTokenized returns — leaving a window where a concurrent model-load gate could observe zero v2 KV between engine admission and the pump starting. Make submit/submitTokenized async (all call sites already await) and record the reservation SYNCHRONOUSLY right after engine.submit succeeds, before returning the stream. The pump now only releases. Tests assert the no-gap invariant (reservation visible the instant submit returns, no polling). Claude (truthfulness): move the engine_v2_kv_quant_unsupported WARN below the makeBridgeIfSelected guard so it fires only when the v2 bridge actually built. On a v2 init-failure fallback the model serves via legacy, which DOES honor kv_quant, so warning there was untruthful. Full suite green (1245). * chore(submodule): bump mlx-swift-lm to engine-v2 tip e4158bc (PR#62 review fixes) * fix(engine-v2): gate v2 admission on the shared KV budget before engine submission The bridge now RESERVES each request's worst-case KV footprint in the process-wide GlobalKVCacheBudget before engine.submit — atomically on the budget actor (check + record in one hop) — and rejects with the canonical token_budget_exhausted capacity error when the shared pool has no headroom, exactly like the legacy scheduler's KV-reserve rejection. Engine rejections after the gate release the reservation. The non-gating recordEngineKV is removed (the gating reserve subsumes it). Degenerate maxTokens<=0 requests skip the gate (the engine finishes them without allocating KV). Round-2 PR#499 P2: with another slot's live KV reserved, record-after-accept could overcommit unified memory on multi-slot providers. * fix(engine-v2): report committed worst-case tokens in heartbeat budget fields The coordinator's token-budget gate (activeTokenBudgetUsed + queuedTokenBudget + requestTokens <= activeTokenBudgetMax) expects 'used' to carry the committed worst-case reservation of accepted requests — the legacy scheduler's Σ(prompt + maxTokens) — and never consults max_tokens_potential. Feeding it materialized KV bytes let the coordinator over-route while accepted long-max_tokens requests still had unprotected growth. Mapping (existing wire fields only, documented in backendSlotCapacity): activeTokens stays engine truth (real KV-resident tokens); activeTokenBudgetUsed and maxTokensPotential carry Σ(prompt+maxTokens); activeTokenBudgetMax is the engine byte capacity / fp16 rate (0 when the rate is unknown, disengaging the gate). Round-2 PR#499 P2. * fix(engine-v2): stable (seed, prompt)-derived engine ids for reproducible seeded sampling The v2 sampler keys its RNG on (seed, requestID.raw, stepIndex), so minting a fresh monotonic id per submission made identical seeded requests sample differently depending on prior traffic. Seeded submissions now derive their engine id from a SplitMix64 chain over (seed, promptTokens), tagged with bit 63 to stay disjoint from the monotonic family; the id is minted in the same synchronous stretch as engine.submit so the liveness collision check cannot race a concurrent identical submission. An identical seeded request that is still live falls back to a fresh id (documented waiver); batching reproducibility remains best-effort per the engine contract. Test: stochastic sampler stub keyed exactly like SamplerV2 proves same-seed + same-prompt submissions reproduce identical output across interleaved traffic, and diverge on different seed/prompt. Round-2 PR#499 P2. * fix(engine-v2): size v2 KV admission ceilings against fleet-wide residency A v2 engine's kvBytesCapacity was derived as if only its own slot's weights were resident, so multi-slot providers could grant Σ(ceilings) far past the unified-memory KV budget. The slot factory now derives the ceiling with EngineV2KVSizing.engineKVBytesCapacity: kvBudgetBytes over ALL resident models' weights (co-resident slots + the new model) minus the construction-fixed ceilings already granted to existing v2 engines. When nothing remains the capacity is 0 and makeProductionEngine throws noKVHeadroom, so the slot serves via the legacy scheduler (whose per-request shared-budget gate — and now the v2 bridge's too — needs no static ceiling). Test hooks now receive the computed capacity ((modelId, kvBytesCapacity)) so wiring tests assert the two-slot derivation end-to-end; pure sizing tests pin the sum-within-budget invariant and degenerate clamps. Round-2 PR#499 P2. * docs(engine-v2): hardening-review rationales — pump-task bound, logprobs lock, cancel scan - pumpTasks: bounded by the engine's own admission (maxConcurrent + maxWaiting, ~68 in production) — pump tasks exist only for engine-accepted requests and self-clear on terminal; cite the contract. - EngineV2LogprobsChannel: why an uncontended NSLock beats actor isolation for a two-party bounded append/swap on the delta path. - EngineV2Runtime.cancel: O(n) scan is over v2-served MODELS (<= slot cap, single digits), not requests; an id->bridge index would add hot-path register/unregister traffic to save a 3-entry scan on a rare event. * fix(engine-v2): duplicate re-check across the gate suspension; scope KV release to the reserving request The shared-budget reserve introduced the bridge's first suspension between the duplicate-id guard and the bookkeeping install. A concurrent same-id submission that SKIPS the gate (degenerate maxTokens / unknown rate) could slip in across that gap and be overwritten; and the pump's unconditional release could drop a same-keyed reservation owned by a different submission. Now: re-check active[id] after the reserve (roll back + reject as duplicate), and the pump releases only when its own submission reserved (holdsSharedReservation). Degenerate maxTokens<=0 requests are covered by a test proving they skip the gate even on an exhausted pool. * chore(submodule): bump mlx-swift-lm to engine-v2 tip ef1ad12 (bench cells on top of e4158bc) * chore(submodule): bump mlx-swift-lm to engine-v2 tip ec541f5 (PR#62 round-2 review fixes) * feat(provider): config keys for startup preload + auto-update rollover jitter New provider.toml keys (all additive; defaults chosen to be fleet-safe): [backend] - startup_preload (default true): preload models before coordinator registration on boot. A fresh install has no persisted serving set, so out-of-the-box behavior is unchanged until a model has been served. - preload_models (default []): explicit preload list, operator order; empty means "the models served before the last restart". - startup_preload_timeout_secs (default 120): upper bound on how long registration is deferred while the preload runs. - startup_selftest (default true): 1-token greedy decode through the serving path after each preload. - startup_selftest_fail_closed (default false): retire a model whose self-test failed instead of keeping it advertised (fail-open). [provider] - update_jitter_seconds (default 300): max random delay between staging a verified auto-update bundle and beginning the drain+restart. * feat(provider): startup model preload + registration readiness gate Fixes the v0.6.30-class cold-restart failure mode: after a release restart the provider registered (and attracted routing) with NOTHING loaded, so first requests paid the full ~26 GB weight load + engine build inside a live request — and at a fleet rollover every box was cold at once (first_chunk_timeout storm). - LoadedModelsStore: persists the live serving set to ~/.darkbloom/loaded-models.json (data dir — survives auto-updates) on every load and every NON-shutdown unload (idle timeout, eviction, retirement). Shutdown teardown deliberately preserves it so a stop/update/restart remembers what was being served. - StartupPreloader: sequential, deps-injected preload driver. Memory admission never evicts — a candidate that does not fit what is free is skipped with a WARN and left to the lazy-load path; failed loads and self-tests degrade, never crash. - ProviderLoop.runStartupPreloadGate(): runs in run() BEFORE the coordinator client exists. Plan = preload_models (operator order) or the persisted set (biggest first), filtered to advertised models and capped at max_model_slots. Each preload is the full ensureModelLoaded (weights + legacy scheduler + v2 bridge/warmup when flagged). Readiness vs availability: registration is deferred until warm, at most startup_preload_timeout_secs (default 120s). On timeout the provider registers anyway — a lone provider for a model must still serve it cold, and the lazy-load path is unchanged as the fallback — while the remaining loads continue in the background. The heartbeat warm_models field (which the coordinator's warm-model bonus scores) stays truthful throughout: it is derived from live slots only. - Startup self-test (startup_selftest, default on): after each preload, a 1-token greedy decode through the REAL serving path (v2 bridge when flagged, else legacy scheduler) forces end-to-end warmth (Metal JIT, compiled buckets, chat-template render). Failure emits WARN engine_health telemetry and is fail-open by default; startup_selftest_fail_closed retires the model from this run's advertised set (registration filters through advertisedModels). Tests (live-isolated, scripted loaders — no weights, no network): persistence round-trip incl. unload removal + the shutdown-preserve guard; preloader ordering/admission-skip/failure-continuation/ fail-open/fail-closed/cancellation; plan order + unknown-id filter + dedup + slot cap; gate defers-until-warm, proceeds-at-timeout with background continuation, and fail-closed retirement. * feat(provider): rollover jitter before background auto-update install Behavior change at defaults (update_jitter_seconds = 300) — justified as the fleet-rollover fix: providers discover a release on aligned 30-minute update ticks (ticks correlate after any previous synchronized restart) and drained + restarted in near-unison, putting the entire fleet cold at the same moment (first_chunk_timeout storm). A uniform random 0-300s delay between staging the verified bundle and beginning the drain decorrelates the restarts; combined with the startup preload gate, warm boxes absorb traffic while each cold box re-warms. Placement: the delay sits strictly AFTER download + signature verification (no security-critical check is deferred; the bundle is already verified and staged, and the provider keeps serving while it waits) and BEFORE beginDraining. Forced/manual updates — the darkbloom-update command and the startup update check — do not go through AutoUpdateController and stay immediate. A 1-hour sanity cap bounds a misconfigured value. Tests: delay bounds with a seeded RNG, zero-max and forced bypasses, the sanity cap, and controller sequencing (jitter strictly between stage and drain; never invoked on up-to-date / check-failed / stage-failed paths). * fix(provider): review fixes — cancel-safe jitter, no-evict preload loads, retirement + persistence hygiene Review round (Codex + Claude) on the startup-preload/jitter feature found four real gaps; all fixed with regression tests: 1. Cancel-safe rollover jitter. The jitter closure slept with `try? await Task.sleep`, so a shutdown cancelling autoUpdateTask inside the (up to 300s) jitter window returned early and the controller proceeded to drain + commit + RESTART a shutting-down provider. AutoUpdateController now checks Task.isCancelled after waitBeforeInstall and aborts with a new .cancelled outcome (resumeServing discards the staged bundle; the next tick retries). 2. No-evict preload loads, enforced in the real load path. The preloader's freeMemoryGb pre-check was advisory only — the actual ensureModelLoaded could still slot-cap-evict or memory-evict an EARLIER preloaded (idle) model to fit a later candidate, and a local-endpoint load interleaving with the driver could make the pre-check stale. ensureModelLoaded/evictUntilAvailable now take allowEviction (default true = live-traffic behavior unchanged); the startup preload passes false, and the check runs inside the serialized isLoadingAny critical section so it cannot go stale. Tested against the production load path (fake HF snapshot + real admission gates): no-evict refuses at the slot cap and the memory gate while the default policy still evicts. 3. Post-registration fail-closed retirement now reaches the coordinator. When the gate times out, registration has already announced the model; a later self-test retirement only edited local state, so the coordinator kept routing it. Retirement now mirrors the hard-swap drop: unadvertise on the client store + forceReconnect (new provider-side CoordinatorClient method; a fresh register is the existing wire mechanism for advertised-set removals — models_update is additive, and NO protocol changes are introduced). End-to-end test: real CoordinatorClient against MockCoordinator observes the second register without the retired model. 4. Loaded-models persistence is inert until run() arms it. The persist-on-load/unload write points fired from ANY ProviderLoop — including unit tests that install/unload slots — clobbering the operator's real ~/.darkbloom/loaded-models.json (the next boot's preload plan). Persistence is now gated on loadedModelsPersistenceEnabled, set only by run() (and explicitly by the test seam with a temp path). * fix(engine-v2): honor memory_reserve_gb in v2 KV sizing; clamp heartbeat budget max to live fleet residency (PR#499 r3 P2) Two coupled capacity-truth fixes on the v2 path: * memory_reserve_gb in v2 ceilings: UnifiedMemoryCap.kvBudgetBytes gains configReserveBytes (effective cap = min(hardCap, physical - reserve), the same max(configReserve, capImplied) hold-back the load gate and the shared KV gate apply); EngineV2KVSizing.engineKVBytesCapacity threads it and the slot factory passes the operator reserve. On 16/32 GiB boxes (default 4 GiB reserve > cap-implied reserve) the bridge no longer advertises/privately admits capacity the shared gate rejects post-acceptance. * Stale activeTokenBudgetMax after later loads: v2 admission ceilings are construction-fixed, so a model loaded later (especially a legacy/non-allowlisted slot) left existing bridges advertising a stale budget the coordinator kept routing against. The heartbeat now CLAMPS the reported max to the sizing function's current answer: ProviderLoop.updateAggregateCapacity snapshots live fleet residency (all slots' weights + operator reserve) into EngineV2Runtime.capacitySummary(fleetKV:), which recomputes each bridge's budget via EngineV2KVSizing.liveEngineKVBytesBudget (min(construction grant, current fleet answer)) and passes it to backendSlotCapacity(kvBytesBudgetClamp:). The engine's private ledger keeps its grant; only the coordinator-visible figure shrinks. Heartbeat cadence only - never the submit path. Tests: kvBudgetBytes reserve regimes (binding / no-op / degenerate), pure sizing with reserve both regimes, live-clamp pure math (later legacy load shrinks by exactly its weights; co-resident v2 grants subtract; never inflates past the grant; clamps to 0), and an end-to-end wiring test: load a v2 slot, register a second slot's weights, heartbeat max shrinks by exactly weights/rate while the engine keeps its construction grant. * fix(engine-v2): preserve queue-full classification on v2 admission rejections (PR#499 r3 P2) EngineV2.submit surfaces BOTH byte-ledger capacity exhaustion and its waiting-queue-full / draining-for-shutdown guards as CBv2KVError.capacityExhausted; the bridge mapped every shape to the generic token_budget_exhausted string, so v2 queue saturation surfaced as a 503 token-budget failure instead of the legacy 429 + Retry-After queue-full path. The two families are distinguishable by shape: the slot-rejection guards throw the sentinel (needed: 1, available: 0) (no byte estimate exists), while byte-ledger rejections carry needed = a multiple of the per-token KV cost (never 1) and available = admissibleBytesCapacity (> 0 for any constructible engine). EngineV2Translation .admissionErrorMessage now maps the sentinel to the exact legacy queue-full string ('token_budget_exhausted: request queue full', BatchSchedulerTypes.RejectionReason.queueFull), which fromSchedulerMessage classifies as .queueFull -> 429. Shutdown-drain riding the same sentinel lands on queue-full too, matching the legacy handler's .queueFull("provider shutting down"). No new classification strings, no wire changes. Tests: sentinel -> queue-full string -> .queueFull -> 429; byte-figure rejection stays .tokenBudgetExhausted -> 503; near-sentinel byte figures (needed 2/available 0, needed 1/available 512) never classify as queue-full. Also carries the bridge/runtime heartbeat-clamp tests for the previous commit (same test file). * fix(engine-v2): bound pending logprobs before the first content frame (PR#499 r3 P2) The inference handler drains the capped EngineV2LogprobsChannel into a local pendingLogprobs array every frame but only clears it when injectLogprobs finds a content-bearing frame - so a stream with a long reasoning-only/tool-only prefix (GPT-OSS reasoning) re-accumulated everything the channel cap had just bounded, growing without limit and eventually serializing thousands of entries into one huge first content chunk. EngineV2LogprobsChannel.capPending mirrors the channel's own policy on the caller-held buffer: cap at maxEntries (4096), drop-OLDEST so the freshest window (the entries nearest the content that eventually renders) is kept, return the evicted count. The handler re-caps after every drain, WARNs once on first drop and logs the request's total at stream end (counts only - never token text; entries are response content and stay inside the E2E stream). Tests: drop-oldest at the real 4096 cap with count reporting and no-op below the cap; frames-loop simulation where a capped window survives reasoning-only frames and the first content chunk carries exactly the retained entries. * chore(submodule): bump mlx-swift-lm to engine-v2 tip 0c5d3a9 (round-3 review fixes) * chore(submodule): pin mlx-swift-lm to main d63b1c8 (squash-merge of engine-v2 PR #62; content-identical to 0c5d3a9) * fix(provider): startup self-test no longer poisons Qwen3.5 slots — stale VLM mrope position ids + e2e state isolation Root cause of the CI e2e regression (test 1 passes, every later test's provider dies on its first inference with '[broadcast_shapes] Shapes (1,8,L,64) and (1,1,12,64) cannot be broadcast'): 1. Qwen3.5-0.8B declares a vision_config, so the provider loads it via VLMModelFactory. The MLXVLM Qwen35 LanguageModel cached the full-prompt mrope position ids (precomputedPositionIds) on the long-lived model module and reused them for ANY later forward at cache offset 0. The startup self-test decode ('Hi', 12 prompt tokens, maxTokens=1) was the first request in every preloading provider, so the first routed request with a longer prompt sliced the stale 12-position ids (MLX clamps the out-of-range slice), producing cos/sin frozen at (1,1,12,64) against the new request's (1,8,L,64) rotary block — process-fatal in the first full-attention layer. Fixed in the mlx-swift-lm submodule (d3b1ae9): always recompute position ids at sequence start, exactly like Qwen3VL and HF transformers; precomputedPositionIds removed. Also covers Qwen35MoE (subclass). NOT self-test-specific: ANY second, longer text-only request on one scheduler crashed the same way. 2. The e2e testbed launched providers with the real user HOME, so ~/.darkbloom/loaded-models.json leaked across testbed instances: test 1's provider persisted its serving set, and test 2's freshly-booted provider startup-preloaded + self-tested it — the trigger that made every post-first test hit (1). Providers now get a per-instance temp state dir (DARKBLOOM_STATE_FILE + DARKBLOOM_LOADED_MODELS_FILE), removed on Stop. Regression test: StartupSelfTestDecodeLiveTests (weight-gated, DARKBLOOM_LIVE_MLX_TESTS) loads Qwen3.5-0.8B through the SAME VLMModelFactory selection production uses and drives the self-test-shaped request then a longer request through MultiModelBatchSchedulerEngine + MLXOpenAIService — crashes with the exact production signature without the submodule fix, passes with it. Validated end-to-end: 4 sequential e2e integration tests green locally (previously test 2+ crashed 502). * feat(provider): v0.7.0 engine posture — v2 default-on for exact prod checkpoints, legacy compiled decode opt-in, finish_reason 'length' threaded end-to-end Three coordinated changes for the v0.7.0 release posture: 1. engine_v2 default TRUE, allowlist narrowed to the EXACT production checkpoint ids (mlx-community/gpt-oss-20b-MXFP4-Q8, mlx-community/gemma-4-26b-a4b-it-8bit) — default-on scope equals the real-model-validated scope; other quantizations/families keep legacy until widened via DARKBLOOM_ENGINE_V2_MODELS. DARKBLOOM_ENGINE_V2=0 stays the absolute per-box kill switch (beats config, no release). The v2-init-failure fallback telemetry (engine_v2_fallback, WARN engine_health) now also carries the human-readable 'error' reason next to error_class + model — it is load-bearing now that v2 is the default. Exact-pattern allowlist matching compares last path components on both sides so bare ids match org-qualified patterns. Retirement (failed self-test) and hard-swap/idle/eviction paths were audited for default-on v2: every unload funnels through ProviderLoop.unloadModel, which unregisters the bridge from the runtime and drains it gracefully before scheduler teardown; the hard-swap dropAdvertisedBuild is a lazy advertised-set drop that leaves in-flight requests on the resident slot untouched. 2. Legacy compiled decode is now OPT-IN (new [backend] key legacy_compiled_decode, default false). The bumped mlx-swift-lm pin ships CompiledDecode default-ON; LegacyCompiledDecodeGate forces DARKBLOOM_COMPILED_DECODE=0 into the process env at both serve entry points (ProviderLoop.run, local standalone) BEFORE any engine/model code latches the library's static — unless the operator opted in via config or set the env var explicitly (explicit env always wins, both directions). Keeps release behavior identical to prod v0.6.30 (the compiled-decode rollback); single-stream speed ships via the v2 canary instead. 3. finish_reason 'length' is threaded end-to-end instead of being flattened to 'stop'. GenerationEvent.info gains finishReason; the legacy engine bridge passes RequestOutput.finishReason through, the v2 bridge maps CBv2FinishReason.length/.stop, the B=1 greedy fast path infers length from a full-budget decode, and MultiModelBatchSchedulerEngine forwards it as ServerGenerationInfo.stopReason (tool_calls still wins). Clients now see finish_reason 'length' on max_tokens truncation from BOTH engines. Tests: LegacyCompiledDecodeGateTests (gate matrix + real-env apply, snapshot/restore), EngineV2 gating suite updated for the exact-id allowlist + default-on selection matrix (kill switch beats default-on), fallback telemetry asserts the new 'error' field, v2 bridge test for .length/.stop/cancelled reason threading. Full provider suite green (1327 tests); live engine-path tests green (weight-gated). * docs(provider): review-nit comment fixes — GenerationEvent.info arity in docs, gate empty-env rationale
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
makeBuffer(bytesNoCopy:), achieving 100% memory coverage for all model formatshypervisor_activefield to attestation challenge-response on both provider (Rust) and coordinator (Go) sidesHow it works
macOS 26 Hypervisor.framework requires
hv_vm_mapaddresses and sizes to be 16 MB-aligned. We discovered this constraint experimentally — smaller mappings returnHV_BAD_ARGUMENT. The solution:mmapa pool sized to 90% of physical RAMhv_vm_mapin 16 MB chunks (60 GB mapped in 2.5 ms)alloc_buffer(size)returns page-aligned pointers within the poolmakeBuffer(bytesNoCopy:)on pool memoryThis gives 100% coverage for ALL model formats — BF16, FP16, FP8, 4-bit, 8-bit, MoE.
Changes
Provider (Rust):
hypervisor.rsmodule: VM lifecycle, 16 MB-aligned pool allocation,alloc_buffer()API for Metal buffer creationsecurity.rs: RDMA policy updated — RDMA enabled + hypervisor active = safemain.rs: VM + pool created at startup before security posture checkhardware.rs: Addedtotal_memory_gb()for pool sizingcoordinator.rs: Freshhypervisor_activestatus in challenge-responseprotocol.rs: Newhypervisor_active: Option<bool>field inAttestationResponseCoordinator (Go):
provider.go: Challenge verification accepts RDMA-enabled providers if hypervisor is activeattestation.go:HypervisorActivefield in attestation blob (alphabetical ordering preserved for SE signature compatibility)messages.go:HypervisorActiveinAttestationResponseMessageValidated experimentally
Test plan