rebase + refactor: graceful drain re-ported onto master's modular split (NWConnection transport) with review fixes and tests#517
Open
github-actions[bot] wants to merge 77 commits into
Conversation
…R-333/DAR-334) (#410) * feat(coordinator): smart early-429 admission + prefill SSE keepalives (DAR-333/DAR-334) DAR-333 — replace admit→5xx for unservable gpt-oss long prompts with an uptime-neutral early 429: - registry.PredictServable: context-window + token-budget structural servability check (fail-open), mirroring the provider load/KV gate; plumb per-model KVBytesPerToken into the routing snapshot. - Proactive preflight gate (EIGENINFERENCE_SERVABILITY_GATE, default off) on the chat + generic inference paths -> 429 + Retry-After. - Always-on dispatch-exhausted backstop: reclassify a provider token-budget/KV/context 5xx to an uptime-neutral 429 (isUnservableProviderError). - OR-formula uptime: inference.request_outcome{model,class} counter (exactly once per request), GET /v1/admin/uptime aggregation, Datadog widgets, docs. DAR-334 — SSE keepalives during long prefill (EIGENINFERENCE_PREFILL_KEEPALIVE_INTERVAL, default off): a dispatched streaming request commits HTTP 200 and emits ": keepalive" comments until the first chunk so OpenRouter's fetch timeout does not fire. The first keepalive only fires after one interval, preserving deferred-commit / invisible-failover for fast requests; post-commit failures surface in-band as SSE error events. Tests: predictor, failure classifier, OR-uptime mapping + endpoint, keepaliver, and end-to-end gate wiring. gofmt/vet and full coordinator go test are green. * refactor(coordinator): drop /v1/admin/uptime endpoint; default prefill keepalives on Codex review found two correctness bugs in the /v1/admin/uptime aggregation: - a token-budget 5xx reclassified to a client-visible 429 still showed as provider_5xx (the persisted per-attempt route outcome keeps the original 5xx, and the endpoint skipped the dispatch-stage rejection row); - it deduped on InferenceRouteRecord.RequestID, which is the per-attempt provider job UUID, so an invisible failover (failed attempt + later success) was counted as both a failure and a success. The live inference.request_outcome counter is emitted exactly once per client request at the run() tail (after all attempts) and is unaffected by both bugs, so it remains the source for the OpenRouter-formula uptime panel; the exact post-commit breakdown stays available via /v1/admin/routes + /v1/admin/rejections. Remove the endpoint and its now-dead helpers (orUptimeClass, dedupeRouteOutcomes, uptimeOutcomeRank, the cancelled class) and their tests. Also enable prefill SSE keepalives by default (defaultPrefillKeepaliveInterval = 10s) so long prefills don't hit OpenRouter's fetch timeout out of the box. The first keepalive still fires one interval in, so fast requests keep clean deferred-commit / invisible-failover; EIGENINFERENCE_PREFILL_KEEPALIVE_INTERVAL overrides the cadence and 0 disables. * fix(coordinator): address Codex review on the gpt-oss reliability follow-up Four findings from the Codex PR review: - servability fail-open hole: a cold provider with valid memory/size data but no post-load KV headroom reports a KNOWN budget of 0; the predictor's `fleetMax > 0` guard then skipped prompt_too_long, so an all-zero-budget fleet failed open and dispatched into a guaranteed provider-side token/KV rejection. Gate on !sawUnknown instead, so a known-zero ceiling rejects. - Responses-API error shape after a keepalive commit: once a prefill keepalive has sent HTTP 200 on a streaming /v1/responses request, a terminal failure emitted chat-completions `data:{...}` + [DONE], which strict Responses clients can't parse. Emit a Responses-shaped `event: error` (no [DONE]) for isResponsesAPI. - non-streaming OR-uptime over-report: the run() tail emitted success for any committed request, but for stream=false `committed` only means a provider chunk arrived and the non-streaming writer can still return a 5xx/504. Emit success at the tail only for streaming; for non-streaming, record the outcome from the status the writer actually emits (via the existing statusWriter wrapper). - servability gate ordering: the gate ran before the alias-capacity fallback, so a request whose desired build was over budget but whose Previous alias build had capacity got 429'd instead of failing over. Run the gate after the fallback in both the chat and generic preflights. Tests: known-zero cold budget rejects; Responses keepalive error shape; existing suites green (gofmt/vet/go test ./...).
… providers (#412) FleetCapacitySnapshot computed network-wide token-budget utilization with a mismatched numerator/denominator: it summed used+queued across all of a provider's per-model slots but divided by only the largest single slot's ActiveTokenBudgetMax, then clamped. Each slot's max already equals that slot's committed tokens plus the provider's SHARED KV headroom (one unified-memory pool, reported once per resident model), so the per-slot maxes are not additive. The old math systematically over-reported multi-model providers (pegging them near 100%) and disagreed with both the per-model ModelCapacity snapshot and the freeMemoryAdmits admission gate, which treat each slot independently. Reconstruct the true pooled budget: total = Sum(committed) + shared free headroom counted once (the largest per-slot remaining). Yields used <= total by construction (drops the clamp), reduces exactly to the old value for single-slot providers, and matches the admission/per-model views. Extracted as a pure providerTokenBudget helper with table tests. Warm/Little's-Law axis unchanged. Bug introduced in #407.
…tor) + v0.6.16 #408 wedge & KV-pin recovery (provider) — DAR-335/336/337/338 (#413) * fix(coordinator): split provider-5xx reclassification — capacity-class->429 vs fault->5xx (DAR-336) * feat(coordinator): per-provider node-health circuit breaker — auto-deroute fault-503 nodes, fail-open (DAR-335) * fix(provider): move KV self-heal off the budget actor + backend liveness watchdog & KV-pin recovery (DAR-338, DAR-337) * fix(coordinator): preflight fails open on node-health breaker (DAR-335) QuickCapacityCheck excluded breaker-open providers, so an all-breaker-open fleet reported 0 candidates / 0 capacity-rejections and the consumer hard-503'd "no_provider" BEFORE dispatch's fail-open valve could serve a probe — the exact model-wide outage the valve exists to prevent during a bad fleet-wide rollout. The preflight now ignores the provider breaker (every other gate, incl. the shape-keyed inference-error cooldown, still honored). Adds a regression test. * docs(coordinator): clean up comments — drop ticket refs, improve readability * docs(provider): clean up comments — drop ticket refs, improve readability * fix(provider): count admission rejections as liveness demand so a pinned KV pool self-restarts * fix(coordinator): address PR review findings (breaker/reclassifier) - Fail-open valve now triggers ONLY when the node-health breaker is the sole reason a request has no route; if a healthy provider is merely busy (capacity) or too slow (TTFT), surface that signal so the request queues/429s instead of routing to a known-bad, breaker-open node. - Breaker treats a 503 'request rejected' as a capacity/backpressure shed (consistent with the api reclassifier and the inference-error breaker), so a healthy-but-busy node is no longer quarantined. - Drop 'service temporarily unavailable' from the api capacity table: the coordinator emits it on its own store/DB errors, which must stay 5xx rather than be hidden as an uptime-neutral 429. * feat(coordinator): deadline-bounded provider failover (replace fixed 3-cap) A request now keeps failing over to fresh healthy providers until one succeeds, candidates are exhausted (each failed provider is excluded, so dispatchPrimary returns outcomeFailFast once none remain), or the request's deadline/context fires (run() checks r.Context() each attempt). maxDispatchAttempts becomes a high safety ceiling (64) against a pathological hot loop rather than the normal stopping point; retries never re-queue, so failover stays fast. Exhaustion messages now report the actual attempt count. Adds a regression test proving failover reaches a healthy provider sitting behind six failing ones (the old 3-attempt cap gave up before reaching it). * fix(coordinator): capacity-class precedence + capacity-shaped 5xx in breaker - inference_failure_class: a provider 'model load failed: <capacity reason>' (insufficient memory/KV, slot-cap) now reclassifies to an uptime-neutral 429 instead of staying 5xx — capacity markers are checked before the model-load-failed wrapper; bad-weights/opaque load failures still stay 5xx. - provider_breaker: ignore a capacity-shaped 5xx regardless of status code, so a 500/502/504 carrying a capacity/backpressure message no longer trips the node-health breaker (some/older provider paths surface capacity as a non-503 5xx; the dispatch reclassifier already 429s them). Renames the breaker's capacity helper/markers to drop the 503-specific name. * fix(provider): keep reporting the real model id (reloading) during liveness self-restart — no nil-engine 500s
…ve base reward (#414) * feat(earn): realistic earnings calculator — 100% utilization + additive base reward Replace the inflated full-saturation projection with a realistic model: total = usage + floor − electricity • Usage — single-stream decode throughput at 100% utilization (the machine serves a request every second it's online). The old calculator multiplied by a 16× batch factor AND assumed 100% paid duty cycle, overstating earnings by ~10–20× vs the live network (which runs at ~4% utilization, ~one concurrent request per machine). Utilization and hours are now FIXED at 100% / always-on — no user-facing knobs — per product direction. • Floor — the provider base-reward floor (PR #282), set by verified-memory tier, ramped by uptime, added ON TOP of usage (additive), not max(). • Electricity — marginal inference watts over idle. Revenue now also counts input tokens (at the network-observed 3.5:1 prompt:completion ratio) in addition to output, so the usage figure matches what providers actually earn per request. Mirror across console-ui /earn and the darkbloom.dev landing calculator, and extract the pure math into a shared module (console-ui/src/app/earn/calc.ts) to keep them in sync. Headline for an M5 Max 128GB on gpt-oss-20b: ~$51/mo (≈$25 usage + $26 base reward − $3 electricity). Note: PR #282's engine currently pays max(usage, floor); this calculator assumes the additive model the program will ship. Numbers will match once #282 switches Draw from max() to additive. Verified: console-ui build green, eslint clean (0 errors), 306/306 vitests pass. * fix(earn): model continuous batching (4x) at 80% utilization Single-stream throughput was wrong for the "busy machine" case: a saturated provider runs continuous batching (the MLX engine batches concurrent requests), so usage should credit that. Switch the usage model to: decode = single_stream × 4 (continuous batch, quality-preserving) × 0.80 (utilization) 4× (not the old 16× ceiling) reflects the concurrency the engine sustains while keeping per-user decode latency acceptable; 80% utilization leaves realistic idle gaps. Marginal electricity now scales with utilization too. M5 Max 128GB / gpt-oss-20b: usage ~$88/mo + $26 base reward = ~$114/mo (was ~$51/mo under single-stream). Mirrored across console /earn and landing. * fix(earn): use 2x continuous batching (was 4x) Lower the continuous-batching factor from 4x to 2x for a more conservative quality-preserving concurrency. M5 Max 128GB / gpt-oss-20b: usage ~$42/mo + $26 base reward = ~$68/mo (decode ~144 tok/s). Mirrored across console /earn and landing. * fix(earn): use 4x continuous batching Set the continuous-batching factor to 4x. M5 Max 128GB / gpt-oss-20b: usage ~$88/mo + $26 base reward = ~$114/mo (decode ~288 tok/s). Mirrored across console /earn and landing.
…-313) (#377) * feat(provider): KV quant benchmark gate with V-only affine4 pass - Add kv-quant-gate CLI and supporting Benchmark/KVQuant modules for PPL, logits, output, NIAH, performance, and memory suites. - Implement custom KVQuantizedCache and VOnlyQuantizedKVCache wrappers with start-token delay and sliding-window preservation. - Wire candidate cache injection into the performance runner so reference and candidate metrics are collected end-to-end. - Resolve HF-cache symlinks before file-extension checks in LocalMLXModelReadiness so downloaded snapshots load correctly. - Shift release thresholds to agreement metrics where the fp16 baseline itself can fail (output/NIAH), and add json_valid/required_substrings agreement rates. - Default candidate is full-v-affine4:g64:start1024; affine4/8 full-KV and turbo modes are currently rejected as unsupported by the gate. V-only affine4 passes quality gates for both Gemma 4 26B and GPT-OSS-20B. * feat(provider): bfloat16 KV cache mode and benchmark-gated quality - Add `bf16-kv:start1024` candidate mode with a custom BFloat16KVCache wrapper. This halves KV-cache precision with no model-attention changes and passes PPL/logits/output/NIAH gates for both Gemma 4 26B and GPT-OSS-20B. - Switch `affine4/8` full-KV modes to mlx-swift-lm upstream dynamic quantization (GenerateParameters.kvBits) and add an incremental teacher-forced scorer so the PPL/logits suites can exercise the same quantized generation path. (Upstream QuantizedKVCache fatal-errors on the single-forward update path, so the old scorer could not be used.) - Reset MLX peak memory between reference and candidate baseline runs so peak-memory ratios are per-run rather than cumulative. - Use peak-memory max (instead of mean) for threshold memory ratios. - Update KVQuantExecutionTests for the new affine4/8/bf16 behavior. * feat(provider): add full-v-bf16 KV cache candidate - Add `full-v-bf16:start1024` candidate mode and VOnlyBFloat16KVCache wrapper: keys stay fp16, values are stored as bfloat16 after the start token. This is intended as a middle-ground compression option that works with model attentions using the plain update(keys:values:) path. - Quality gate passes for Gemma 4 26B; memory/perf impact is still being measured against the current total-GPU-peak metric. - Update KVQuantExecutionTests for the new mode. * feat(provider): protocol-safe quantized cache wrapper and policy alignment - Add ProtocolSafeQuantizedKVCache: wraps upstream QuantizedKVCache and implements update(keys:values:) with a dequantizing fallback. This lets full-KV affine modes run through both the native quantized attention path and the plain update path used by single-forward scoring. - Switch affine4/8 execution back to a custom cache factory (the protocol-safe wrapper) so the gate can evaluate full-KV quantization end-to-end. - Add a minimum PPL ratio threshold (0.99) so large candidate-perplexity decreases are also flagged as failures. - Update KVQuantPolicy to recommend the validated `full-v-affine4:g64:start1024` candidate for Gemma 4 and GPT-OSS, and adjust KVQuantPolicyTests. * feat(kvquant): prove kernel correctness, fix scorer measurement bug, add capacity metric First-principles instrumentation for the KV-cache capacity objective: - kv-attn-selftest executable: deterministic numerical probes proving quantizedScaledDotProductAttention == dequantize+SDPA exactly (MHA/GQA/ causal/decode, 8/4-bit), so the kernel is correct after the mask-fill fix. Outlier + multi-step + per-channel isolation cases included. - Fix mask-fill bug in quantizedScaledDotProductAttention (vendored): masked positions used +leastNormalMagnitude (~0) instead of -inf. - Fix a measurement bug: the single-forward PPL/logits scorer let start-delayed caches silently return fp16 (never engaging quantization). Candidate scoring now runs true incremental token-by-token through the candidate's own cache. KVQuantCacheProbe deterministically validates single-forward vs incremental divergence (one-shot diff 0.0 vs incremental 0.31) — no 26B run needed. - Add a model-free capacity suite + per-mode stored K/V bits and capacityRatioVsFP16 (incl. affine scale/bias overhead). Exposes that bf16 modes are 1.0x (no capacity gain) and affine8 is ~1.78x (overhead), making KV-bytes-per-token the headline metric. Finding: Gemma 4 K8V8 collapse (top1 0.56) is genuine compounded quantization loss across ~48 layers, NOT a kernel bug (dequant path == kernel path == 0.5595). Per-layer cos 0.999 still compounds to large end-to-end drift. * feat(kvquant): trustworthy generation-fidelity gate; honest quality/capacity Pareto Raw-text teacher-forced PPL/logits was an untrustworthy gauge (reference NLL ~5.3 nats / PPL ~200 on trivial text; ppl.ratio<1). Replace the logits suite with generation-fidelity: the reference greedily continues a coherent prompt, the candidate is teacher-forced over that continuation, and we measure top-1 / top-5 agreement on confident tokens through the correct generation forward (quantization engaged incrementally). reference-self match = 1.000 validates it. Honest Gemma 4 26B Pareto (capacity includes affine scale/bias overhead): V-only 4-bit start1024 : 1.56x, top1 1.000 K8V8 start0 : 1.78x, top1 0.931, top5 0.993 K4V4 start0 : 3.56x, top1 0.892, top5 0.941 Conclusion: earlier 'catastrophic' 0.56 numbers were gauge artifacts, not real quantization loss. V4 is effectively lossless; K8V8 is already 0.93 with no start delay. Next: long-context stress + start-delay/residual window + asymmetric K8/V4 to reach quality-preserving 2x. * feat(kvquant): establish quality-preserving ~2x via long-context gen-fidelity sweep Long-context generation-fidelity (2048-3072 tok prompts, ~768 samples) gives a trustworthy quality x capacity Pareto on BOTH target models: K8V8 g128 -> 1.94x : Gemma 0.973 top1, GPT-OSS 0.982 top1 (robust ~2x) K6V6 g64 -> 2.46x : Gemma 0.977 top1, GPT-OSS 0.932 (min 0.78, riskier) V4 -> 1.56x : ~0.995 (near-lossless anchor) Path selection (both store quantized = capacity gain): - Gemma 4: kernel path (ProtocolSafeQuantizedKVCache, quantized attention). - GPT-OSS: kernel path FATALS ('Quantized attention does not support non-zero sinks'); use dequant path (KVQuantizedCache) which is sink-safe and still stores quantized. Gate plumbing: long coherent prompts truncated to target context; prompt primed in one batched forward then decoded token-by-token; KVQ_BITS/KVQ_GROUP/KVQ_DEQUANT env knobs for Pareto sweeps. Conclusion: ~2x capacity at high fidelity is real; K8V8 g128 is the conservative cross-model pick. * DAR-313: make validated KV-quant configs first-class modes, remove env-var sweep hacks - Remove KVQ_BITS/KVQ_GROUP/KVQ_DEQUANT env reads from affine4/affine8 execution. - Add first-class modes: k8v8:g128, k8v8:g64:dequant, k6v6:g64, k6v6:g64:dequant. - Wire all KVQuantCandidateMode switches (types, cacheSpec, execution config). - Update effectiveBits to use the mode's actual group size for accurate capacity ratio. - Add parse + factory-non-nil tests for the four new modes. * chore(submodule): bump mlx-swift-lm to darkbloom/kvquant (KV-quant mask fix) [DAR-313] * chore(kvquant): point engine submodule at merged main (#43 -> a9e0ee1) mlx-swift-lm #43 squash-merged to main (a9e0ee1) and its branch was deleted, orphaning the prior pointer. Re-point to the merged main commit so the submodule is fetchable from a clean checkout/CI.
) * feat(provider): KV quant benchmark gate with V-only affine4 pass - Add kv-quant-gate CLI and supporting Benchmark/KVQuant modules for PPL, logits, output, NIAH, performance, and memory suites. - Implement custom KVQuantizedCache and VOnlyQuantizedKVCache wrappers with start-token delay and sliding-window preservation. - Wire candidate cache injection into the performance runner so reference and candidate metrics are collected end-to-end. - Resolve HF-cache symlinks before file-extension checks in LocalMLXModelReadiness so downloaded snapshots load correctly. - Shift release thresholds to agreement metrics where the fp16 baseline itself can fail (output/NIAH), and add json_valid/required_substrings agreement rates. - Default candidate is full-v-affine4:g64:start1024; affine4/8 full-KV and turbo modes are currently rejected as unsupported by the gate. V-only affine4 passes quality gates for both Gemma 4 26B and GPT-OSS-20B. * feat(provider): bfloat16 KV cache mode and benchmark-gated quality - Add `bf16-kv:start1024` candidate mode with a custom BFloat16KVCache wrapper. This halves KV-cache precision with no model-attention changes and passes PPL/logits/output/NIAH gates for both Gemma 4 26B and GPT-OSS-20B. - Switch `affine4/8` full-KV modes to mlx-swift-lm upstream dynamic quantization (GenerateParameters.kvBits) and add an incremental teacher-forced scorer so the PPL/logits suites can exercise the same quantized generation path. (Upstream QuantizedKVCache fatal-errors on the single-forward update path, so the old scorer could not be used.) - Reset MLX peak memory between reference and candidate baseline runs so peak-memory ratios are per-run rather than cumulative. - Use peak-memory max (instead of mean) for threshold memory ratios. - Update KVQuantExecutionTests for the new affine4/8/bf16 behavior. * feat(provider): add full-v-bf16 KV cache candidate - Add `full-v-bf16:start1024` candidate mode and VOnlyBFloat16KVCache wrapper: keys stay fp16, values are stored as bfloat16 after the start token. This is intended as a middle-ground compression option that works with model attentions using the plain update(keys:values:) path. - Quality gate passes for Gemma 4 26B; memory/perf impact is still being measured against the current total-GPU-peak metric. - Update KVQuantExecutionTests for the new mode. * feat(provider): protocol-safe quantized cache wrapper and policy alignment - Add ProtocolSafeQuantizedKVCache: wraps upstream QuantizedKVCache and implements update(keys:values:) with a dequantizing fallback. This lets full-KV affine modes run through both the native quantized attention path and the plain update path used by single-forward scoring. - Switch affine4/8 execution back to a custom cache factory (the protocol-safe wrapper) so the gate can evaluate full-KV quantization end-to-end. - Add a minimum PPL ratio threshold (0.99) so large candidate-perplexity decreases are also flagged as failures. - Update KVQuantPolicy to recommend the validated `full-v-affine4:g64:start1024` candidate for Gemma 4 and GPT-OSS, and adjust KVQuantPolicyTests. * feat(kvquant): prove kernel correctness, fix scorer measurement bug, add capacity metric First-principles instrumentation for the KV-cache capacity objective: - kv-attn-selftest executable: deterministic numerical probes proving quantizedScaledDotProductAttention == dequantize+SDPA exactly (MHA/GQA/ causal/decode, 8/4-bit), so the kernel is correct after the mask-fill fix. Outlier + multi-step + per-channel isolation cases included. - Fix mask-fill bug in quantizedScaledDotProductAttention (vendored): masked positions used +leastNormalMagnitude (~0) instead of -inf. - Fix a measurement bug: the single-forward PPL/logits scorer let start-delayed caches silently return fp16 (never engaging quantization). Candidate scoring now runs true incremental token-by-token through the candidate's own cache. KVQuantCacheProbe deterministically validates single-forward vs incremental divergence (one-shot diff 0.0 vs incremental 0.31) — no 26B run needed. - Add a model-free capacity suite + per-mode stored K/V bits and capacityRatioVsFP16 (incl. affine scale/bias overhead). Exposes that bf16 modes are 1.0x (no capacity gain) and affine8 is ~1.78x (overhead), making KV-bytes-per-token the headline metric. Finding: Gemma 4 K8V8 collapse (top1 0.56) is genuine compounded quantization loss across ~48 layers, NOT a kernel bug (dequant path == kernel path == 0.5595). Per-layer cos 0.999 still compounds to large end-to-end drift. * feat(kvquant): trustworthy generation-fidelity gate; honest quality/capacity Pareto Raw-text teacher-forced PPL/logits was an untrustworthy gauge (reference NLL ~5.3 nats / PPL ~200 on trivial text; ppl.ratio<1). Replace the logits suite with generation-fidelity: the reference greedily continues a coherent prompt, the candidate is teacher-forced over that continuation, and we measure top-1 / top-5 agreement on confident tokens through the correct generation forward (quantization engaged incrementally). reference-self match = 1.000 validates it. Honest Gemma 4 26B Pareto (capacity includes affine scale/bias overhead): V-only 4-bit start1024 : 1.56x, top1 1.000 K8V8 start0 : 1.78x, top1 0.931, top5 0.993 K4V4 start0 : 3.56x, top1 0.892, top5 0.941 Conclusion: earlier 'catastrophic' 0.56 numbers were gauge artifacts, not real quantization loss. V4 is effectively lossless; K8V8 is already 0.93 with no start delay. Next: long-context stress + start-delay/residual window + asymmetric K8/V4 to reach quality-preserving 2x. * feat(kvquant): establish quality-preserving ~2x via long-context gen-fidelity sweep Long-context generation-fidelity (2048-3072 tok prompts, ~768 samples) gives a trustworthy quality x capacity Pareto on BOTH target models: K8V8 g128 -> 1.94x : Gemma 0.973 top1, GPT-OSS 0.982 top1 (robust ~2x) K6V6 g64 -> 2.46x : Gemma 0.977 top1, GPT-OSS 0.932 (min 0.78, riskier) V4 -> 1.56x : ~0.995 (near-lossless anchor) Path selection (both store quantized = capacity gain): - Gemma 4: kernel path (ProtocolSafeQuantizedKVCache, quantized attention). - GPT-OSS: kernel path FATALS ('Quantized attention does not support non-zero sinks'); use dequant path (KVQuantizedCache) which is sink-safe and still stores quantized. Gate plumbing: long coherent prompts truncated to target context; prompt primed in one batched forward then decoded token-by-token; KVQ_BITS/KVQ_GROUP/KVQ_DEQUANT env knobs for Pareto sweeps. Conclusion: ~2x capacity at high fidelity is real; K8V8 g128 is the conservative cross-model pick. * DAR-313: make validated KV-quant configs first-class modes, remove env-var sweep hacks - Remove KVQ_BITS/KVQ_GROUP/KVQ_DEQUANT env reads from affine4/affine8 execution. - Add first-class modes: k8v8:g128, k8v8:g64:dequant, k6v6:g64, k6v6:g64:dequant. - Wire all KVQuantCandidateMode switches (types, cacheSpec, execution config). - Update effectiveBits to use the mode's actual group size for accurate capacity ratio. - Add parse + factory-non-nil tests for the four new modes. * chore(submodule): bump mlx-swift-lm to darkbloom/kvquant (KV-quant mask fix) [DAR-313] * feat(kvquant): batched quantized-cache numerical gate + submodule bump (DAR-314) Adds QuantizedBatchKVCache deterministic correctness gate to kv-attn-selftest (single-row/left-padding/GQA/growth + finalize/extend/filter, dequant-same-data reference) and bumps mlx-swift-lm to the QuantizedBatchKVCache commit. * feat(provider): wire K8V8 KV-quant into batching engine, off by default (DAR-317/318) Opt-in backend.kv_quant flag (default off); Gemma-4-only via KVQuantPolicy. Quantized full-attention BatchKVCache (g128/b8), fp16 sliding; KV byte budget cut ~1.94x so admission grants ~2x tokens; prefix-cache/MTP disabled when on. Flag-off verified byte-for-byte no-regression. Bumps mlx-swift-lm pointer. * feat(provider): kv-engine-demo (e2e capacity proof) + --local flag + policy doc (DAR-318) kv-engine-demo builds the real BatchedEngine fp16 vs K8V8 g128 on Gemma 4: capacity kv_bytes/token 0.516x -> token budget 1.95x; word-level output parity (sanity; rigorous quality = single-seq top1 0.973); decode tok/s ~parity. StartCommand --local now honors backend.kv_quant; KVQuantPolicy doc reflects k8v8:g128. Read-only telemetry accessors for the harness. * feat(provider): GPT-OSS KV-quant on continuous batching via dequant cache (DAR-322) resolveKVQuantScheme: Gemma -> kernel g128; GPT-OSS -> dequant g64 (head_dim guard); flag default off. Byte budget ~1.88x for GPT-OSS. e2e (real BatchedEngine): capacity 1.88x, rigorous top1 0.971, decode ~0.85x (stable to 512 tok). Bumps submodule pointer; demo label derives from resolved scheme. * feat(kv-engine-demo): long-context decode sweep + dequant microbench (DAR-323) Adds --prompt-tokens sweep, --dequant-iters microbench, --capacity-only. Used to show GPT-OSS dequant decode is a FLAT ~10-15% (not O(n^2)): quant/fp16 ratio 0.86-0.92 across 512..16384; engine overlaps ~85-90% of raw dequant. * fix(kvquant): gate live K+V schemes by capacity * fix(kvquant): address Codex review (#378) — diagnostics, tests, gate - kv-engine-demo: label the quantized run by the actual resolved scheme (Gemma g128 / GPT-OSS dequant|kernel / fp16 for unsupported) and reject unsupported model families up front instead of reporting a bogus fp16-vs-fp16 'quantized' comparison. - kv-engine-demo: race stream consumption against --generation-timeout so a stall before/between chunks aborts instead of hanging forever. - KVQuantPerformanceRunner: only apply the decode-TPS regression gate to the performance suite; the memory suite shares the runner and must not fail on throughput. - kv-attn-selftest: fold the live g128 finalize/extend/filter checks into the DAR-314 gate (dar314Ok) instead of a followUpOk that is re-declared later and discarded their failures. - KVQuantEngineTests: mark the suite .serialized; tests mutate the process-wide DARKBLOOM_KV_GPTOSS_KERNEL env and raced under parallel execution. * fix(kvquant): address Codex round-2 review (#378) + submodule leak bump - BatchScheduler: reserve VLM media requests at the fp16 KV rate, not the quantized rate. VLM media streams through container.generate (fp16 KV, not the quantized batched cache), so reserving at the ~0.52x quantized rate under-counts ~2x and risks unified-memory OOM under concurrent media. New fp16KVBytesPerToken; no-op when KV quant is off. - kv-attn-selftest: exit(1) when any DAR gate fails (was always exit 0 -> CI would treat a failing correctness gate as success). - kv-engine-demo: enforce --generation-timeout in the concurrency sweep (per-stream timeout race) so a stalled stream aborts instead of hanging; also dump full fp16-vs-quant generations for eyeball quality diffing. - kv-quant-gate: --candidate auto honors DARKBLOOM_KV_GPTOSS_KERNEL=1 (kernel vs dequant), mirroring the live scheduler. - Bump mlx-swift-lm -> quantized-cache IOGPU leak fix (#43). Note: Codex 'kernel silently dequant for Gemma' (Scheduler.swift:947) verified as a FALSE POSITIVE — the served gemma-4-26b checkpoint has vision_config, so it loads via the VLM Gemma4 class which DOES use the quantized kernel (Gemma4.swift:839/893); the update-only LLM Gemma4Text class is never instantiated for it. * fix(kvquant): gate enforcement + Gemma head_dim guard + selftest exit; bump engine Provider/gate (safe-set review fixes): - kv-quant-gate: exit non-zero when the report is failed; skipped is non-zero unless --allow-missing-data (wires the previously-unused flag). CI can now enforce the gate without parsing JSON. - KVQuantThresholdEvaluator: fail closed on unreadable/malformed threshold JSON, empty rule sets, and zero matched metrics (was silently ignored) + unit tests. - resolveKVQuantScheme: guard Gemma 4 on (global_head_dim ?? head_dim) % 128 == 0 so a malformed/variant config disables KV quant instead of trapping the quantized cache precondition at decode + unit tests. - kv-attn-selftest: fold runCase/runMultiStepCacheTest verdicts into the exit status (early probes could print [FAIL] but still exit 0). - Bump libs/mlx-swift-lm to the engine commit that restricts KV-quant to plain KVCacheSimple and forces the cold path under quantization. Deferred (separate pass): GPT-OSS dequant peak-memory accounting, hybrid sliding-window fp16 reservation, VLM-reservation -> coordinator capacity, and release.json metric emission/pruning. * perf(provider): keep engine concurrency cap in sync with effective cap TASK A — concurrency-cap sync (throughput at concurrency). The continuous-batching engine's per-step concurrency is set by the scheduler via engine.setMaxNumSeqs(...). It was told the cold-start cap exactly ONCE at load and never heard the adaptive ramp: - load pinned dynamicMaxConcurrentRequests to min(4, maxConcurrentRequests) and pushed that single value to the engine; - AdaptiveBatchCapPolicy later raised dynamicMaxConcurrentRequests but only mirrored to the engine when the *adaptive* value changed, and the cold-start burst has no TPS samples yet so the ramp hadn't engaged. Net effect: with maxConcurrentRequests=8 (memory admitting) the engine stayed pinned at 4, so 8 concurrent requests ran as two serialized waves of 4 — roughly halving aggregate throughput at concurrency. Fix: - Add syncEngineConcurrency(): computes the effective cap IDENTICALLY to effectiveMaxConcurrentRequests — max(1, min(maxConcurrentRequests, dynamicMaxConcurrentRequests, memoryBoundMaxConcurrentRequests)) — and pushes it to the engine only when it changed from lastPushedMaxNumSeqs. It is the single setMaxNumSeqs push point. Called at load (replacing the raw setMaxNumSeqs) and on every finish via updateDynamicMaxConcurrentRequests (now re-syncs unconditionally, so a moved memory clamp is honored even when the adaptive cap is unchanged). - Cold-start seed: seed dynamicMaxConcurrentRequests to the configured ceiling max(1, maxConcurrentRequests) at all three sites (init, applyPostLoadBudgets, stopCurrentEngine) instead of min(4, ...). The memory clamp is authoritative in effectiveMaxConcurrentRequests / syncEngineConcurrency, so the engine is NEVER told more than memoryBoundMaxConcurrentRequests (OOM safe) or maxConcurrentRequests. Seeding at the ceiling (rather than min(ceiling, memoryBound)) also lets the effective cap track a RISING memory bound immediately instead of waiting for the slow adaptive ramp. (memoryBoundMaxConcurrentRequests is file-private to the telemetry extension and not referenceable from the seed sites, which is also why the seed defers the clamp to sync.) - Reset lastPushedMaxNumSeqs = -1 in stopCurrentEngine so a freshly-built engine (which starts at its own config.maxNumSeqs default) is always re-pushed on the next load. Adds BatchSchedulerConcurrencyCapTests (non-live, no GPU): with maxConcurrentRequests=8 and memory admitting, the effective cap and the value syncEngineConcurrency pushes are both 8; with bridges that raise the average reservation the memory clamp holds the pushed value at 4 (OOM safety); sync is idempotent; a 1-slot config stays at 1. TASK B — demo throughput metric. kv-engine-demo's concurrency sweep reported aggregate decode tok/s as total decode tokens / the GLOBAL first-token→last-token wall window, which includes later waves' queue-wait/prefill and so under-reports steady-state decode. runConcurrentBatch now returns two clearly-labeled numbers: steadyStateDecodeTps (sum of per-stream decode rates, each over that stream's own decode window) as the capacity headline, and wallClockDecodeTps (the previous total/global-window number) for contrast. Measurement-only; no engine/scheduler behavior change. * perf(provider): bump engine to perf line + TTFT/prefill demo instrumentation Submodule mlx-swift-lm -> 0f46814 (darkbloom/kvquant, #43): on-GPU penalty sampler, KV-quant SDPA mx.compile + n==1 decode-mask skip, per-token decode wins (logSumExp skip, B==1 .item(), resource-debug off-by-default). kv-engine-demo: add submit->first-token TTFT and prefill tok/s measurement (measurement-only). * feat(kvquant): compose KV-quant + prefix cache for the dequant scheme (DAR-319 v2) Relax the mutual-exclusion gate so GPT-OSS (.dequant) gets BOTH ~2x KV capacity AND prefix-cache TTFT. The engine checkpoint-restore now rebuilds QUANTIZED batched caches (mlx-swift-lm 48866bc, restoredFullAttentionCache), so restored rows are concrete-class-compatible with quantized cold rows. The native-kernel scheme (Gemma g128) stays excluded pending its own composition; drafter-MTP unchanged. Submodule mlx-swift-lm -> 48866bc. * chore(kvquant): clean ticket refs from comments + bump engine Remove internal tracker numbers from KV-quant comments and self-test/demo labels for readability (no logic change). Bump mlx-swift-lm to pick up the finish-time prompt-cache extract optimization. * feat(kvquant): document kv_quant opt-in switch + log activation at load Fix the stale BackendSettings.kvQuant doc (KV-quant supports GPT-OSS AND Gemma 4, not Gemma-only) and document the per-provider opt-in. Add a kv-quant logger that, at model load, records whether kv_quant actually engaged for the model (scheme + KV bytes/token + prefix-cache state) or fell back to fp16 — so a beta provider can confirm the switch via 'darkbloom logs'. * chore(kvquant): point engine submodule at merged main (#43 -> a9e0ee1) mlx-swift-lm #43 squash-merged to main as a9e0ee1 and its branch was deleted, orphaning the prior submodule pointer. Re-point to the merged main commit (identical engine tree to the tested head, no behavior change) so the submodule is fetchable from a clean checkout/CI. * chore(kvquant): point engine submodule at merged main (#46 -> 8a9bc7c) Picks up #46 (quantize sliding-window layers: QuantizedBatchRotatingKVCache / DequantBatchRotatingKVCache + windowSize front-trim + sink-safe GPT-OSS kernel) plus the rotating analog of restoredFullAttentionCache so KV-quant composes with prefix-cache restore on sliding-window layers. Engine main 8a9bc7c == previously tested integration (byte-identical tree).
The calculator listed every catalog build — gemma-4-26b, gemma-4-26b-qat-4bit and gemma-4-26b-8bit (rollback) all showed as separate rows alongside gpt-oss-20b (4 entries). Dedupe by a normalized base-model key (strips quant/rollback suffixes) and keep the canonical build, so only the two real models show: Gemma 4 26B and GPT-OSS 20B. No hardcoded model list — derived from the live catalog. Mirrored across console /earn and landing; added unit tests.
…hes (#46) (#418) PR #46 (merged to engine main 8a9bc7c) made the cache factory quantize sliding-window layers too: a RotatingKVCache now maps to QuantizedBatchRotatingKVCache (kernel kind) / DequantBatchRotatingKVCache (dequant kind) instead of staying fp16 BatchRotatingKVCache. KVQuantEngineTests still encoded the pre-#46 contract ('sliding stays fp16'), so the kernel-kind factory test failed on master once #378's submodule was bumped to #46. Update the assertion to the new behavior and add the missing sliding-window assertion to the dequant-kind test. Test-only change; matches the engine-side CBCacheFactorySelectionTests already updated in #46. Verified: full KVQuantEngineTests suite (16 tests) green against engine 8a9bc7c.
The 'Inject release version into ProviderCore' step used
`grep -o '"[^"]*"'` to read the current version, which matches every
quoted string in the file. A recent comment added quoted words
("crashed"/"reloading"/"idle") on ProviderCore.swift:70, so CURRENT
became multi-line and the follow-up sed failed with 'unterminated
substitute pattern' -- breaking the v0.6.16 release run.
Replace it with a line-targeted, self-verifying sed that rewrites only
the `public static let version = "..."` line and hard-fails if the
injection didn't take.
…erence error_reason telemetry — DAR-341 (#422) gpt-oss-20b (Harmony) returns HTTP 500 on multi-turn / tool requests because its chat template deliberately raise_exception()s when an assistant message's content/thinking still carries raw <|channel|> framing replayed by a client/upstream. ~35% of gpt-oss tool requests. This is a distinct, previously-unfixed mode from the DAR-329 null-bridge crash (both collapse to the same opaque "(Jinja.TemplateException error 1.)"). Provider (Swift): - New dependency-free ProviderCoreFoundation sanitizer stripHarmonyChannelFraming(): unwraps <|channel|>final<|message|>... to the final answer and drops <|channel|>analysis<|message|>...<|end|> blocks (matches the template's own prior-turn CoT-drop contract). Conservative: non-Harmony strings pass through byte-identical. - Single shared implementation applied at every chat-template chokepoint: sanitizeJinjaMessages (stream + apply-template + prompt recount), BatchScheduler.submit, and the scan-time TemplateRenderCheck self-check (keeps "renders in self-check == renders at request time"). - Normalized inference error_reason classified from the real Error at the catch site (jinja_channel_tags / jinja_null_bridge / jinja_template / model_load), plumbed OutboundMessage -> codec -> wire. Logs the offending message index+role only, never content (E2E privacy). - Provider version 0.6.16 -> 0.6.17. Coordinator (Go): - Add optional error_reason to InferenceErrorMessage and persist it on inference_routes (in-memory + Postgres + idempotent migration). Provider-supplied reason wins (whitelisted so raw provider text can't leak), else derived from existing class/code, else unknown. - Datadog inference.error{reason,model} counter so the modes are alertable. Tests: Harmony unit + render fixtures, error classifier (incl. privacy), protocol round-trip (omitempty), coordinator precedence/persistence, and a live gpt-oss multi-turn channel-tag replay (passes against the real model: 128 tokens, no 500).
…es (#423) * feat(provider): `darkbloom beta` command to enable opt-in beta features Adds a config-backed beta-feature toggle so providers can opt into experimental capabilities — starting with KV-cache quantization (kv-quant). - BetaFeature registry (ProviderCore/Config/BetaFeatures.swift): maps each feature to a ProviderConfig field (kv-quant -> backend.kvQuant). The command and status are driven entirely off this list, so new beta flags need no per-feature code. - `darkbloom beta list|status|enable|disable` modeled on the autoupdate read-modify-write precedent; `list` supports --json. - `darkbloom status` now shows active beta features. - Docs: docs/provider/beta-features.md, kv_quant added to the [backend] reference (quickstart), and a beta section in the CLI reference. Config-backed (not env-var) on purpose: the launchd daemon only inherits an allowlist of DARKBLOOM_* vars (LaunchAgent.passthroughEnvKeys), so a TOML field is the only toggle that reliably reaches the daemon. Default stays off (fp16). Tests: BetaFeaturesTests (7) + ConfigTests green; darkbloom CLI builds; manual smoke of enable/disable/list/status/json/unknown-feature paths. * fix(provider/beta): persist toggle to post-migration canonical config (Codex P2) loadRuntimeSnapshot may migrate a legacy config to the canonical ~/.config/darkbloom/provider.toml, but snapshot.configPath stays the legacy path. Saving the beta toggle there left the canonical file — which the restarted daemon resolves first — on the stale value, so enable/disable reported success but the daemon ignored it. When no explicit --config is given, persist to ConfigManager.defaultConfigPath() (the post-migration canonical path the daemon reads); honor an explicit --config path as-is. No-op check now tests the actual save target.
* fix(coordinator): restore inference route persistence * fix(coordinator): address route persistence CI failures * test(coordinator): fix postgres store assumptions
…ccount (#427) * fix(billing): allow changing Stripe Express country by creating new account Stripe locks the country on Express accounts after creation, so users who first onboarded with the wrong country (often defaulted to US) could never change it. We now store the locked country and create a new Express account when the user selects a different country. Backend: - Add stripe_account_country column to users and User struct. - Update SetUserStripeAccount to accept and persist country. - On onboard, create a new Stripe account if requested country differs from the locked country. - Return stripe_account_country from the status endpoint. Frontend: - Pre-select the locked country in billing/earnings pages. - Show the locked country and let users pick a different one to create a new account. Tests cover both country passthrough and country-change account creation. * fix(billing): handle unknown Stripe account country
* fix(provider): normalize GPT-OSS tool template payloads * refactor(provider): isolate chat template fixes by model
#434) * feat(coordinator): route Gemma 4 only to dedicated providers; 429 shed Restrict a dedicated model family (default: gemma-4, via EIGENINFERENCE_DEDICATED_MODELS) to providers whose entire advertised catalog is that family. Enforced via one shared predicate (providerExcludedByDedicatedRuleLocked) at every public selection site: the routing gate (providerPassesRoutingGatesLockedEx, covering both the dispatch hot path and the OpenRouter capacity preflight), cold-spill, the warm-pool target picker, and the model-swap planner/warm-detection — so routing, shedding, and pre-warming can never drift. Self-route to one's own machine is exempt. When the fleet serves the model but no dedicated box is available, shed to OpenRouter as a transient 429 + Retry-After instead of 503. * feat(coordinator): fast-429 saturated dedicated boxes (skip queue-before-shed) Dedicated-family models (e.g. Gemma 4) bypass the queue-before-shed window when their dedicated boxes are at capacity: holding an OpenRouter request up to 120s would blow its TTFT SLA, so shed immediately with a 429 + Retry-After for a clean failover. Applied in both handleChatCompletions and handleGenericInference. Non-dedicated models keep queue-before-shed unchanged. Addresses Codex PR review (saturated-dedicated queue-before-shed). The structurally-unroutable 429-vs-503 nit is intentionally left at 429 per product decision (clean OpenRouter failover over an honest 503). * fix(coordinator): apply dedicated gate to alias routability checks providerCanRouteBuildLocked (used by alias resolution, constrained resolution, and RoutableProviderIDsForBuild) now applies providerExcludedByDedicatedRuleLocked, so alias routability matches dispatch routability. Fixes staged-rollout failover: an alias whose Desired build is advertised only by a mixed box now correctly resolves to a Previous build on a dedicated box instead of resolving Desired and 429ing at dispatch. Owner self-route (allowPrivate) stays exempt. Addresses Codex PR review (alias routability gate).
#393) * feat(coordinator): graceful HTTP drain for zero-downtime upgrades DAR-327 Phase 1. Lets the coordinator drain in-flight inference before a restart/swap so no request is cancelled mid-flight. - Server: atomic httpInflight counter + coordinatorDraining flag with SetDraining/IsDraining/Inflight + inc/dec helpers (new coordinator/api/drain.go). - POST /v1/admin/drain (admin-gated via isAdminAuthorized, registered raw like /v1/admin/metrics): sets draining=true; optional {"draining": false} body un-drains for rollback. - drainGate middleware (outermost) on the 4 inference routes: rejects new requests with 429+Retry-After while draining, else counts the request as in-flight for the handler's lifetime. - GET /readyz (unauthenticated): {draining,inflight,ready}; 200 when ready, 503 while draining so LBs/deploy treat draining as not-ready. - main.go: SetDraining(true) before http.Server.Shutdown on SIGINT/SIGTERM so /readyz flips immediately and in-flight requests finish within the 15s deadline. Coordinator-only; no protocol/Swift changes. Names avoid collision with the provider-side drain concepts. Tests in coordinator/api/drain_test.go (real HTTP path via srv.Handler(), no mocks) cover admin gating, 429-while-draining + Retry-After, /readyz 200/503 states, inflight inc/dec back to 0, and the non-draining pass-through regression. * fix(coordinator): un-drain honors chunked/unknown-length body (Phase 1 review) The admin-drain handler only parsed the JSON body when ContentLength > 0, so a {"draining":false} rollback sent with chunked transfer-encoding / unknown length (ContentLength == -1) was silently skipped and defaulted to draining=true, failing to un-drain. Decode whenever a body may be present (r.Body != nil && r.ContentLength != 0), tolerating an empty body (io.EOF) as the draining=true default while still capping the body for safety. Adds a regression test posting a chunked {"draining":false} with no Content-Length and asserting IsDraining() flips to false. * fix(coordinator): address Phase 1 Codex review for graceful drain Phase 1 Codex review (DAR-327, PR #393): A) /v1/admin/drain Privy-admin auth was dead — the route was registered raw with no middleware, so auth.UserFromContext was never populated and only EIGENINFERENCE_ADMIN_KEY worked (scripts/admin.sh uses a Privy token, got 403). Wrap with requireAuth (the canonical pattern shared by the other isAdminAuthorized/requireAdminKey endpoints) so a Privy admin JWT is parsed into context and the admin key is accepted as a pseudo-account; isAdminAuthorized then authorizes either. B) drainGate count-before-check race — increment in-flight BEFORE reading the drain flag (decrement + reject if draining) so /readyz inflight is a strict upper bound on requests past the gate and can never report 0 while a request is still running. C) /health is now drain-aware — returns 503 + draining:true while draining so EigenCloud Caddy marks the instance not-ready instead of keeping it in rotation (where clients hit drain 429s). D) /v1/models/capacity reports empty + draining:true while draining (not cached) so OpenRouter-style routers stop selecting a draining instance. E) SIGTERM shutdown waits for Inflight()==0 up to EIGENINFERENCE_DRAIN_GRACE (default 60s) before http.Server.Shutdown, instead of cutting streams at the 15s deadline; the 15s Shutdown remains the hard backstop. Drain logic stays in drain.go (DrainGraceFromEnv, WaitForInflightZero). Tests added/updated for A-E incl. a -race concurrency test for the gate. * fix(coordinator): keep /health as liveness (200 while draining); readiness lives on /readyz EigenCloud Caddy health-checks the single coordinator upstream on /health with health_status 200, so a 503-while-draining marked the only backend down and made the admin rollback (POST /v1/admin/drain {"draining":false}) and /readyz unreachable. /health is now a pure liveness probe (always 200, reports draining in the body); /readyz remains the readiness signal the deploy script + LBs consult. Addresses Codex re-review of #393. * fix(coordinator): DAR-327 Phase 1 review — align drain grace to 120s, correct /health liveness doc, gate-future-routes note, merge-order note * fix(coordinator): align drain grace with inference timeout
…equests (#437) * fix(coordinator): DAR-347 stop dispatch-storming unservable gpt-oss requests Oversized gpt-oss requests (prompt exact-tokenizes past the 131K context) were dispatched and retried a median of 22x (max 63, ceiling maxDispatchAttempts=64), ~8.7 min, 0% eventual success, before a final 429 — burning provider CPU on a 90%-idle fleet. Root cause: the len/4 routing estimate undercounts dense content (observed actual/est p50 1.19, p90 2.1, max 5.9), so prompts that exact-tokenize over the model context slip past the servability context tier and the provider rejects with token_budget_exhausted; the pre-content failover loop, built for transient faults, then walks the whole fleet retrying a deterministic failure. - classifyRejection: split a capacity rejection into deterministic-context ("exceeds batch token budget" / "exceeds ... context" — identical fleet-wide) vs transient (this node's KV budget / queue / drain). - dispatch loop: shouldStopFailover at the two post-dispatch retry choke points stops on the FIRST deterministic rejection, and caps transient-capacity failover at maxCapacityClassRetries=3; the exhausted ladder emits one uptime-neutral 429 (reason oversized_request) instead of a storm. Genuine fault/timeout failover unchanged. - preflight: per-family prompt-token calibration (gpt-oss x1.3, tunable via EIGENINFERENCE_PROMPT_CALIBRATION) biases the servability context check so more oversized prompts are caught before any dispatch; dispatch-time stop is the exact backstop. - telemetry: routing.oversized_request_rejected{model,stage}, routing.dispatch_to_capacity_503{model,reason}. - tests: classifier + calibration unit tests; dispatch integration tests (deterministic stop=1 attempt, transient still fails over, transient capped at 3); calibration-sheds-context preflight test. * docs: require a before/after diagram in every PR Add a Pull Requests convention to CLAUDE.md and AGENTS.md: every PR must include a before-and-after Mermaid diagram in its description detailing what changed in both behavior and code path. * docs: require before/after PR diagram in docs/AGENTS.md too Mirror the Pull Requests convention into docs/AGENTS.md so it lives in all three agent-guideline files (CLAUDE.md, AGENTS.md, docs/AGENTS.md). * fix(coordinator): DAR-347 make oversize-stop budget-aware + survive speculative race losers Two correctness gaps in the dispatch-time storm stop: #1 — "batch token budget" is not always fleet-wide deterministic. The provider rejects at min(context, activeTokenBudget) (BatchScheduler resolvedMaxTokensPerBatch), and activeTokenBudget is memory-aware (BatchScheduler+Telemetry tokenBudgetMax, floored at 1024): under pressure it drops below the context window. So the bare string can mean "prompt > context" (deterministic — every provider rejects) OR "prompt > THIS node's shrunk KV budget" (transient — a healthier provider serves). Stopping at one attempt on the latter turns a recoverable transient into a hard 429. Fix: classifyRejection now takes the rejecting provider's reported budget (ActiveTokenBudgetMax) and the model context; a "batch token budget" rejection is deterministic UNLESS we have positive evidence of memory pressure (reported budget known AND below context), in which case it stays transient (failover, capped at 3). An explicit "exceeds ... context" string is deterministic regardless of budget. Unknown budget/context preserves the prior storm-stop default. #2 — a deterministic verdict from a speculative race LOSER was dropped. The first racer to fail records into speculative tracking but NOT d.lastErr (the surviving racer owns that), so a deterministic context overflow from the loser was masked by the survivor's later transient/timeout error and the loop kept storming on the speculative path. Fix: latchDeterministicLoser classifies each loser's error (budget-aware) at the loser site and latches d.unservable; shouldStopFailover honors a latched verdict at the next retry point regardless of the survivor. Coordinator-only; no protocol change. New registry accessor Provider.ReportedTokenBudgetMaxForModel feeds the budget signal. Tests: classifyRejection budget cases (pressured→transient, unpressured/unknown→ deterministic, explicit-context ignores budget); dispatch integration (pressured batch-budget fails over; unpressured stops at 1); latch unit tests (latches deterministic, ignores transient + pressured, shouldStopFailover honors the latch). gofmt/vet clean, api+registry suites green, Linux cross-compile OK. * fix(coordinator): DAR-347 review follow-ups — refresh context after alias fallback; document budget-staleness residual Quality-gate (codex-rescue) findings on the budget-aware oversize-stop: - Stale modelMaxContext after alias fallback (real bug): handleChatCompletions rewrites `model = fallbackModel` in the capacity- and TTFT-fallback paths AFTER modelMaxContext was computed from the originally-resolved build, so the dispatch loop could compare a provider's budget against the wrong model's context window. Fix: refresh modelMaxContext from the FINAL model immediately before building the dispatchState (overwrite only on a successful store lookup). - Budget-staleness residual (documented, not a regression): providerBudget is the last heartbeat's ActiveTokenBudgetMax, not the live budget the provider rejected against, so a provider whose budget shrank below context between heartbeat and request can still be classified deterministic. This is strictly better than the pre-fix all-deterministic behavior and degrades only to a rare uptime-neutral 429, never a storm. classifyRejection now documents the limitation; the complete fix is provider-side (emit a distinct context-vs-budget rejection reason) and is a protocol-level follow-up out of scope for this coordinator-only change. go build/vet clean, api+registry suites green, Linux cross-compile OK. * fix(coordinator): DAR-347 treat past-tense/marker context-overflow wording as deterministic classifyRejection's explicit-context branch matched only "exceeds" (present tense), so capacity-class strings like "context length exceeded" / "context window exceeded" (both already listed as context markers in capacityClassMarkers and the provider breaker) fell through to rejectionTransientCapacity — retried across up to maxCapacityClassRetries providers and mis-bucketed as transient instead of stopping on the first deterministic rejection. Match both tenses ("exceeds"/"exceeded") and the bare "context length" / "context window" markers as deterministic (context overflow is model-intrinsic, fleet-wide, independent of any provider's KV budget). Latent with the current Swift provider (which emits "request exceeds batch token budget"), but closes the gap the markers list already anticipates. Tests added for the past-tense and marker-only phrasings.
…p dedicated pool warm (#439) * fix(coordinator): cap per-box concurrency at quality_concurrency; keep dedicated pool warm Gemma's dedicated pool was collapsing under load: ~80% of requests failed (53% cancelled, 27% error, telemetry over 30m) because the per-provider concurrency cap is a flat 24 while a 26B model only sustains ~2 concurrent at the 15 tok/s quality bar. Routing piled 10-23 concurrent onto a few warm boxes (decode collapsed to ~5 tok/s) while ~48 cold dedicated boxes sat idle: a cold box's ~30s model-load TTFT exceeds the 5s ceiling, so it is never routable on demand, and nothing was warming it. - Universal per-box cap = quality_concurrency x overcommit, computed from the STATIC single-stream decode rate (not the observed-under-load EWMA, which collapses under the very overload the cap prevents and would force cap=1). Slow models capped tight (Gemma->2); fast/over-provisioned models effectively unchanged (gpt-oss->20). Default on; EIGENINFERENCE_QUALITY_CONCURRENCY_CAP / _OVERCOMMIT (default 2.0). A provider-reported per-slot MaxConcurrency is honored over the computed cap. - Dedicated pools (e.g. Gemma) keep the WHOLE eligible pool warm so idle boxes enter service, with per-reason EligibleCold disqualifier instrumentation. - Disable cache-affinity pinning for dedicated models (concurrency-bound, not prompt-cache-bound) so it cannot reintroduce concentration. Coordinator-only. Cap defaults off in registry.New, so existing unit tests and the e2e testbed are unaffected. * fix(coordinator): address PR review — trusted-rate gate, cap-aware capacity feed, demand-gated dedicated warming Codex review on #439 (1 P1, 2 P2), all valid: - P1: Swift providers don't report decode_tps, so resolvedDecodeTPS falls back to sqrt(memory_bandwidth) — a model-agnostic hardware proxy that under-estimates fast models (~57 tok/s gpt-oss reads as ~28). Hard-capping a fast model from it could shed healthy traffic. Now the bandwidth fallback only caps DEDICATED models (known-slow, urgently need it); a non-dedicated model without a real benchmark keeps the flat cap until its provider reports decode_tps. - P2: ModelCapacitySnapshot (/v1/models[/capacity]) used the flat-24 headroom, so a capped Gemma box was advertised routable up to 24 and upstream routers kept sending requests this path 429s. Now uses the same quality-capped headroom as routing. - P2: the dedicated whole-pool warm target is now gated on per-build demand, so a box advertising multiple matching builds (e.g. desired+previous during an alias migration) doesn't force-warm every matching build across the pool — only the one receiving traffic. Tests: added the fallback-rate-only-caps-dedicated regression; warm-target test now covers demand-gated vs idle. Full go test ./registry/... ./api/... green. * fix(coordinator): apply quality cap in the final admit re-check (TOCTOU) Codex follow-up review (P2): the cap was applied when building the selection snapshot and in the preflight, but ReserveProviderEx's final admit re-check (providerCanAdmitLockedEx) still used the legacy flat-cap headroom. If a heartbeat bumped NumRunning between snapshot and reservation, a box that just reached its quality cap could be over-admitted via the re-check. Now the re-check uses the same quality-capped headroom. Added TestQualityCapAppliedAtAdmitRecheck.
* feat: provider base rewards — economical earnings floor (max(earned, floor)) Squashed PR #282 (5 commits) for rebase onto master: - Phase 0 bounded floor + restart-safe idempotent settlement - Phase 1 verified capacity + correctness probe - Phase 2/3 taper (pure functions, runs at taper=1) - console-ui BaseRewardsPanel + admin proxy - 24GB/32GB floor tiers; per-machine (per-account cap disabled by default) - Codex review fixes Original 5-commit history preserved at tag backup/pr282-pre-rebase. * feat(base-rewards): additive base income — earned + floor (k=0 default) Switch the provider base reward from a draw-against-earnings backstop (max(earned, floor), k=1) to additive base income: payout = earned + floor. Providers now keep 100% of organic inference earnings AND their full memory-tier floor on top; real usage is pure upside. Cost is bounded by the fixed monthly pool (FLOOR_POOL_B) and the eligibility gate rather than a per-machine clawback. - floor.go: DefaultReductionK 1.0 -> 0.0 (k=0 => Draw == floor). The k knob is retained as an optional clawback (k=1 = legacy max(earned,floor) backstop). - server_config.go: EIGENINFERENCE_BASE_REWARDS_K env default 1.0 -> 0. - engine.go/alloc.go: doc + comment updates; pool water-fill now rations a constrained pool (it no longer self-liquidates per machine). - tests: add TestDraw_K0_AdditiveDefault; happy-path now expects full $18 floor on top of $5 earned (was $13 draw under the max model). - docs/base-rewards.md: rewrite the model/decision sections for additive base income; honest note that cost no longer self-liquidates. This matches console-ui/src/app/earn/calc.ts, which already models the floor as additive ('total = usage + floor', 'not max(usage, floor)'). * chore(base-rewards): drop dead completion-drain machinery orphaned by rebase The PR originally refactored the non-streaming consumer handler to share a typed drainCompletion/completionError/completionResult helper. The rebase kept master's inline consumer drain, so only the background prober still uses this file (via awaitCompletion). Collapse completion.go to just awaitCompletion and inline the drain — the typed-error kinds, status codes, and refund-suffix fields were set but never read (the prober only needs assembled text + completion metadata). No behavior change: api tests green. * docs(base-rewards): fix stale 'reduces the draw' comment in self-route test The self-route exclusion test gets $0 because it fails the proven-work gate, not because earnings reduce the draw (the additive model has no reduction). Reword to describe the gate, matching the k=0 model. * chore(base-rewards): remove inactive taper scaffolding and tighten PR scope Remove the unwired Phase-2 taper/off-ramp scaffolding for now so the base-rewards PR matches the code we intend to ship: - delete taper.go and taper tests - remove unused taper config fields from the engine - simplify ScaledFloor to tier floor × uptime availability - remove taper/sunset copy from docs and /earn UI text - drop unrelated AGENTS/CLAUDE PR-diagram convention changes from this feature PR The shipped model is now easier to read: additive base reward (k=0), uptime availability, fixed monthly pool, work gate, and optional k-clawback if needed. * fix(base-rewards): preserve provider earning timestamps in postgres Postgres RecordProviderEarning previously omitted created_at and always used the table default NOW(). The memory store preserves caller-provided CreatedAt, and the base-rewards organic earnings tests rely on fixed timestamps for window filtering. Persist CreatedAt when supplied, falling back to time.Now() for production callers. Also address a few low-risk review cleanups: - use crypto/rand for probe jitter - replace hand-rolled insertion sorts with sort.Slice This should fix the CI-only Postgres failures in TestSumProviderEarningsByKey_Filters and TestHasBilledJobSince. * feat(base-rewards): settle prorated rewards every 5 minutes Keep the public tier floors and pool budget monthly, but prorate both into 5-minute settlement periods so provider balances move shortly after machines stay online. - add 5-minute UTC period ids and period/month proration helpers - settle the previous closed 5-minute period on startup and every 5 minutes - use prorated period floor and period pool budget in allocation - expose period_seconds and monthly_pool_budget in admin status - update tests, docs, and /earn copy for 5-minute accrual * fix(base-rewards): handle probe freshness, session grace, and M5 caps Address current Codex review findings and M5 hardware support: - include open provider sessions that overlap a settlement period via last_seen + grace - make probe eligibility depend on the latest probe outcome since the work window - attribute probe results to the provider connection actually reserved - bound probe concurrency and fail closed if crypto/rand jitter fails - add M5 Mac model identifiers and conservative memory ceilings - cap unknown hardware models to zero for base rewards until explicitly catalogued - correct Mac14,13 to the M2 Max 96GB ceiling Adds regressions for grace-overlapping sessions, latest-probe freshness, unknown hardware ineligibility, and M5 model caps. * chore(base-rewards): remove demand/probe gate and pay eligible uptime Simplify base rewards per product direction: pay eligible providers for being attested, online, healthy, loaded, linked, and above the uptime threshold regardless of organic demand. - remove the coordinator prober and shared probe completion helper - remove probe result storage, migration, tests, and store interface methods - remove billed-job/probe work-gate checks from settlement - keep organic earnings only for display and optional k-clawback math - update docs and /earn copy to describe demand-independent base rewards M5 caps and conservative unknown-model behavior remain in the hardware cap table. * fix(base-rewards): classify base rewards separately from inference work Base rewards are inserted into provider_earnings with model=base_reward so they appear in provider earnings details and summaries. Treat those rows as reward earnings in leaderboard/network totals, not inference work: they no longer add work_micro, jobs, or tokens. Adds coverage for a base-reward-only provider row in leaderboard and network totals. * fix(base-rewards): complete current Mac model memory caps Audit current Apple Silicon Mac identifiers against Apple Support, EveryMac, and AppleDB. Add the missing Mac Studio M3 Ultra identifier (Mac15,14) with its 512GB maximum-ever offered cap, document Mac16,9 as M4 Max only, and add Mac17,5 (MacBook Neo / A18 Pro) as an explicit below-floor 8GB machine. This keeps unknown future model identifiers ineligible until explicitly catalogued while ensuring current shipping Macs are represented. * fix(base-rewards): pre-merge correctness fixes 1. Dedupe existing provider_earnings.job_id duplicates before creating the unique index so the migration does not fail on production databases. 2. Abort settlement on SettleProviderFloorDraw error instead of continuing to settle lower-priority providers (preserves the failed provider's allocation for retry). 3. Treat StatusServing providers as online for reward eligibility snapshots so actively-serving providers are not dropped at settlement time. 4. Exclude base_reward rows from job counts in earnings summaries and Postgres settlement CTEs (rewards add money but are not inference jobs). 5. Clamp stale-closed sessions to min(disconnected_at, last_seen + grace) so eviction delays do not overcount uptime. 6. Skip unknown-model candidates entirely instead of settling a $0 draw that permanently blocks payment if the identifier is later catalogued.
…ashboard warning (#436) * fix: reuse durable Apple MDA attestation across reconnect + correct dashboard warning Providers showed 'Apple Device Attestation incomplete' after a restart even though they held hardware trust and were routing/earning normally. Root cause (coordinator): the MDA cert chain was discarded on every reconnect (RestoreProviderState zeroed it) and a FRESH DevicePropertiesAttestation was re-fetched — which Apple rate-limits to ~1/device/7d and which rides the same throttled MicroMDM->APNs channel as SecurityInfo. The DAR-326 trust-reuse fast-skip also grants hardware without running the MDA leg at all, so a restart left mda_verified=false indefinitely. Treat the Apple-signed MDA cert chain as durable device identity: - Recover the chain from a LIVE store read on reconnect (stageDurableMDAChain); the store record survives provider disconnect, so this works under both the in-memory and Postgres stores. The chain is persisted in Postgres (mda_cert_chain JSONB), so the proof also survives a coordinator restart. - Persist a freshly-earned chain immediately (verifyAppleDeviceAttestation and the late-MDA callback) instead of waiting on the next throttled heartbeat. - attachCachedMDAProof re-verifies the chain locally against Apple's pinned root (rejecting expired/not-yet-valid certs — the relying-party freshness check, so reuse stays correct over time) and REQUIRES the SE-key binding (FreshnessCode == sha256 of this connection's SE public key; serial cross-checked when present). A serial-only match cannot let a rotated SE key inherit a chain. - verifyAppleDeviceAttestation short-circuits on a valid cached proof; the fast-skip path attaches it too. A fresh attestation is requested only when there is no valid bound cached chain. Trust preserved: hardware trust (secure boot + SIP via MicroMDM SecurityInfo) is still required to route; the chain is re-verified every reconnect; the SE-key binding keeps the anti-relay property. Dashboard (console-ui): mda_verified is NOT a coordinator routing factor, but the /providers dashboard flagged its absence at 'degrading' severity ('EARNING (reduced priority)', 'consumers will skip this machine'). Downgraded to 'info' ('Apple Device Attestation pending — earning at full priority; earned automatically and reused across restarts') and replaced the misleading fix CTA with guidance. Tests: injectable test root (OverrideRootCAForTest); coordinator tests for live-store reuse without MDM, the cached short-circuit, expiry, anti-relay, and the SE-key/serial binding matrix; console-ui vitest asserting info-severity and routing-invariance to mda_verified. Provider Swift doctor already treats hardware trust as sufficient (no MDA warning) — verified, no change needed. * fix(coordinator): address PR review — persist reused MDA + cover ACME grant Two issues from the PR's automated review (Codex): - Persist after a successful cached reuse. attachCachedMDAProof attached the proof only in memory; the grant-path PersistProvider runs before it, so a reuse that disconnected before the next throttled heartbeat left the row without mda_cert_chain — forcing a fresh, rate-limited refetch on the next reconnect. Now persists immediately, mirroring the fresh-MDA path. Asserted by extending the live-store reuse test to check the chain is durable afterward. - Attach cached MDA when ACME grants hardware. The challenge handler's else branch can reach hardware via retryACMETrust without the live MDM verify, so verifyAppleDeviceAttestation never ran and mda_verified stayed false for ACME-trusted reconnects despite a valid cached chain. Now attaches the cached proof on that path too. (The threat-model bot's FreshnessCode concern was verified a non-issue: the reuse and fresh paths compute sha256([]byte(attestResult.PublicKey)) identically.) * fix(coordinator): bound MDA store read with a timeout (threat-model review) - stageDurableMDAChain runs on the attestation path; bound its GetProviderBySerial read with a 2s timeout so a slow/unavailable Postgres can't stall attestation (was context.Background()). On timeout we skip staging and fall back to a fresh attestation — best-effort, no behavior change on the happy path. - Document the FreshnessCode binding invariant (the reuse digest must match the fresh path's nonce formula) and fix a misplaced doc comment. The bot's store-injection note is acknowledged as low-risk: a tampered stored chain is rejected by the Apple-root re-verify in attachCachedMDAProof (DoS-only, falls back to fresh — no trust escalation). * fix(coordinator): recover newest non-empty MDA chain by serial (PR review) Codex flagged a durability race: persists are async (persistProviderNow spawns a goroutine), so a reconnect's grant-path persist can snapshot an empty chain and land AFTER attachCachedMDAProof's chain persist. The new row (fresh provider id, same serial) then shadows the prior chain-bearing row — GetProviderBySerial / the memory serial index return the newer empty row, so the next reconnect falls back to a fresh, rate-limited Apple attestation. Add store.GetMDAChainBySerial: returns the newest NON-EMPTY mda_cert_chain for a serial, looking past empty rows (memory: scan by serial + LastSeen; Postgres: WHERE mda_cert_chain IS NOT NULL ORDER BY last_seen DESC). stageDurableMDAChain uses it, so reuse survives the persist race regardless of write ordering. No write-semantics change and no display drift (staging stays staging). Covered by a memory-store test that shadows a chain row with a newer empty row. * fix(coordinator): attach cached MDA on late-SecurityInfo grant (PR review) Fourth and final hardware-grant path. ApplyLateSecurityInfo upgrades a provider to hardware when SecurityInfo arrives after the synchronous verify timed out; it grants via GrantHardwareIfNotUntrusted and the MDM loop then exits on seeing hardware, so verifyAppleDeviceAttestation never runs and a staged durable chain is never attached — leaving the provider hardware/mda_verified=false despite a valid reusable proof. Attach the cached proof on that path too, matching the synchronous-MDM, fast-skip, and ACME grant paths.
…p path (#440) Coordinator startup ran an expensive `DELETE ... GROUP BY job_id` dedupe on the hot provider_earnings table (~900k rows / ~443MB) inside NewPostgres -> migrate. On deploy it full-scanned and held a relation lock for ~15m, blocking the subsequent CREATE INDEX statements so the HTTP server never bound :8080 — a production outage. Prod duplicate count is 0, so it did no useful work. - Remove the boot-time dedupe DELETE entirely. - Build the partial unique index idx_provider_earnings_job(job_id) WHERE job_id <> '' after the migration loop via ensureProviderEarningsJobIndex: valid-index fast path (no-op on every boot after the first), a non-destructive duplicate-count guard that fails loudly with an actionable message instead of deleting rows, and a CONCURRENTLY build (no write-blocking lock, so a blue-green old coordinator is unaffected) via the simple query protocol. - The ON CONFLICT (job_id) DO NOTHING write path (RecordProviderEarning, CreditProviderAccount) is already idempotent and now reliably backed by the index. - Offline dedupe moved to coordinator/store/migrations/dedupe_provider_earnings.sql for the (currently non-existent) duplicate case. - Tests: always-on source guard against reintroducing the boot dedupe, plus a postgres-gated test asserting a valid index after startup, migrate() re-entrancy, and idempotent earnings writes.
…eanly as 429 (#443) * fix: inference reliability — kill retry storm, eject zombies, shed cleanly as 429 RCA of prod OpenRouter-facing uptime (~70% gpt-oss / ~83% gemma excl-400, dipping to 43% at peak) traced every symptom to a small set of root causes. OpenRouter is ~99% of traffic and does NOT count 429 as downtime, so the strategy is: convert anything unservable into a pre-dispatch 429 / one-shot 4xx — never a served 5xx, never a dispatch-storm. All coordinator changes are gated + fail-open with live kill-switches; no prod-config flips; warm-pool untouched. Coordinator (shippable now, fully tested + dual-reviewer PASS): - C1: StatusCode-driven non-retryable failover stop on client-shape 4xx {400,413,422,415} (excl 404/429), mirrored at the speculative-loser latch and surfaced once through the exhausted ladder; new client_error route-outcome class (no AdmittedButFailed). Kills the 29x/max-63 retry storm on deterministic 400s. - C4: pre-dispatch reject of remote/non-data: media URLs (the ~537/hr gemma vision-400 storm) with one clean 400; fail-open on unknown shapes. - C3 (core): servability shed uses live remaining budget (committed-subtracted) so predictable-oversized -> pre-dispatch 429; classifyRejection consumes P1 structured reasons with the legacy string heuristic as the old-provider fallback. - C2: stable-identity (serial/SE-key/account) health-ejection that survives reconnect churn -> ejects zombie providers (0 successes, constant reconnects); reads attestation via the thread-safe accessor (race-clean). - C5: decode-quality floor uses the tps-registry median for idle boxes so historically-slow gemma boxes are deprioritized (client_gone). Provider (release-gated; Go contract verified, Swift compile is the release build): - P1: emit a distinct context-overflow vs node-budget rejection so the coordinator classifies deterministic-vs-transient without guessing from a stale heartbeat. - P2: normalize parallel tool_calls into sequential Harmony turns instead of a hard 400, recovering gpt-oss agent traffic that was previously unservable. Kill switches: EIGENINFERENCE_DISABLE_CLIENT_ERROR_STOP, DARKBLOOM_VISION_REJECT_REMOTE_URLS, EIGENINFERENCE_SERVABILITY_GATE, EIGENINFERENCE_HEALTH_EJECTION, EIGENINFERENCE_DECODE_FLOOR_USE_FLEET_MEDIAN. Deferred (gated default-off, follow-up): C3 max_tokens auto-clamp / route-reject / 413 / first_chunk_timeout->429. Tests: go test ./api/ ./registry/ green (unit + httptest integration + -race on the new health-ejection breaker); P1/P2 Swift tests run at the release gate. * fix(coordinator): C2 — read stable identity directly in routing gate (fix self-deadlock) The prior C2 race-fix routed the identity read through p.GetAttestationResult(), which re-locks p.mu. But providerPassesRoutingGatesLockedEx is called from snapshotProviderLocked* with p.mu ALREADY HELD (it reads p.Status/p.TrustLevel the same direct way), so re-locking self-deadlocked the routing path — TestAlgorithm_P1_ RejectsProviderTooSmallForModel hung 600s. Read p.AttestationResult/p.AccountID directly in stableProviderIdentityLocked (consistent with the gate's other p-field reads, p.mu held by the caller); the only lock-free caller, GetProviderStableIdentity, takes p.mu itself before calling (not nested with r.mu → no deadlock, no race). Full registry suite green (1.1s, was 600s timeout); -race clean on the routing/ejection path. * chore: drop PLAN.md/PLAN-specs.md from PR (planning artifacts, not repo content)
…rovider) (#444) * refactor(provider): delete dead single-request inference/SSE/formatter stack The production inference path is MultiModelBatchSchedulerEngine+Translation + BatchScheduler. The single-request driver, its SSE formatter, and the chat-prompt formatter were only referenced by tests. Remove them and the types they orphaned: - delete SingleRequestInference.swift, SSEChunkFormatting.swift, ChatPromptFormatting.swift - drop orphaned InferenceUsage + UsageAccumulator (UsageAccounting.swift), keeping the live StreamedGenerationUsage / CancelledGenerationTerminal - drop orphaned SSEChunk + streaming ChatCompletionChunk/ChunkChoice/ ChunkDelta (ChatRequest.swift); keep ChunkUsage + ChatCompletionResponse (still used by the live response path / InferenceLiveTests) - trim InferenceTests.swift to the live cancellation-registry test - fix the now-dangling ChatPromptFormatter mention in the translate() doc Behavior-preserving: swift build green; swift test 1049 pass / 0 fail (baseline 1054 minus the 5 removed dead tests). Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(coordinator): delete dead parallel routing implementation Remove the score-based FindProvider / FindProviderWithTrust / ScoreProvider cluster (plus the now-unused TrustMultiplier and the MaxConcurrentRequests alias) from the registry. Production dispatch routes through the cost-based ReserveProviderEx -> selectBestCandidate path; these functions were a second routing brain reachable only from tests, giving false confidence by testing behavior that diverged from production (6m vs 16m challenge freshness window, score-vs-cost selection, crashed-provider routing). Tests that exercised only the dead path (scoring math, score-based selection, dead-specific challenge boundaries, crashed-only routing) are deleted. Tests that used FindProvider as a routability probe for gates SHARED with production (privacy caps, manifest/SIP checks, trust, catalog, eviction, challenge freshness, concurrency cap) are repointed to a findRoutableProvider helper that drives the real ReserveProviderEx path; stale challenge times were bumped past the 16m production freshness window to preserve staleness coverage. No production behavior change. build/vet/test/gofmt all green. Co-authored-by: Cursor <cursoragent@cursor.com> * chore(coordinator): untrack built binary and drop dead R2 site-packages config - git rm --cached coordinator/coordinator: the 40MB built binary was tracked despite already being in .gitignore. Stop committing the artifact. - Remove Server.r2SitePackagesCDNURL plus its ServerConfig field and env wiring (EIGENINFERENCE_R2_SITE_PACKAGES_CDN_URL). It was written from env but never read: install.sh stopped substituting __DARKBLOOM_R2_SITE_PACKAGES_CDN_URL__ post-Swift-cutover (v0.5.0+), so the field was pure dead config. Co-authored-by: Cursor <cursoragent@cursor.com> * chore(coordinator): stop tracking the built coordinator binary The 40MB coordinator/coordinator artifact was tracked despite already being listed in .gitignore. Untrack it so builds don't keep committing the binary. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(coordinator): collapse dead config->env re-export shim config/config.go only re-exported env helpers (EnvOr/FirstNonEmpty/EnvFloat/ EnvInt). FirstNonEmpty/EnvFloat/EnvInt had zero callers, and EnvOr was used only by app_config.go in the same package. Point app_config.go at env.EnvOr directly, move the package doc onto app_config.go, and delete the shim so there is a single env-helper source of truth. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(coordinator): merge alias capacity/TTFT fallbacks into one helper maybeFallbackAliasCapacity and maybeFallbackAliasTTFT were near-identical copies differing only in the TTFT ceiling check and the failure-path return model. Merge into maybeFallbackAlias(parsed, mode, ...) with an aliasFallbackMode (capacity vs ttft) flag; capacity callers pass ttftThreshold=0. Failure-path return semantics are preserved exactly (capacity reports currentModel, TTFT reports the probed previous build). No behavior change. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(provider): extract Benchmark into its own ProviderBenchmark target Benchmark/ (~5.2k LOC, incl. fatalError protocol-conformance stub caches) was compiled into the ProviderCore library and thus linked into every provider/enclave binary. Move it into a dedicated `ProviderBenchmark` SwiftPM target that depends on ProviderCore, so only the benchmark-bearing executables link it (darkbloom `benchmark`, kv-quant-gate, kv-attn-selftest, kv-engine-demo) and the shipped enclave/publish/core surface no longer carries benchmark code. - new target `ProviderBenchmark` (Sources/ProviderBenchmark), 18 files moved - the two engine-facing types the live scheduler needs stay in ProviderCore: KVQuantCandidateMode (split out of KVQuantTypes.swift) and the self-contained KVQuantPolicy.swift, now under ProviderCore/Inference/KVQuant - widen KVEstimation / parseModelArchitecture / resolvedKVBytesPerToken / ModelArchitecture to public (already used by the live scheduler) so the benchmark target can reach them across the module boundary - wire ProviderBenchmark into the four executables + the four benchmark test files; add `import ProviderCore` to the moved files Behavior-preserving: swift build green; swift test 1049 pass / 0 fail. Co-authored-by: Cursor <cursoragent@cursor.com> * chore(coordinator): drop dead vllm-mlx full-stack test and fix stale comments - Delete api/fullstack_integration_test.go (~1155 LOC): it spawned real vllm-mlx subprocesses (the retired subprocess backend), was gated behind LIVE_FULLSTACK_TEST=1 (never run in CI), and the repo-root e2e/ harness covers the current MLX-Swift provider over the WebSocket relay. - Refresh stale vllm-mlx comments to MLX-Swift / E2E-relay reality in consumer.go, protocol/messages.go, and registry.go (comments only; no wire-type changes). - Fix install.sh path comment (internal/api -> api) in server.go. - Reword the telemetry symmetry test comment so it states the Go test pins the shape and the Swift/TS mirrors must match, without asserting mirror tests this package cannot see. Co-authored-by: Cursor <cursoragent@cursor.com> * test(provider): add telemetry wire-symmetry tests mirroring the Go canonical coordinator/protocol/telemetry_symmetry_test.go references a Swift mirror that did not exist. Add Tests/ProviderCoreTests/TelemetrySymmetryTests.swift to pin: - enum casing (source=provider, severity=error, kind=backend_crash) - snake_case keys + nil-optional omission (machine_id/account_id/request_id/ stack/fields), matching Go omitempty - the TelemetryKind set + count (mirror of Go KnownKinds()) - TelemetrySource/TelemetrySeverity raw values Add CaseIterable to TelemetryKind (the Swift equivalent of KnownKinds()) and fix two stale `coordinator/internal/protocol|api/...` comment paths to the current `coordinator/protocol|api/...` locations. swift build green; swift test 1052 pass / 0 fail (+3 new). Co-authored-by: Cursor <cursoragent@cursor.com> * fix(coordinator): stop silently swallowing billing/credit and WS-send errors Replace `_ = ...` discards with explicit error handling + a DogStatsD counter: - Stripe webhook: CompleteBillingSession / Referral.Apply failures now log an error and increment billing.{session_complete,referral_apply}_failed instead of vanishing after a successful deposit. - Settlement: failed consumer refund and platform-fee Credit calls now log an error and increment billing.credit_failed{op} — these are financial and must never be silent (a dropped refund over-charges the consumer). - Provider WS: best-effort EnqueueText sends (runtime/trust status) now log at debug and increment provider.enqueue_failed{msg} so a wedged send is visible. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(provider): centralize Harmony assistant-message stripping The assistant-message Harmony wrapper (role guard + content/thinking/ reasoning_content field iteration) was duplicated in JinjaSanitization and TemplateRenderCheck. Centralize it in ProviderCoreFoundation alongside the existing string transform: - add `harmonyAssistantTextFields` + `stripHarmonyFramingFromAssistantMessage` in HarmonySanitization.swift (Linux-buildable, no Apple deps) - TemplateRenderCheck now calls the shared helper (drops its private copy) - JinjaSanitization iterates the shared field-list constant - BatchScheduler already routes through the shared `stripHarmonyChannelFraming` Behavior-preserving: swift build green; swift test 1052 pass / 0 fail. Co-authored-by: Cursor <cursoragent@cursor.com> * test(coordinator): cover auth/env/telemetry; doc pendingModelLoads routing scope - auth: first unit tests for the previously-untested package — PEM/key parsing (NewPrivyAuth), ES256 JWT verification incl. expiry/issuer/audience/wrong-key/ alg-confusion rejection (VerifyToken), user provisioning (GetOrCreateUser), and Config.Check. Uses a real in-memory store and real generated keys; no mocks of the unit under test. - env: 100% coverage of EnvOr/FirstNonEmpty/EnvFloat/EnvInt/EnvBool. - telemetry: Emitter default-fill, metric counting (capturing fake sink), nil-emitter/nil-sink safety, and version defaulting. - registry: document that pendingModelLoads is consulted ONLY by warm-pool/swap planning and never participates in routing/admission (see AGENTS.md state model). Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(provider): consolidate formatDuration into DurationFormatting Four near-identical duration formatters had diverged in user-visible ways (download progress shows seconds; idle/status logs use compact "2h30m"; the scheduler uses spaced "2h 30m"; trailing zero-minute elision differed). Add one `DurationFormatting.compact` helper whose flags reproduce each style exactly, and delegate the four call-site helpers to it. Output is unchanged at every call site; the logic now lives in one place. Behavior-preserving: swift build green; swift test 1052 pass / 0 fail. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(coordinator): extract request introspection helpers from consumer.go Move the pure request-introspection / token-estimation cluster out of the 5.7k-LOC consumer.go into a focused request_introspection.go (token + billing cost estimation, media/tool detection, cache-affinity keys, provider-serial allowlist parsing, and the media-cost constants). Same package, no behavior change — these functions have no Server dependency. consumer.go drops from ~5700 to ~5311 LOC. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(provider): surface CoordinatorClient encode failures instead of silent {} Heartbeat / outbound / inference-error encoding used `try?` and degraded to hardcoded `{}` or minimal JSON on failure — silent protocol corruption the coordinator can't attribute. Replace with do/catch that logs at error and emits a `protocol_error` telemetry event, and make the inference-error path fall back to a parseable, correctly-typed payload (carrying the real request_id/error/status_code) rather than `{}`. Behavior-preserving on the success path: swift build green; swift test 1052 pass / 0 fail. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(provider): replace force casts with guards; modernize a GCD timeout - PersistentEnclaveKey: guard the Keychain `result` optional and throw a typed keyLookupFailed instead of force-unwrapping nil (the CFTypeRef->SecKey cast is compiler-guaranteed). - ProtocolSafeQuantizedKVCache.copy(): assert the inner-copy type via guard + preconditionFailure with a clear message rather than a blind `as!`. - ProviderLoop.waitForPreloads: replace `DispatchQueue.global().asyncAfter` with a structured `Task.sleep(for:)` timeout (OneShotBoolContinuation still dedupes, so the first-resume-wins race is unchanged). Behavior-preserving: swift build green; swift test 1052 pass / 0 fail. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(coordinator): extract API key handlers from consumer.go Move the consumer-facing API key management endpoints and their validation/response helpers (create/list/get/update/rotate/delete, apiKeyToResponse, validateKeyLimitInputs, keyModelAllowed, checkKeySpendCap, applyKeyPatch) into apikey_handlers.go, matching the existing apikey_handlers_test.go. Same package, no behavior change. consumer.go drops from ~5311 to ~4884 LOC. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(provider): extract ProviderLoopConfig and ProviderLogger from ProviderLoop First, low-risk step of breaking up the ProviderLoop god file: move the two standalone top-level value types out of the 3.6k-line ProviderLoop.swift into their own single-responsibility files (no actor internals touched, no access loosened). The loop file now holds the actor + its behavior, not its inputs or its logger type. Note: the actor's instance-method sections are intentionally NOT split across files here — ProviderLoop deliberately keeps method extensions in-file to reach its `private` model registry without loosening ~40 members to `internal` (see the documented rationale at the local-endpoint extension). That deeper split is deferred pending an explicit encapsulation decision. Behavior-preserving: swift build green; swift test 1052 pass / 0 fail. Co-authored-by: Cursor <cursoragent@cursor.com> * test(provider): cover the darkbloom-publish hash CLI entrypoint darkbloom-publish (the security-critical model-manifest hasher used by the registry publish flow) had zero tests on its CLI surface. Add a DarkbloomPublishTests target exercising the `hash` subcommand end-to-end against a temp snapshot dir: - writes a manifest.json that decodes back to the expected id/version/files/ 64-hex aggregate - hashing is deterministic across runs on identical bytes - rejects unsafe model ids and version tags (spaces, '..', '/', empty) before any hashing happens (The ManifestBuilder library itself remains covered by ProviderCoreFoundationTests.) swift build green; swift test 1052 swift-testing + 4 new XCTest pass / 0 fail. Co-authored-by: Cursor <cursoragent@cursor.com> * chore(provider): remove unused dev-only harness targets vlm-smoke (explicitly "Safe to delete") and kv-engine-demo (DAR-318 capacity demo, "NOT a product") are dev-only executables referenced nowhere outside Package.swift — no CI, scripts, Makefile, or docs build them. Delete both (~1040 LOC) to shrink the default build surface. Kept kv-se-harness: it is the only way to exercise the Secure-Enclave KEK + Keychain round-trip on real SE hardware (a path `swift test` cannot reach), so it retains test value despite not being a product. Behavior-preserving: swift build green; swift test 1052 pass / 0 fail. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(e2e): route via ReserveProviderEx after FindProvider removal Round-1 cleanup deleted registry.FindProvider but only re-ran go test inside coordinator/, missing the e2e package. The lone surviving caller in TestIntegration_SwiftProviderRealRoutingGates broke the e2e build (Blacksmith e2e/Build). Mirror the in-coordinator migration: add a local findRoutableProvider helper that probes routability through the production ReserveProviderEx path (same structural/privacy/trust/challenge/capacity gates), reserves then releases capacity, and returns the selected provider. No deleted methods or shims reintroduced. Verified: go build ./... and go vet ./... (compiles e2e/) green from repo root. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(provider): split KV-quant benchmark out of shipped darkbloom scripts/install.sh ships the `darkbloom` binary, but the `darkbloom` target depended on ProviderBenchmark, which carried the KV-quant gate/eval code AND the fatalError protocol-conformance KV-cache stubs (ProtocolSafeQuantizedKVCache, the V-only/BFloat16 caches). Those benchmark-only stubs were linking into every installed provider, defeating round-1's goal. Split ProviderBenchmark into two targets along the existing clean boundary (verified: the lightweight runners and the KVQuant/ dir do not reference each other): - ProviderBenchmark (lean): ModelBenchmark, ThroughputSweep(+report), DecodeBandwidthModel — the only benchmark code `darkbloom benchmark` needs. Still linked by `darkbloom`, so the command keeps working. - ProviderBenchmarkKVQuant (heavy, has the fatalError stubs): the KVQuant/ dir, linked ONLY by kv-quant-gate, kv-attn-selftest, and their tests — never by `darkbloom`. kv-quant-gate / kv-attn-selftest / the 3 KVQuant test files now import ProviderBenchmarkKVQuant; the darkbloom benchmark command and ThroughputSweepTests keep importing the lean ProviderBenchmark. Bundle-safe: release builds by product (`swift build -c release --product darkbloom`); product names, install paths, and LatestProviderVersion are unchanged. Verified: `swift build --build-tests` green (all targets + tests). nm proof: darkbloom links 0 ProviderBenchmarkKVQuant symbols (939 lean ProviderBenchmark symbols intact); kv-quant-gate links 3694. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(coordinator): tighten maybeFallbackAlias branching Behavior-preserving cleanup of the alias capacity/TTFT fallback helper. Both fallback modes already share a SINGLE Previous-build capacity probe (QuickCapacityCheckWithTTFTForRequest) — the desired build is probed once at the call site and the previous build once here, on different models, so there is no redundant capacity check to remove. This reduces the residual branching instead: - merge the two identical modelShed / !IsModelInCatalog early-return guards (both pure, both return the same tuple) into one - hoist the duplicated `mode == aliasFallbackTTFT` test into a single `enforceTTFT` bool and drop the `tooSlow` temp The failure-path model selection is unchanged (TTFT mode reports the probed Previous build the caller uses as the alternate TTFT estimate; capacity mode keeps the current build it discards), so all returns are byte-identical to before. Verified: coordinator/api alias fallback tests + full `go test ./...` green. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>
… (stacked on #444) (#445) * refactor(provider): delete dead single-request inference/SSE/formatter stack The production inference path is MultiModelBatchSchedulerEngine+Translation + BatchScheduler. The single-request driver, its SSE formatter, and the chat-prompt formatter were only referenced by tests. Remove them and the types they orphaned: - delete SingleRequestInference.swift, SSEChunkFormatting.swift, ChatPromptFormatting.swift - drop orphaned InferenceUsage + UsageAccumulator (UsageAccounting.swift), keeping the live StreamedGenerationUsage / CancelledGenerationTerminal - drop orphaned SSEChunk + streaming ChatCompletionChunk/ChunkChoice/ ChunkDelta (ChatRequest.swift); keep ChunkUsage + ChatCompletionResponse (still used by the live response path / InferenceLiveTests) - trim InferenceTests.swift to the live cancellation-registry test - fix the now-dangling ChatPromptFormatter mention in the translate() doc Behavior-preserving: swift build green; swift test 1049 pass / 0 fail (baseline 1054 minus the 5 removed dead tests). Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(coordinator): delete dead parallel routing implementation Remove the score-based FindProvider / FindProviderWithTrust / ScoreProvider cluster (plus the now-unused TrustMultiplier and the MaxConcurrentRequests alias) from the registry. Production dispatch routes through the cost-based ReserveProviderEx -> selectBestCandidate path; these functions were a second routing brain reachable only from tests, giving false confidence by testing behavior that diverged from production (6m vs 16m challenge freshness window, score-vs-cost selection, crashed-provider routing). Tests that exercised only the dead path (scoring math, score-based selection, dead-specific challenge boundaries, crashed-only routing) are deleted. Tests that used FindProvider as a routability probe for gates SHARED with production (privacy caps, manifest/SIP checks, trust, catalog, eviction, challenge freshness, concurrency cap) are repointed to a findRoutableProvider helper that drives the real ReserveProviderEx path; stale challenge times were bumped past the 16m production freshness window to preserve staleness coverage. No production behavior change. build/vet/test/gofmt all green. Co-authored-by: Cursor <cursoragent@cursor.com> * chore(coordinator): untrack built binary and drop dead R2 site-packages config - git rm --cached coordinator/coordinator: the 40MB built binary was tracked despite already being in .gitignore. Stop committing the artifact. - Remove Server.r2SitePackagesCDNURL plus its ServerConfig field and env wiring (EIGENINFERENCE_R2_SITE_PACKAGES_CDN_URL). It was written from env but never read: install.sh stopped substituting __DARKBLOOM_R2_SITE_PACKAGES_CDN_URL__ post-Swift-cutover (v0.5.0+), so the field was pure dead config. Co-authored-by: Cursor <cursoragent@cursor.com> * chore(coordinator): stop tracking the built coordinator binary The 40MB coordinator/coordinator artifact was tracked despite already being listed in .gitignore. Untrack it so builds don't keep committing the binary. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(coordinator): collapse dead config->env re-export shim config/config.go only re-exported env helpers (EnvOr/FirstNonEmpty/EnvFloat/ EnvInt). FirstNonEmpty/EnvFloat/EnvInt had zero callers, and EnvOr was used only by app_config.go in the same package. Point app_config.go at env.EnvOr directly, move the package doc onto app_config.go, and delete the shim so there is a single env-helper source of truth. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(coordinator): merge alias capacity/TTFT fallbacks into one helper maybeFallbackAliasCapacity and maybeFallbackAliasTTFT were near-identical copies differing only in the TTFT ceiling check and the failure-path return model. Merge into maybeFallbackAlias(parsed, mode, ...) with an aliasFallbackMode (capacity vs ttft) flag; capacity callers pass ttftThreshold=0. Failure-path return semantics are preserved exactly (capacity reports currentModel, TTFT reports the probed previous build). No behavior change. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(provider): extract Benchmark into its own ProviderBenchmark target Benchmark/ (~5.2k LOC, incl. fatalError protocol-conformance stub caches) was compiled into the ProviderCore library and thus linked into every provider/enclave binary. Move it into a dedicated `ProviderBenchmark` SwiftPM target that depends on ProviderCore, so only the benchmark-bearing executables link it (darkbloom `benchmark`, kv-quant-gate, kv-attn-selftest, kv-engine-demo) and the shipped enclave/publish/core surface no longer carries benchmark code. - new target `ProviderBenchmark` (Sources/ProviderBenchmark), 18 files moved - the two engine-facing types the live scheduler needs stay in ProviderCore: KVQuantCandidateMode (split out of KVQuantTypes.swift) and the self-contained KVQuantPolicy.swift, now under ProviderCore/Inference/KVQuant - widen KVEstimation / parseModelArchitecture / resolvedKVBytesPerToken / ModelArchitecture to public (already used by the live scheduler) so the benchmark target can reach them across the module boundary - wire ProviderBenchmark into the four executables + the four benchmark test files; add `import ProviderCore` to the moved files Behavior-preserving: swift build green; swift test 1049 pass / 0 fail. Co-authored-by: Cursor <cursoragent@cursor.com> * chore(coordinator): drop dead vllm-mlx full-stack test and fix stale comments - Delete api/fullstack_integration_test.go (~1155 LOC): it spawned real vllm-mlx subprocesses (the retired subprocess backend), was gated behind LIVE_FULLSTACK_TEST=1 (never run in CI), and the repo-root e2e/ harness covers the current MLX-Swift provider over the WebSocket relay. - Refresh stale vllm-mlx comments to MLX-Swift / E2E-relay reality in consumer.go, protocol/messages.go, and registry.go (comments only; no wire-type changes). - Fix install.sh path comment (internal/api -> api) in server.go. - Reword the telemetry symmetry test comment so it states the Go test pins the shape and the Swift/TS mirrors must match, without asserting mirror tests this package cannot see. Co-authored-by: Cursor <cursoragent@cursor.com> * test(provider): add telemetry wire-symmetry tests mirroring the Go canonical coordinator/protocol/telemetry_symmetry_test.go references a Swift mirror that did not exist. Add Tests/ProviderCoreTests/TelemetrySymmetryTests.swift to pin: - enum casing (source=provider, severity=error, kind=backend_crash) - snake_case keys + nil-optional omission (machine_id/account_id/request_id/ stack/fields), matching Go omitempty - the TelemetryKind set + count (mirror of Go KnownKinds()) - TelemetrySource/TelemetrySeverity raw values Add CaseIterable to TelemetryKind (the Swift equivalent of KnownKinds()) and fix two stale `coordinator/internal/protocol|api/...` comment paths to the current `coordinator/protocol|api/...` locations. swift build green; swift test 1052 pass / 0 fail (+3 new). Co-authored-by: Cursor <cursoragent@cursor.com> * fix(coordinator): stop silently swallowing billing/credit and WS-send errors Replace `_ = ...` discards with explicit error handling + a DogStatsD counter: - Stripe webhook: CompleteBillingSession / Referral.Apply failures now log an error and increment billing.{session_complete,referral_apply}_failed instead of vanishing after a successful deposit. - Settlement: failed consumer refund and platform-fee Credit calls now log an error and increment billing.credit_failed{op} — these are financial and must never be silent (a dropped refund over-charges the consumer). - Provider WS: best-effort EnqueueText sends (runtime/trust status) now log at debug and increment provider.enqueue_failed{msg} so a wedged send is visible. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(provider): centralize Harmony assistant-message stripping The assistant-message Harmony wrapper (role guard + content/thinking/ reasoning_content field iteration) was duplicated in JinjaSanitization and TemplateRenderCheck. Centralize it in ProviderCoreFoundation alongside the existing string transform: - add `harmonyAssistantTextFields` + `stripHarmonyFramingFromAssistantMessage` in HarmonySanitization.swift (Linux-buildable, no Apple deps) - TemplateRenderCheck now calls the shared helper (drops its private copy) - JinjaSanitization iterates the shared field-list constant - BatchScheduler already routes through the shared `stripHarmonyChannelFraming` Behavior-preserving: swift build green; swift test 1052 pass / 0 fail. Co-authored-by: Cursor <cursoragent@cursor.com> * test(coordinator): cover auth/env/telemetry; doc pendingModelLoads routing scope - auth: first unit tests for the previously-untested package — PEM/key parsing (NewPrivyAuth), ES256 JWT verification incl. expiry/issuer/audience/wrong-key/ alg-confusion rejection (VerifyToken), user provisioning (GetOrCreateUser), and Config.Check. Uses a real in-memory store and real generated keys; no mocks of the unit under test. - env: 100% coverage of EnvOr/FirstNonEmpty/EnvFloat/EnvInt/EnvBool. - telemetry: Emitter default-fill, metric counting (capturing fake sink), nil-emitter/nil-sink safety, and version defaulting. - registry: document that pendingModelLoads is consulted ONLY by warm-pool/swap planning and never participates in routing/admission (see AGENTS.md state model). Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(provider): consolidate formatDuration into DurationFormatting Four near-identical duration formatters had diverged in user-visible ways (download progress shows seconds; idle/status logs use compact "2h30m"; the scheduler uses spaced "2h 30m"; trailing zero-minute elision differed). Add one `DurationFormatting.compact` helper whose flags reproduce each style exactly, and delegate the four call-site helpers to it. Output is unchanged at every call site; the logic now lives in one place. Behavior-preserving: swift build green; swift test 1052 pass / 0 fail. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(coordinator): extract request introspection helpers from consumer.go Move the pure request-introspection / token-estimation cluster out of the 5.7k-LOC consumer.go into a focused request_introspection.go (token + billing cost estimation, media/tool detection, cache-affinity keys, provider-serial allowlist parsing, and the media-cost constants). Same package, no behavior change — these functions have no Server dependency. consumer.go drops from ~5700 to ~5311 LOC. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(provider): surface CoordinatorClient encode failures instead of silent {} Heartbeat / outbound / inference-error encoding used `try?` and degraded to hardcoded `{}` or minimal JSON on failure — silent protocol corruption the coordinator can't attribute. Replace with do/catch that logs at error and emits a `protocol_error` telemetry event, and make the inference-error path fall back to a parseable, correctly-typed payload (carrying the real request_id/error/status_code) rather than `{}`. Behavior-preserving on the success path: swift build green; swift test 1052 pass / 0 fail. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(provider): replace force casts with guards; modernize a GCD timeout - PersistentEnclaveKey: guard the Keychain `result` optional and throw a typed keyLookupFailed instead of force-unwrapping nil (the CFTypeRef->SecKey cast is compiler-guaranteed). - ProtocolSafeQuantizedKVCache.copy(): assert the inner-copy type via guard + preconditionFailure with a clear message rather than a blind `as!`. - ProviderLoop.waitForPreloads: replace `DispatchQueue.global().asyncAfter` with a structured `Task.sleep(for:)` timeout (OneShotBoolContinuation still dedupes, so the first-resume-wins race is unchanged). Behavior-preserving: swift build green; swift test 1052 pass / 0 fail. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(coordinator): extract API key handlers from consumer.go Move the consumer-facing API key management endpoints and their validation/response helpers (create/list/get/update/rotate/delete, apiKeyToResponse, validateKeyLimitInputs, keyModelAllowed, checkKeySpendCap, applyKeyPatch) into apikey_handlers.go, matching the existing apikey_handlers_test.go. Same package, no behavior change. consumer.go drops from ~5311 to ~4884 LOC. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(provider): extract ProviderLoopConfig and ProviderLogger from ProviderLoop First, low-risk step of breaking up the ProviderLoop god file: move the two standalone top-level value types out of the 3.6k-line ProviderLoop.swift into their own single-responsibility files (no actor internals touched, no access loosened). The loop file now holds the actor + its behavior, not its inputs or its logger type. Note: the actor's instance-method sections are intentionally NOT split across files here — ProviderLoop deliberately keeps method extensions in-file to reach its `private` model registry without loosening ~40 members to `internal` (see the documented rationale at the local-endpoint extension). That deeper split is deferred pending an explicit encapsulation decision. Behavior-preserving: swift build green; swift test 1052 pass / 0 fail. Co-authored-by: Cursor <cursoragent@cursor.com> * test(provider): cover the darkbloom-publish hash CLI entrypoint darkbloom-publish (the security-critical model-manifest hasher used by the registry publish flow) had zero tests on its CLI surface. Add a DarkbloomPublishTests target exercising the `hash` subcommand end-to-end against a temp snapshot dir: - writes a manifest.json that decodes back to the expected id/version/files/ 64-hex aggregate - hashing is deterministic across runs on identical bytes - rejects unsafe model ids and version tags (spaces, '..', '/', empty) before any hashing happens (The ManifestBuilder library itself remains covered by ProviderCoreFoundationTests.) swift build green; swift test 1052 swift-testing + 4 new XCTest pass / 0 fail. Co-authored-by: Cursor <cursoragent@cursor.com> * chore(provider): remove unused dev-only harness targets vlm-smoke (explicitly "Safe to delete") and kv-engine-demo (DAR-318 capacity demo, "NOT a product") are dev-only executables referenced nowhere outside Package.swift — no CI, scripts, Makefile, or docs build them. Delete both (~1040 LOC) to shrink the default build surface. Kept kv-se-harness: it is the only way to exercise the Secure-Enclave KEK + Keychain round-trip on real SE hardware (a path `swift test` cannot reach), so it retains test value despite not being a product. Behavior-preserving: swift build green; swift test 1052 pass / 0 fail. Co-authored-by: Cursor <cursoragent@cursor.com> * test(coordinator): behavior-lock the five routing-eligibility gate fns Characterization suite pinning the exact current decisions of providerPassesRoutingGatesLockedEx, providerCanRouteBuildLocked, providerHasWarmModelLocked, publiclyRoutableLocked, warmPoolCandidateReasonLocked (+ modelLoadCandidatePendingLocked) across a matrix of provider states, including the intentional differences (owner self-route relaxation, breaker-ignoring preflight bypass, warm-only loaded check, model-agnostic public check, granular warm-pool reason labels). Pre-refactor safety net for the routing-gate pipeline unification. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(provider): split ProviderLoop god-file by concern (3577->525 LOC) Split the 3,577-LOC ProviderLoop actor into focused same-module extension files mirroring the existing ProviderLoop+SSEParser/+ErrorMapping pattern: Serve, InferenceHandler, Preload, Prefetch, Testing, Trust, MemoryProtection, IdleTimeout, Capacity, AutoUpdate, ModelLoading, Cancellation, AttestationChallenge, LocalEndpoint. Method bodies are byte-identical; only access control changed. Stored state and methods reached across the split were widened private->internal (Swift private is file-scoped); purely-local members stay private. Behavior unchanged. Build green; swift test 1052/73 pass (unchanged from baseline). Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(coordinator): unify the routing-eligibility gate pipeline Five functions re-implemented overlapping subsets of the provider eligibility gates. Extract the two exactly-shared sub-pipelines into registry/routing_eligibility.go: - providerLivenessGateLocked: status/private-only/trust/runtime/ private-text/challenge-freshness, with minTrust+allowPrivate relaxation - providerServesRoutableModelLocked: catalog membership + dedicated-box rule Repoint providerPassesRoutingGatesLockedEx, providerCanRouteBuildLocked, providerHasWarmModelLocked, publiclyRoutableLocked, and modelLoadCandidatePendingLocked onto them — exact semantics preserved (verified by the characterization suite). warmPoolCandidateReasonLocked is kept separate (documented) because it emits granular per-gate reason labels that a boolean helper cannot preserve, and interleaves warm-pool-specific gates whose order determines the reported reason. Behavior-preserving: routing-gate characterization suite + full coordinator go test ./... stay green. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(provider): split BatchScheduler god-file by concern (2503->538 LOC) Split the BatchScheduler actor core into focused same-module extension files alongside the existing +EngineBridge/+KVEstimation/+Telemetry/+Liveness/ +KVQuantScheme: ModelLifecycle, EngineFactory, CheckpointRestore, Admission, Submit, PrefixCacheSizing, Testing. Method bodies are byte-identical; only access control changed. 13 members reached across the split were widened private->internal (incl. resolving a checkpointLayerSignatures static-vs-instance name shadow); purely-local members stay private. Behavior unchanged. Build green; swift test 1052/73 pass (unchanged from baseline). Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(coordinator): extract shared inference admission preflight handleChatCompletions and handleGenericInference carried byte-identical copies (~290 lines each) of the self-route / prefer / public capacity-and-TTFT preflight: QuickCapacityCheck -> alias-capacity fallback -> unservable shed -> model-too-large -> capacity 429 / queue-spill -> no-eligible-provider shed -> TTFT gate, with identical rejections and routing.decisions metrics. Extract runInferenceAdmission (api/inference_admission.go). The only divergence (verified by diffing the two blocks) is the forward-body refresh after an alias fallback rewrites parsed["model"]: chat re-marshals its threaded rawBody (re-lowering Responses input->chat, which can 400), generic rebuilds from parsed at dispatch. That single difference is threaded as the onModelFallback callback (nil for generic). Behavior-preserving: full api go test ./api/... (incl. failover, shed, servability, dedicated, cold-dispatch, route-outcome suites) stays green. consumer.go 4884 -> 4407 LOC. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(provider): split ModelCatalog god-file by concern (1202->171 LOC) Split ModelCatalog.swift into focused files: catalog data types stay in ModelCatalog.swift; ModelCatalogClient (coordinator catalog HTTP client), ModelDownloader core, and the ModelPrefetcher abstraction move to their own files. ModelDownloader's interleaved methods split into +Download (manifest/ legacy orchestration), +Prefetch (resume-aware prefetch), and +HTTP (file fetch/stream/hash/publish) extensions. Bodies byte-identical; ModelDownloader's 4 stored props + cross-file private methods and CatalogResponse widened private->internal. Behavior unchanged. Build green; swift test 1052/73 pass (unchanged from baseline). Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(coordinator): extract shared inference balance reservation handleChatCompletions and handleGenericInference held identical pre-flight balance reservation + per-key spend-cap blocks (reservationCost -> checkKeySpendCap -> reserveInitialBalance with the same insufficient_quota / insufficient_funds / DB-error rejections). Extract reserveInferenceBalance (api/inference_admission.go), returning (reservedMicroUSD, serviceReservation, handled). Behavior-preserving: full api go test ./api/... stays green. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(provider): split StartCommand god-file by concern (1133->121 LOC) Split the Start command struct into same-target extension files: core keeps the command declaration, options, and run() orchestrator; serving modes (standalone/foreground/scheduled), preflight + inline login, launchd daemon install, interactive catalog picker, and the raw-mode TUI picker move to StartCommand+Modes/+Preflight/+Daemon/+Picker/+TUIPicker. Bodies byte-identical; 7 private methods reached across the split widened to internal. Behavior unchanged. Build green; swift test 1052/73 pass (unchanged from baseline). Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(coordinator): segregate store.Store into domain sub-interfaces The ~150-method Store god-interface is split into 12 cohesive domain sub-interfaces (APIKeyStore, UsageStore, TelemetryStore, LedgerStore, BillingStore, ModelRegistryStore, ReleaseStore, UserStore, DeviceAuthStore, InviteStore, ProviderEarningsStore, ProviderStore) in store/interface_domains.go; Store now embeds all 12. Pure interface refactor: every method keeps its exact signature, both MemoryStore and PostgresStore are unchanged and still satisfy the composed Store (compile-time assertions), and a reflect-based method-set dump confirms the flattened Store exposes the identical 150 methods before and after. store go test ./store/... stays green (Postgres tests run only when DATABASE_URL is set). Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(provider): split CoordinatorClient god-file by concern (1173->213 LOC) Extract self-contained top-level types into their own files (CoordinatorClient Types: events/config/messages/errors; State: atomic stats + provider state + concurrency primitives; OutboundRouter; ReachabilityMonitor) and split the actor into +Connection (reconnect/session/receive + telemetry), +Inbound (frame->event dispatch), +Registration (register + heartbeat), and +Outbound (wire encoding). Bodies byte-identical; 21 members reached across the split widened private->internal. The file-private `Logger` typealias was renamed to a unique `CoordinatorWSLogger` (kept internal) so widening the actor's logger property doesn't shadow `Logging.Logger`/`os.Logger` elsewhere in the module. Build green; swift test 1052/73 pass (unchanged from baseline). Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(coordinator): split api/consumer.go into focused files Same-package modularization of the consumer god-file (no behavior change): - httputil.go: writeJSON + the OpenAI-compatible errorResponse envelope - responses_translate.go: Responses-API request lowering (input/tools/ tool_choice/text.format -> chat-completions shape) - models_endpoints.go: GET /v1/models and /v1/models/{id} + alias listing consumer.go 4884 -> 3794 LOC. Full api go test ./api/... stays green. Co-authored-by: Cursor <cursoragent@cursor.com> * test(provider): cover doctor command pure logic (CLI test seams) Add DoctorChecksTests for buildDoctorChecks (snapshot-driven hardware/config/ model-count checks + the deterministic check backbone), describeMDMEnrollment, and CheckStatus markers. Environment-independent: only asserts snapshot-driven behavior, so host Metal/SIP/codesign state can't flake the suite. swift test 1059/74 pass (baseline 1052/73 + 7 new; no production changes). Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(coordinator): extract APNs code-attest from api/provider.go Move the v0.6.0 APNs code-identity attestation flow (codeAttestMetric, codeAttestLoop, tryCrossVersionReuse, maybeRearmCodeAttest, sendCodeIdentityChallenge, handleCodeAttestationResponse) into api/provider_codeattest.go. Same-package move, no behavior change; provider.go 3563 -> 3150 LOC. Full api go test ./api/... stays green. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(coordinator): extract provider persistence from registry.go Move the registry<->store persistence bridge (SetStore, LoadStoredProviders, RestoreProviderState, providerRecordStats, PersistProvider[Throttled], persistReputation[Throttled], persistProviderNow) into registry/persistence.go. Same-package move, no behavior change; registry.go 4693 -> 4365 LOC. registry go test ./registry/... stays green. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(e2e): route via ReserveProviderEx after FindProvider removal Round-1 cleanup deleted registry.FindProvider but only re-ran go test inside coordinator/, missing the e2e package. The lone surviving caller in TestIntegration_SwiftProviderRealRoutingGates broke the e2e build (Blacksmith e2e/Build). Mirror the in-coordinator migration: add a local findRoutableProvider helper that probes routability through the production ReserveProviderEx path (same structural/privacy/trust/challenge/capacity gates), reserves then releases capacity, and returns the selected provider. No deleted methods or shims reintroduced. Verified: go build ./... and go vet ./... (compiles e2e/) green from repo root. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(provider): split KV-quant benchmark out of shipped darkbloom scripts/install.sh ships the `darkbloom` binary, but the `darkbloom` target depended on ProviderBenchmark, which carried the KV-quant gate/eval code AND the fatalError protocol-conformance KV-cache stubs (ProtocolSafeQuantizedKVCache, the V-only/BFloat16 caches). Those benchmark-only stubs were linking into every installed provider, defeating round-1's goal. Split ProviderBenchmark into two targets along the existing clean boundary (verified: the lightweight runners and the KVQuant/ dir do not reference each other): - ProviderBenchmark (lean): ModelBenchmark, ThroughputSweep(+report), DecodeBandwidthModel — the only benchmark code `darkbloom benchmark` needs. Still linked by `darkbloom`, so the command keeps working. - ProviderBenchmarkKVQuant (heavy, has the fatalError stubs): the KVQuant/ dir, linked ONLY by kv-quant-gate, kv-attn-selftest, and their tests — never by `darkbloom`. kv-quant-gate / kv-attn-selftest / the 3 KVQuant test files now import ProviderBenchmarkKVQuant; the darkbloom benchmark command and ThroughputSweepTests keep importing the lean ProviderBenchmark. Bundle-safe: release builds by product (`swift build -c release --product darkbloom`); product names, install paths, and LatestProviderVersion are unchanged. Verified: `swift build --build-tests` green (all targets + tests). nm proof: darkbloom links 0 ProviderBenchmarkKVQuant symbols (939 lean ProviderBenchmark symbols intact); kv-quant-gate links 3694. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(coordinator): tighten maybeFallbackAlias branching Behavior-preserving cleanup of the alias capacity/TTFT fallback helper. Both fallback modes already share a SINGLE Previous-build capacity probe (QuickCapacityCheckWithTTFTForRequest) — the desired build is probed once at the call site and the previous build once here, on different models, so there is no redundant capacity check to remove. This reduces the residual branching instead: - merge the two identical modelShed / !IsModelInCatalog early-return guards (both pure, both return the same tuple) into one - hoist the duplicated `mode == aliasFallbackTTFT` test into a single `enforceTTFT` bool and drop the `tooSlow` temp The failure-path model selection is unchanged (TTFT mode reports the probed Previous build the caller uses as the alternate TTFT estimate; capacity mode keeps the current build it discards), so all returns are byte-identical to before. Verified: coordinator/api alias fallback tests + full `go test ./...` green. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(provider): widen split-file symbols to internal (ProviderLoopError, OSAllocatedUnfairLock) — fixes #445 clean-build access errors The god-file split moved cross-file references that the warm incremental .build cache never recompiled, hiding two access violations: - ProviderLoopError (ProviderLoop.swift) is thrown from ProviderLoop+Serve.swift. The throw is inside `#if !DEBUG`, so `swift build` (debug) compiled it out and masked the error; a release build fails with "'ProviderLoopError' is inaccessible due to 'private' protection level". Widen private -> internal. - OSAllocatedUnfairLock shim (CoordinatorClientState.swift) is instantiated in OutboundRouter.swift. While `private` it silently resolved to Apple's os.OSAllocatedUnfairLock (OutboundRouter imports os) rather than the project shim. Widen private -> internal so OutboundRouter uses the intended shim (matching PongTracker/ManagedAtomic). Validated: `swift build -c release --product darkbloom` (whole-module, #if !DEBUG active) reaches "Build complete"; `swift test` -> 1059/1059 pass. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>
…#448) Three layout defects on the Network Stats page: 1. Invisible card surfaces — every card/chart/panel used bg-bg-primary, the same token as the page background, so surfaces were near-invisible in light mode and fully invisible in dark mode. Switched the 12 top-level surfaces to bg-bg-white, matching billing/models and the rest of the console. 2. Broken scroll container — the page root carried overflow-y-auto, nesting a second scroller inside AppShell's <main> and letting the TopBar scroll away. Reworked to the canonical pattern used by every other page: non-scrolling root (flex flex-col h-full) + pinned TopBar + a single flex-1 overflow-y-auto wrapper. 3. Whole-document scroll — an sr-only, absolutely-positioned toggle checkbox in ActiveModelsSection escaped the scroller's clip (its containing block fell through to <body>), stretching the document to ~3x the viewport. Scrolling while the pointer was over the (non-scrolling) sidebar dragged the entire app upward. Fixed by making the scroll wrapper position:relative so absolutely positioned descendants are contained and clipped. Class/token-only change; no logic touched. Verified: tsc error set unchanged vs master, eslint 0 errors on the file, next build succeeds, /stats prerenders.
…(behavior-neutral) (#447) * feat(coordinator): TTFT-contention Phase 0 — shadow + measurement (behavior-neutral) Implements the corrected Phase-0 slice: make the EXISTING per-(provider,model) occupancy drive an occupancy-aware TTFT estimate + a shadow gate at the verified ~10s base, and measure spread-to-idle — all OFF by default (env kill-switches), so prod behavior is byte-for-byte unchanged until shadow validates. Telemetry correctness (Fix 5/E): - actual_ttft_ms now derives from FirstContentAt (first delivered content) vs the committed attempt's DispatchedAt, not the held-preamble FirstChunkAt — kills the retried-request shared-Timing -378s rows. Negative clamps to 0 and emits routing.invalid_ttft{reason:negative}. Adds MarkFirstContentArrived/ FirstContentAtSafe (timingMu-guarded; race-tested). dispatch_to_first_chunk_ms stays the preamble diagnostic (also clamped). Removes the redundant FirstChunkAt-based TTFT override in handleComplete. - completion_tokens persists 0 on terminal cancel/error/timeout via a CompletionTokensSet flag (postgres CASE gated on $24 OR <>0; memory merge sticky) so 0-token cancels are no longer NULL/invisible. Both store impls covered. Occupancy-aware estimate (Fix 1/B): - ttftMsFromSnapshot gains an occupancy term reusing the existing max(pendingForModel, backend_running+backend_waiting) (new snapshotOccupancy helper, also dedups the cost path) and projectedPerRequestDecodeTPS. Gated by EIGENINFERENCE_TTFT_OCCUPANCY_ALPHA (default 0 => no term => byte-for-byte today). Shadow admission + spread (Fix A/shadow + deadline base): - EIGENINFERENCE_TTFT_DEADLINE_BASE_MS (default 10000) used ONLY by the shadow evaluator; live consumer.go ttftDeadline (5s) untouched (documented). - EIGENINFERENCE_TTFT_ADMISSION_MODE=off|shadow|enforce (default off). shadow/ enforce compute would_shed (estimate>~10s base) + would_redirect_to_idle (a loaded-idle peer for the same model existed) and emit routing.ttft_admission / routing.ttft_spread WITHOUT changing the routing decision. enforce currently behaves like shadow (reserved for a future enforce step). Config: - EIGENINFERENCE_QUALITY_CONCURRENCY_OVERCOMMIT kept at 2.0; documented the intended 2.0->1.5->1.0 staging (no behavior change by default). Tests: occupancy estimate monotonic + crosses the 10s knee (alpha>0) and neutral at alpha=0; shadow would_shed/would_redirect emitted while the decision is unchanged; actual_ttft_ms reflects FirstContentAt and clamps the -378s case + fires the metric; completion_tokens persists 0 on a cancelled row in both stores. go test ./... green, gofmt clean, go vet clean, race detector clean. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(ttft): address PR #447 review — content-stamp ordering, shadow gate accuracy, occupancy rate, alpha/HARD_REJECT decoupling Resolves the four Codex P2 functional bugs + the Centaur env-validation INFO on the Phase-0 TTFT shadow slice. All changes remain behavior-neutral by default (occupancy alpha=0, admission mode=off); only telemetry correctness changes are always-on. A. FirstContentAt stamped too late (Codex P2, dispatch.go / consumer.go / provider.go). actual_ttft_ms now derives solely from FirstContentAt, but the only stamp ran in writeCommittedResponse — AFTER waitFirstChunk's defer wrote successRoutingOutcome() and after handleComplete could build completeRouteOutcome() for a fast single-chunk completion, leaving TTFT 0/NULL. Fix: stamp at the content-commit site via a new commitFirstContent helper (all 8 dispatch variants), stamp the generic /v1/completions and /v1/messages first-content path, and add a universal fallback in handleComplete (stamp when the provider reported delivered tokens) so the cross-goroutine race can never persist a 0/NULL TTFT for a delivered reply. Regression test (api) drives the fast-completion race and fails without the fix. B. Idle shadow scan ignored selection gates (Codex P2, registry/ttft_shadow.go). loadedIdleAlternativeExistsLocked returned true on any loaded+idle peer, so a multimodal or oversized-token request recorded false would_redirect_to_idle. Fix: apply the SAME gates the scheduler does — vision capability (providerServesVisionModelLocked) + capacity/token-budget/hardware-fit feasibility (buildCandidateWithReason). Registry test: a vision request with only text-only idle peers ⇒ no idle alternative. C. Occupancy term used backendRunning for the rate (Codex P2, scheduler.go). In the herd case (pendingForModel > backendRunning) the per-request decode rate was projected at the idle gauge, UNDER-stating the term. Fix: new projectedPerRequestDecodeTPSAtBatch projects the rate at occ (reapply at the batch the request actually joins; unwind still uses the heartbeat's backendRunning the observation was paired with). Test asserts the term grows with the pending burst at equal heartbeat gauge. D. alpha is now a PURE shadow knob (scheduler.go / ttft_shadow.go). The occupancy term is removed from ttftMsFromSnapshot (the live cost / MaxTTFTMs ceiling / bestTTFT input) and lives only in the new occupancyAwareTTFTMsFromSnapshot used by the shadow evaluator. With prod's HARD_REJECT (MaxTTFTMs from the 5s ttftDeadline), raising alpha can no longer tighten the live ceiling and over-shed ~2x. Invariant test: alpha>0 + shadow leaves the live decision/TTFTMs identical to alpha=0 while the shadow still computes occupancy-aware would_shed. E. Env bounds validation (Centaur INFO, cmd/coordinator/main.go). Validate EIGENINFERENCE_TTFT_OCCUPANCY_ALPHA (negative→0; >1e6 / non-finite →warn+ default 0) and EIGENINFERENCE_TTFT_DEADLINE_BASE_MS (range [1000,120000]; else warn+default ~10s); warnings log the offending value. Table tests. go test ./... + go test -race ./registry/... ./api/... + gofmt + go vet green. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(ttft): idle shadow scan honors full selection eligibility (exclude/allowlist/prefer-owner) Second Codex review pass on the Phase-0 shadow slice found the idle-spread scan (loadedIdleAlternativeExistsLocked) still missed three selection-eligibility filters the real selector applies, each corrupting would_redirect_to_idle for a traffic class. The vision + capacity gates from 3ffa01b are confirmed good; these are additional gaps. Default behavior remains byte-for-byte (shadow mode off). Robust fix — the idle scan now mirrors selectBestCandidateScanLocked's per-provider eligibility in the same order, using the same helpers, with a documented correspondence block so it cannot drift from selection again: 1. excludeIDs (registry/scheduler.go ~285, registry/ttft_shadow.go): ReserveProviderEx now threads its retry/speculative-backup excludeIDs into evaluateTTFTShadowLocked → loadedIdleAlternativeExistsLocked, which skips excluded providers (the selector's excludeSet). Previously an excluded-but-idle provider falsely counted. 2. provider allowlist (registry/ttft_shadow.go): honor AllowedProviderSerials via providerMatchesAllowedSerial — an idle non-allowlisted peer no longer counts. 3. prefer-owner pool (registry/ttft_shadow.go): when prefer-owner selected an OWNED winner, the selector's pool was owned-only, so a public idle peer is not a real alternative (winnerOwnedPool && !owned skip), mirroring the owned-pool block. Also (cheap nit): final-status string constants (finalStatus*) in api/route_outcome.go replace bare "cancelled"/"error"/"timeout"/"success"/ "partial_success" comparisons and constructors. Tests (registry, all verified fail-without-fix): prefer-owner with only public idle peers ⇒ would_redirect_to_idle=false (and owned/plain contrasts); allowlist with only non-allowlisted idle peers ⇒ false; retry with the idle peer in excludeIDs ⇒ false. Existing vision/capacity/occupancy/HARD_REJECT tests still pass. go test ./... + go test -race ./registry/... ./api/... + gofmt + go vet green. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(ttft): scope first-content fallback to committed attempt; reuse selector for idle-spread scan; DRY/cleanliness Third Codex review pass on the Phase-0 shadow slice found two new P2s. Default behavior stays byte-for-byte (alpha=0, mode=off). P2 #1 (functional) — handleComplete first-content fallback could stamp an ABANDONED attempt's shared Timing (api/provider.go). Retries share one RequestTiming pointer (dispatch.go sets Timing: d.timing per attempt); an abandoned attempt completing late (provider finished after the coordinator timed it out and retried) could win RemovePending and, via the ungated fallback, stamp FirstContentAt first — and since it is first-write-wins, the real committed retry could not replace it, clamping/zeroing its actual_ttft_ms. Fix: add a per-attempt contentCommitted flag (registry.PendingRequest.MarkContentCommitted / ContentCommittedSafe) set by commitFirstContent (dispatch) and the generic first-content stamps (consumer.go); the fallback now only stamps when ContentCommittedSafe() — strictly the committed attempt — so an abandoned attempt can never corrupt the shared Timing. Regression test (TestAbandonedAttemptDoesNotStampCommittedTTFT) + the committed-attempt test updated; both verified to fail without the fix. P2 #2 (metric accuracy) — idle-spread scan missed the post-candidate POOL filters (registry/ttft_shadow.go + scheduler.go). The scan re-derived eligibility and kept missing filters (3rd round: AvoidVersion diverse-version pool + MinDecodeTPS quality floor). Fix robustly by REUSE, not piecemeal mirroring: extracted scanCandidatesLocked as the SINGLE SOURCE of routing eligibility (every per-provider gate AND the post-candidate narrowing: prefer-owner / AvoidVersion / MinDecodeTPS); selectBestCandidateScanLocked now builds the pool via it and only adds cost-ranking, and loadedIdleAlternativeExistsLocked derives idle alternatives from the same pool (any non-winner candidate that is loaded + zero-occupancy). The shadow scan can no longer drift from selection. Tests: TestLoadedIdleAlternativeHonorsAvoidVersionPool + ...MinDecodeTPS (fail without the fix). The earlier exclude/allowlist/prefer-owner/vision/capacity tests still pass through the unified pool. DRY/cleanliness: - projectedPerRequestDecodeTPS already delegates to projectedPerRequestDecodeTPSAtBatch (single implementation; from the prior commit) — no further change needed. - The selector-reuse refactor is the cleanliness win: one eligibility source, ~45 fewer lines of duplicated gate logic in the shadow scan, documented doc-comments on candidateScan / scanCandidatesLocked / the committed-attempt flag. go test ./... + go test -race ./registry/... ./api/... + gofmt + go vet green. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>
…#449) The provider/consumer geo resolver was hardwired to the free http://ip-api.com endpoint (45 req/min). Read EIGENINFERENCE_IPAPI_KEY (a secret, injected via KMS / Secret Manager) and, when present, switch to the unmetered https://pro.ip-api.com endpoint with &key=<key>, tagging the result Source as "ip-api-pro". When unset, behavior is byte-for-byte identical to today (graceful free-tier fallback), so dev without a key still works. Make the ip-api base URL and HTTP client injectable so URL construction and response parsing are unit-tested against httptest without hitting the live service. The key is never logged (debug logs record only a pro bool) or committed; tests use a fake key. Co-authored-by: Cursor <cursoragent@cursor.com>
…#450) * chore(console-ui): phase 0 — add bundle-size analysis tooling Reconstructs per-route First Load JS (raw + gzip) from the Next 16 / Turbopack build output, which prints no size columns. Baseline captured: shared 789.6 KB gz, chat route 924.4 KB gz. Used to track bundle deltas through the refactor and wired into a CI budget in phase 7. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(console-ui): phase 1 — shared primitives (format/json/constants/http/server) Create the shared foundation the rest of the refactor builds on and delete the first wave of duplication (proposal F4/F5/F6): - lib/format/{currency,number,time,text} + index barrel: one implementation of each USD/number/relative-time/serial formatter. api-keys/format.ts and providers/dashboard/format.ts now re-export from here (public API unchanged). - lib/json.ts: the asRecord/asString/asNumber/asBoolean/asStringArray/ compactObject coercers (stats-model-filter now consumes them). - lib/constants.ts: STORAGE_KEYS single source (api-keys/constants, useAuth, encryption, api.ts wired to it). - lib/http/proxy-client.ts: proxyHeaders/managementHeaders/apiError/jsonOrThrow (lib/api.ts consumes them; removes its inline copies). - lib/server/coordinator.ts: coordinatorUrl()/privyAuth/passthrough/ missingPrivyToken — all 27 /api/* routes resolve the coordinator URL here, so the "https://api.darkbloom.dev" literal lives in exactly one place. Also drops the client-honored x-coordinator-url header from the 6 Stripe/telemetry routes (no client sets it; matches the SSRF-prevention intent already asserted in the route tests). - lib/coordinator-url.ts: client-safe localStorage->env->default resolver (encryption.ts wired). - components/ui/Modal.tsx: promoted shared dialog shell (api-keys/Modal re-exports it). - Move lib/verification-mode.tsx -> components/providers/ (UI context out of lib). Tests: +__tests__/{json,format}.test.ts. Invariant held: build green, eslint 0 errors / 137 warnings, vitest 335 pass (314 + 21 new) / 4 pre-existing fail. Co-authored-by: Cursor <cursoragent@cursor.com> * perf(console-ui): lazy-load Privy SDK + decouple AppShell ready gate (F1/F2/F1b/F14) The Privy auth SDK (~409 KB gz) was statically imported in the root layout and shipped on all 13 content routes — 52% of the shared baseline. Now: - PrivyClientProvider loads the SDK via next/dynamic(ssr:false) as an on-demand chunk after first paint; a synchronous AuthContext renders children immediately and reconciles when Privy resolves (new PrivyRealProvider module). - AppShell no longer blocks all content on Privy `ready` — the shell paints immediately, so Privy is off the LCP critical path (F2). - embeddedWallets createOnLogin:"off" (nothing uses Solana wallets) + email-only login sheds the @solana/viem code paths (F1b). - next.config: experimental.optimizePackageImports:["lucide-react"] (F14). Measured (scripts/analyze-bundle.mjs): shared baseline 789.6 -> 216.2 KB gz (-73%); chat route 924.4 -> 360.0 KB gz (-61%); every route ~573 KB lighter. Invariant held: build green, eslint 0/137, vitest 335 pass / 4 pre-existing fail. Co-authored-by: Cursor <cursoragent@cursor.com> * perf(console-ui): cache headers + visibility-gated polling + LCP font preload (F5a/F6/F12) - useVisiblePolling hook: runs on mount, polls only while the tab is visible, pauses when hidden, refetches once on regain. Applied to the fleet dashboard (15s) and the stats page (cadence 10s -> 15s) — background request volume on a hidden tab drops to ~0 (F6). - Cache-Control (public, s-maxage + stale-while-revalidate) on read-only proxy routes so the edge serves repeats: pricing 300/600s, models (public branch) 30/120s, models/capacity + leaderboard + network/totals + stats 10/30s (F5a). - Preload the Louize hero font (the LCP heading on chat empty state + /login) to cut LCP text delay and font-swap CLS (F12; woff2 conversion deferred — needs offline font tooling + a commercial-license review). Tests: +__tests__/use-visible-polling.test.ts. Invariant held: build green, eslint 0/137, vitest 338 pass / 4 pre-existing fail. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(console-ui): phase 2 — dedup Stripe payouts into components/payouts/ (F3) The entire Stripe Connect payouts feature was duplicated across billing and earnings (~340 byte-identical lines; the country dropdown was even duplicated twice within earnings). Extracted a shared module: - PayoutModal, MethodOption, StripeWithdrawModal, WithdrawalsList — verbatim shared leaves. - CountryPicker — the searchable dropdown (now also replaces billing's plain <select>, unifying the cosmetic fork the proposal flagged). - StripePayoutsCard — one parameterized card (title/icon/className/noun + balance slot) for both pages. - useStripePayouts — the shared status/onboard/withdraw state machine, with page-supplied post-withdraw reload + optional analytics. BillingContent 922 -> ~470 lines, EarningsContent 850 -> ~340 lines. Earnings poll is now visibility-gated and uses the central clientCoordinatorUrl resolver. Tests: +__tests__/payouts.test.tsx (hook flows + card states). Invariant held: build green, eslint 0 errors / 130 warnings (down from 137), vitest 344 pass / 4 pre-existing fail. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(console-ui): phase 3 — split lib/api.ts + decompose streamChat (F1/F8/F9) The 851-line god-client and its 354-line streamChat (ESLint cognitive complexity 119) are broken up behind a barrel that preserves the @/lib/api import path (zero caller churn): - lib/api/{types,models,pricing,billing,keys,providers,health}.ts + index.ts over the shared lib/http/proxy-client. lib/api.ts deleted. - lib/chat/{think-parser,sse,stream}.ts: streamChat is now a thin orchestrator over a ThinkStreamParser (the multi-format think FSM), an SSE payload iterator, and small transport/error/metrics helpers. - F9: ChatMessage now imports parseThinkFromContent from the shared think-parser instead of carrying a second copy of the grammar. - F8: useAuth's provisioning sequence is one `provisionApiKey` callback used by both the mount effect and the key-expired handler (kills the promise-nesting warnings). Tests: +__tests__/think-parser.test.ts. eslint warnings 130 -> 116 (streamChat cog-119 + handleContentToken cog-31 + useAuth promise nesting all gone). Invariant held: build green, vitest 352 pass / 4 pre-existing fail. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(console-ui): phase 5e — chat module + streaming perf (F3/F7/F10-any) Split components/ChatMessage.tsx into a components/chat/ module (UserMessage, ChatMessage thin orchestrator, MessageBody, ThinkingBlock, StreamMetrics, CodeBlock, typed markdown.tsx) and fix the streaming re-render storm: - React.memo(ChatMessage) + an id-keyed, referentially-stable onRetry so only the streaming message re-renders, not the whole list (perf F3). - Render plain text while streaming and parse markdown once on completion — removes the O(n^2) re-tokenization of the growing reply (perf F7). - New hooks/useChatStream.ts unifies the near-duplicate send + retry orchestration and batches per-token store writes with requestAnimationFrame (~one store update per frame instead of per token) (perf F3c). app/page.tsx is now a thin orchestrator. - Narrow Zustand selectors in ChatInput / TopBar / Sidebar so they no longer re-render on every streamed token (perf F3). - markdown.tsx is typed against react-markdown's Components (drops the 4 `any`, F10). Invariant held: build green, eslint 0 errors / 113 warnings (down from 116), vitest 352 pass / 4 pre-existing fail. Chat route 360 KB gz (runtime win; the eager react-markdown/pkijs trim lands with verification phase F4). Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(console-ui): phase 5d — verification module + lazy cert-verify (F4/F7/F5) Split the 653-line VerificationPanel into a components/verification/ module (thin VerificationPanel orchestrator + useDeviceVerification hook + NormalMode / TechnicalMode bodies + StatusLine/VerifyStepLine + types) and lazy-load the heavy X.509 verifier: - pkijs + asn1js (~76 KB gz) are no longer statically imported. The verifier is dynamically imported inside the verify action (useDeviceVerification, and the same minimal change in providers/dashboard/AttestationPanel); the types stay as erased type-only imports (perf F4). - VerificationPanel + E2ELockIndicator now use the shared maskSerial (F5 — kills two of the four forked copies; identical output for real <=12-char serials). Measured: chat route 360 -> 275 KB gz, providers route 323 -> 238 KB gz (both -85 KB). Invariant held: build green, eslint 0 errors / 112 warnings, vitest 352 pass / 4 pre-existing fail. (stats still imports cert-verify — fixed in the stats phase.) Co-authored-by: Cursor <cursoragent@cursor.com> * chore(console-ui): phase 7 (part 1) — CI bundle-size budget (F-AUTO) Turbopack prints no bundle sizes, so growth was invisible. Add npm scripts that run the phase-0 analyzer in budget mode and fail on a breach: - bundle:check — enforce budgets against an existing .next build - bundle:budget — build then enforce Budgets (gzip): shared <= 450 KB, any route <= 650 KB (current: shared 216 KB, heaviest route 333 KB). The ESLint max-lines guardrail lands in the final pass, calibrated to the post-split file sizes. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(console-ui)!: route direct browser->coordinator fetches through /api/* (F9/F9b) ISOLATED, NON-BEHAVIOR-PRESERVING: this commit moves the remaining direct browser->coordinator fetches behind same-origin proxies, so they no longer do a cross-origin preflight and no longer ignore NEXT_PUBLIC_COORDINATOR_URL (the hardcoded "https://api.darkbloom.dev" in client code made dev builds talk to prod). - New /api/me/earnings proxy; EarningsContent polls it (F9). - New /api/device/approve proxy; the /link DeviceLinkForm posts it. - VerificationPanel's verify now fetches the existing /api/attestation proxy; the technical-mode "download cert chain" link uses clientCoordinatorUrl. - /api/attestation gains ?summary=1 returning a slim { count, last_verified }; PreSendTrustBanner uses it instead of downloading the full cert blob (100 KB–1 MB+) just to show a provider count (F9b). Tests: +__tests__/proxy-routes.test.ts. Invariant held: build green, eslint 0 errors / 112 warnings, vitest 358 pass / 4 pre-existing fail. (The stats page's direct-coordinator *fallbacks* are addressed in the stats phase.) Co-authored-by: Cursor <cursoragent@cursor.com> * perf(console-ui): phase 5f (part 1) — stats lazy cert-verify + memoized aggregation (F4/F10) - Lazy-load the X.509 verifier in the stats page (type-only import + dynamic import inside the verify handler), removing pkijs/asn1js (~76 KB gz) from the stats First Load: stats route 333 -> 248 KB gz. - Memoize the per-poll aggregation (buildModelInventory + activeNetworkPowerWatts) above the early returns so it no longer recomputes on tab switches / unrelated re-renders (F10). All three heavy routes are now near the shared baseline (chat 275, stats 248, providers 238 KB gz, vs ~900 each at baseline). Invariant held: build green, eslint 0/112, vitest 358 pass / 4 pre-existing fail. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(console-ui): phase 5a — api-console feature module (F7/F6/F8) Split the 543-line api-console page (~310 lines of static data) into a module: - content.ts: the ENDPOINTS reference + Endpoint/CodeSnippet types + the SDK snippet builders (sdkSetupExamples/chatExamples/modelsExamples). Pure data. - EndpointRow.tsx: the one interactive island (expand + analytics). - page.tsx: thin orchestrator (543 -> ~145 lines) wired to STORAGE_KEYS + clientCoordinatorUrl (drops its private storage-key/coordinator-URL constants, F6). Extracting the static content sets up an RSC conversion later (F8). Invariant held: build green, eslint 0 errors / 112 warnings, vitest 358 pass / 4 pre-existing fail. Co-authored-by: Cursor <cursoragent@cursor.com> * chore(console-ui): phase 7 (part 2) — ESLint max-lines guardrail Add max-lines (warn, 500 code lines) so a new 3000-line god-file can't land silently. Calibrated to flag only the genuinely oversized files that remain — stats/page.tsx (3076), earn/page.tsx (708), api/stats/route.ts (526) — which documents the remaining split work without churning the rest of the tree. eslint: 0 errors / 115 warnings (was 137 at baseline). Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(console-ui): phase 5c — earn feature module (F7) Split the 780-line earn page (calc.ts already held the model). New module: - useEarningsCalculator.ts: all hardware/model selection state + the useMemo derivations + handlers (the orchestration that was tangled into the page). - SetupProviderCTA / HardwareSelector / ModelSelector / EarningsResults / PillButton: dumb presentation over the hook result. - page.tsx: thin orchestrator (780 -> ~70 lines). Clears the earn/page max-lines guardrail flag. Invariant held: build green, eslint 0 errors / 114 warnings, vitest 358 pass / 4 pre-existing fail (earn-page.test still green). Co-authored-by: Cursor <cursoragent@cursor.com> * fix(console-ui): address Codex review — gate sign-in CTAs on Privy readiness Codex (P2) flagged that, with Privy now lazy-loaded and AppShell no longer blocking on `ready`, children render against SSR_AUTH whose `login` is a no-op until the SDK chunk loads. The chat-input and provider-earnings sign-in CTAs called `login`/`onLogin` without gating on `ready`, so a click during the brief lazy-load window did nothing. Gate both on `ready` (disabled + "Loading…") to match the already-gated chat hero and earn CTA. ChatInput gains a `ready` prop (default true), threaded from the chat page; EarningsContent reads `ready` from useAuth. Regression test: __tests__/login-gating.test.tsx — the CTA is disabled and does not invoke onLogin while !ready, and enabled + forwards the click once ready. Co-authored-by: Cursor <cursoragent@cursor.com> * chore(console-ui): drop unused imports flagged by code-quality bot Remove three unused named imports the full-tree CodeQL/code-quality review flagged as errors in new files: - scripts/analyze-bundle.mjs — drop `statSync` - __tests__/payouts.test.tsx — drop `waitFor` - __tests__/think-parser.test.ts — drop `vi` No behavior change. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(console-ui): gate ApiKeysManager sign-in on Privy readiness (completes Codex #1) Re-review follow-up to the Codex #1 dead-click finding. Of the 3 flagged sibling CTAs, only one was genuinely ungated: - ApiKeysManager — FIXED: its "Sign in to manage API keys" button called the no-op `login` with no `ready` gate while the lazy Privy SDK loads. Thread `ready` through useApiKeys (from useAuth) and gate the button (disabled + "Loading…"), matching the chat-input / earnings / chat-hero / earn / login pattern. The other two were already gated by upstream `!ready` early-returns, so no change (would be dead/misleading code): - providers/dashboard SignInGate is only rendered by ProviderDashboard after `if (!ready) return <LoadingState/>` (ProviderDashboard.tsx:60), before the SignInGate at :61. - link/DeviceLinkForm already returns a full-page Loading via `if (!ready)` (DeviceLinkForm.tsx:87) before its sign-in button at :135 — it does use `ready`. Regression tests: extend __tests__/login-gating.test.tsx with ApiKeysManager cases (disabled + no-op while !ready; enabled + forwards click once ready). Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>
…e_health telemetry (measurement only) (#451) * feat(provider): instrument the first-token wedge (measurement only) Add NON-PRIVATE engine-health signals so the coordinator and the offline telemetry trail can SEE the MLX/Metal first-token wedge from docs/reports/2026-06-22-cancel-root-cause-and-fix.md §C: the provider emits the CPU-only preamble, then the first blocking eval never returns, so token production freezes while num_running stays 0 and the wedge is invisible today. Provider (Swift): - WedgeMonitor: pure per-load counters (admits, first-tokens, engine steps) and the ">=N admits / 0 first-tokens / >=T seconds" wedge primitive. - Fold the signals into BackendSlotCapacity: steps_executed, admits, first_tokens_emitted, seconds_since_last_step, seconds_since_last_first_token, wedge_suspected (all omit-empty; admit at bridge start, first-token on first content -- independent of admittedAt, which a prefill wedge never sets). - Emit engine_health telemetry events (model-load milestones, periodic engine snapshots, wedge-suspected transitions) via the existing TelemetryClient. Coordinator (Go): - Mirror the 6 BackendSlotCapacity fields and decode them into routingSnapshot. - New engine_health telemetry kind; allowlist the new operational fields. - Emit provider.first_token_wedge_suspected from heartbeats (fleet visibility). Console UI (TS): mirror the engine_health kind + allowlist. INSTRUMENTATION ONLY: no routing/watchdog behavior change. Protocol + telemetry symmetry tests added/updated across Go/Swift/TS; WedgeMonitor logic + wire round-trip/omit/backcompat tests included. telemetry_events root cause: the coordinator no longer persists telemetry to the DB (Datadog is the sole sink -- api/telemetry_handlers.go), so the dead table is by design; provider internals were invisible because the provider never emitted engine-internal events. Fixed by emitting engine_health events. Co-authored-by: Cursor <cursoragent@cursor.com> * feat: add eval-in-flight, idle-clear, and prefill-health wedge signals (measurement only) Extends the first-token-wedge instrumentation with three more NON-PRIVATE signals and bumps the mlx-swift / mlx-swift-lm submodule pins to the commits that carry the engine-side probes. #1 Prefill-sampling health (provider only): counts why observed_prefill_tps stays 0 — accepted / dropped-below-1ms-floor / dropped-above-8000-ceiling samples + the raw last sample rate. The bounds check is refactored into a pure, unit-tested `classifyPrefillSample` (accept path byte-for-byte unchanged). #2 Eval-in-flight (mlx-swift EvalProbe, pin ac678221->f84a56d8): brackets the blocking mlx_eval inside evalLock so the provider reports how long the CURRENT eval has run — the direct wedge smoking gun. Heartbeat scalar eval_in_flight_ms + engine_health counts; coordinator emits provider.eval_in_flight_long. #3 Idle-clear-in-flight (mlx-swift-lm EngineCore, pin 8a9bc7ce->3293f8ed): marks the M4-panic-race synchronize+clearCache so a clear that never exits pins candidate #1. Heartbeat scalar idle_clear_in_flight_ms + engine_health count. Wire: 2 new BackendSlotCapacity scalars mirrored Swift<->Go + decoded into the routing snapshot; 10 new engine_health field keys mirrored across the Go/Swift/TS allowlists. Symmetry/round-trip/omitempty + classifier + EvalProbe-wiring tests added. No routing/watchdog/eval behavior change (cheap timestamps; writes under the eval lock or engine loop, lock-free reads — same pattern as stepsExecuted). SUBMODULE NOTE: the two pins point to pushed feature branches (Layr-Labs/mlx-swift#provider-wedge-instrumentation, Layr-Labs/mlx-swift-lm#provider-wedge-instrumentation). Those submodule commits must be merged (and ideally the pins re-pointed to the merge commits) for CI to build a stable pin. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: address 4 Codex-flagged false-positives in wedge/prefill instrumentation All measurement-only / behavior-neutral; each fix has a regression test. 1. wedgeSuspected now requires the engine-step flatline too (WedgeMonitor.swift): was admits>=N && dryStreak>=T, which tripped on 3 legitimately slow prefills whose stepsExecuted kept advancing. Add `secondsSinceLastStep >= T` (the file's own documented 3-part signature); never-sampled reads 0 ⇒ won't trip. Test: notSuspectedWhenStepsAdvancing. 2. Clear the dry streak on a no-first-token terminal (WedgeMonitor + runBridge): only the first-token path cleared it, so client_gone cancels (which end before the first token) accumulated into a false wedge on healthy boxes. New WedgeMonitor.recordTerminalWithoutFirstToken() decrements the currently-hanging streak (floor 0, clears anchor at 0), called from both terminal exits when !sawFirstToken. Test: terminalWithoutFirstTokenClearsStreak. 3. eval_in_flight_long is provider-wide, not per-model (provider_wedge_telemetry.go): eval_in_flight_ms is the process-global EvalProbe value copied onto every slot, so the per-slot loop fired once per loaded model and tagged idle models as stalled. Emit once per heartbeat from the max across slots, untagged by model (keep first_token_wedge_suspected per-model). Test: TestProviderWideEvalInFlightLong (3×800ms ⇒ not long; max-based, not summed). 4. Reset prefillHealth on model swap (BatchScheduler.stopCurrentEngine): the EWMA it tracks IS reset there, but prefillHealth was not, so a new model inherited the prior model's accepted/dropped/last-sample counts. Add the reset next to the EWMA reset and correct the false "not reset per load" comment. Test: modelSwapResetsPrefillHealth. No protocol/wire change; no submodule change; coordinator metric semantics: provider.eval_in_flight_long is now provider-wide (see PR body). Co-authored-by: Cursor <cursoragent@cursor.com> * test: assert provider.eval_in_flight_long emits once provider-wide Direct end-to-end regression for Fix 3 via the UDP statsd collector: 3 loaded models all carrying the same process-global in-flight eval must emit provider.eval_in_flight_long exactly once and without a model: tag. Fails (n==3) without the provider-wide fix. Co-authored-by: Cursor <cursoragent@cursor.com> * chore(submodule): bump mlx-swift pin to EvalProbe review fixes (c133e2e) Re-points libs/mlx-swift f84a56d8 → c133e2ed (Layr-Labs/mlx-swift#6): - EvalProbe now brackets MLXArray.eval() so implicit realizations (item()/asArray()/asData()) are tracked too. - EvalProbe state moved behind OSAllocatedUnfairLock (was a torn-read data race on nonisolated(unsafe) scalars). Measurement-only; provider `swift test` (1077/77) and `go test ./...` green against the new pin. Merge-ordering still holds: merge mlx-swift#6 first, then re-point this pin at the resulting main commit. Co-authored-by: Cursor <cursoragent@cursor.com> * chore(submodule): re-point mlx-swift-lm pin to main (#48 merged) Co-authored-by: Cursor <cursoragent@cursor.com> * chore(submodule): bump mlx-swift pin to NSLock EvalProbe (3c50ad6) Re-points libs/mlx-swift c133e2ed → 3c50ad69 (Layr-Labs/mlx-swift#6): EvalProbe now uses a cross-platform Foundation NSLock instead of the Darwin-only OSAllocatedUnfairLock, so mlx-swift builds on Linux (EvalProbe.swift is not in the Linux excludes). Behavior-neutral on Darwin. Provider `swift test` (1079/77) + coordinator `go test ./...` green against the new pin. Merge-ordering still holds: merge mlx-swift#6 first, then re-point this pin at the resulting main commit. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(wedge): anchor dry-streak to oldest hanging admit + epoch-gate callbacks Two Codex P2 fixes on the wedge monitor (measurement-only, behavior-neutral): B1 — dry-streak anchor could point at an ended admit (WedgeMonitor). After a no-first-token terminal decremented a non-last hanging admit, the streak anchor could still reference a request that already ended, so a mix of old+new admits reported wedge_suspected before the newer admits had actually stalled 10s. Replace the single anchor with a FIFO of hanging-admit timestamps: append on admit, remove the front on a no-first-token terminal, clear all on a first token (engine proven alive). dryStreakSeconds now measures from the OLDEST still- hanging admit. consecutiveAdmitsWithoutFirstToken is derived from the FIFO. Regression test: dryStreakAnchorsToOldestStillHangingAdmit (codex A/B/C/A-cancel/D scenario ⇒ NOT wedge_suspected at t=10). B3 — bridge callbacks didn't check the load generation, so a stale bridge callback after stopCurrentEngine/reload could corrupt the fresh model's monitor. Capture generationEpoch at bridge start and thread it through recordWedgeAdmit/FirstToken/TerminalWithoutFirstToken; ignore the event when the epoch no longer matches (mirrors engineStillCurrent). Regression test: staleEpochWedgeCallbackIgnoredAfterReset. B2 — EVALUATED, intentionally KEPT the bridge-start admit. Verified in code that this engine emits a RequestOutput ONLY from the decode phase (Scheduler.swift processGenResponses, lines 1262/1301) — there is no prefill/admission-stage output — so a wedged first eval produces NO RequestOutput; moving the admit to "first RequestOutput" would make the wedge invisible. The step-flatline gate (wedgeSuspected requires frozen steps) prevents a merely-queued request on a healthy engine from false-tripping. Documented the reasoning in the runBridge admit comment. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>
…e latency) v0.6.25 migrated from URLSessionWebSocketTask to NWConnection but did not set TCP_NODELAY. NWConnection defaults to Nagle's algorithm ON, which batches small WebSocket frames (~200 bytes) and delays sending until the previous segment is ACKed or enough data fills an MSS (~1460 bytes) — adding 40-200ms latency per chunk. URLSessionWebSocketTask sets TCP_NODELAY internally, so the old path did not have this problem. Production DB confirms the regression: - Per-chunk overhead: 61.6ms (0.6.24) → 185.6ms (0.6.25) = 3x worse - TTFT: 1,342ms → 13,357ms = 10x worse - Solo TPS: 48.1 → 28.0 = -42% Fix: set NWProtocolTCP.Options.noDelay = true on the transport protocol before connection setup. This is the NWConnection equivalent of setsockopt(TCP_NODELAY).
…rnel send) (#480) The cooperative thread pool scheduling gap between AsyncStream.yield() and the outbound for-await loop is the dominant per-chunk overhead (~30-40ms). MLX inference saturates the pool, starving the outbound consumer Task. This adds ~58ms per chunk × 115 chunks = 6.7s of unnecessary delay, reducing 130 TPS local to ~50 TPS coordinator-measured. Fix: inference chunks now bypass the OutboundRouter → AsyncStream → for-await path entirely and write directly to the NWConnection from the inference task thread via a dedicated serial DispatchQueue. Control messages (heartbeats, attestation, load status) stay on the existing AsyncStream path. Three optimizations in one: 1. Direct send (ChunkSender): nonisolated, Sendable facade that encodes the outbound message and enqueues it on the ChunkBatcher serial queue. No actor hop, no cooperative pool scheduling. 2. Batch coalescing (ChunkBatcher): accumulates sends on a serial DispatchQueue, flushes on next dispatch turn or at ~1400-byte MSS threshold. Under concurrent load (B=4), multiple chunks from different requests coalesce into a single TCP write via NWConnection.batch{}. 3. Pre-allocated send contexts (ChunkFrameWriter): reuses a single NWProtocolWebSocket.Metadata and ContentContext across all chunk sends, eliminating 5 heap allocations per chunk (575 per request). Ordering: SendHandle.send() calls chunkSender.flush() (synchronous queue.sync drain) before routing any terminal message (inferenceComplete, inferenceError). All terminal paths go through this barrier. Reconnect: ChunkBatcher.bind/unbind is identity-guarded — an old session teardown cannot clobber a freshly reconnected writer. Pending chunks during a reconnect window are dropped (requests already cancelled via cancelAllInflight). Measured: 370x faster than AsyncStream path under cooperative pool saturation (2.25ms vs 832ms for 500 frames). Expected: coordinator-observed TPS ~50 → ~120-130 (limited only by one-way network latency on the final chunk). Zero coordinator changes. Same WS protocol. OutboundRouter untouched.
Relax the ChunkSenderTests throughput assertion from >=10x to >=1.5x. On real Apple Silicon with MLX saturating the cooperative pool, the direct path is 100-370x faster. On CI runners (limited cores, no MLX load) the gap is ~2-5x because the pool is not truly starved. The 10x assertion is correct for production but flaky on CI. Bump provider 0.6.26 → 0.6.27, coordinator LatestProviderVersion.
Eliminate the cooperative pool round-trip from the decode loop and lower
streamInterval for better user latency.
Engine loop fix (mlx-swift-lm submodule):
The engine loop bounced through the cooperative thread pool on every
decode step: Task { engineLoop() } → withCheckedContinuation →
engineQueue.async → continuation.resume(). Each step paid ~3-8ms of
scheduling gap while the GPU sat idle, halving effective decode TPS.
Fix: pure GCD self-rescheduling on engineQueue. engineStep() dispatches
the next step via engineQueue.async (~1-5μs). No cooperative pool
involvement. GPU utilization goes from ~50% to ~100%.
streamInterval 8 → 4:
With the engine loop fix eliminating scheduling gaps, the per-chunk
transport overhead is negligible. Lower streamInterval gives lower
latency to the user — each token visible ~31ms sooner. At 130 TPS,
a 4-token burst every ~31ms is smoother than 60fps.
Submodule: libs/mlx-swift-lm c945bfe → e5d0a94
Provider: 0.6.27 → 0.6.28
X25519 DH scalar multiplication (~150us) was computed on every chunk, producing the same shared key each time (same keypair within a request). Fix: precompute once per request via sodium.box.beforenm(), then use symmetric-only XSalsa20-Poly1305 per chunk (~1-2us). Drops per-chunk provider-side encryption from ~150us to ~2us. Coordinator-side precompute, TPS measurement fix, single-pass JSON unmarshal, binary WS frames, and normalizeSSEChunk elimination are designed and tested but require a coordinated deploy — follow-up PR.
…64→78 TPS, +23%) (#482) * perf(provider): gemma4 MoE decode optimizations — compiled decode + fused gate+up + bf16 Pin mlx-swift-lm to c0065d2 which includes three MoE decode optimizations: 1. Compiled decode for B=1: wraps model forward in compile(), fusing ~970 Metal kernel dispatches into a compiled graph. Uses new CompilableRotatingKVCache for Gemma4 mixed sliding/full cache. Per-layer promotion handles heterogeneous caches (first to do this — even vmlx skips Gemma4 compiled decode). Opt-in: DARKBLOOM_COMPILED_DECODE=1. 2. Fused gate+up gatherQuantizedMM: lazy-concat gate+up expert weights into single tensor, one dispatch instead of two during decode. Default limit 1024 MB (Gemma4 8-bit experts are ~540 MB per pair). 3. bf16 weight conversion: converts fp16 params to bf16 at load to eliminate AsType cascades. No-op for models already shipping bf16. Measured on gemma-4-26b-a4b-it-8bit, M4 Max 128GB: Baseline B=1: 64 tok/s All opts B=1: 78.5 tok/s (+23%) Submodule: libs/mlx-swift-lm e5d0a94 -> c0065d2 * perf(provider): re-pin mlx-swift-lm -> 5d0a084 (P1/P2 compiled-decode fixes) Advances libs/mlx-swift-lm from 29602e0 to 5d0a084 (merged via Layr-Labs/mlx-swift-lm#60), which adds the two Codex P1/P2 fixes from the latest review: - P2: fused MoE learned-bias broadcast (expandedDimensions axis -2) - P1: correct compiled prefix-capture snapshot (no stale rotating-cache state) NOTE: supersedes the earlier commit message that referenced c0065d2; the correct, current pin is 5d0a084 (c0065d2 is an OLD pre-fix commit — do not revert to it).
…k encryption (#487) Bumps ProviderCore.version and coordinator LatestProviderVersion 0.6.28 -> 0.6.29. Release contents (already on master since v0.6.28): - #482 Gemma4 MoE compiled decode + fused gate+up + bf16 (pin mlx-swift-lm 5d0a084), incl. P1/P2 compiled-decode correctness fixes. - #483 precompute DH shared key for chunk encryption (~150us -> ~2us per chunk).
… to 0.6.28 engine) (#489) Reverts libs/mlx-swift-lm 5d0a084 (compiled decode + bf16 conversion, shipped in v0.6.29) -> e5d0a94 (v0.6.28's engine). v0.6.29's compiled-decode cold start (26GB model load + bf16 conversion + compile() graph trace) blew the TTFT deadline for Gemma under the 00:00 UTC surge; the fleet-wide rollover cold-started into peak and Gemma dropped to ~2% success (503 / 504 first_chunk_timeout). This restores 0.6.28's known-good decode path. v0.6.30 = 0.6.28 engine + version bump. Kept #483 (DH-key chunk-encryption precompute) — unrelated to the breakage, proven safe since 0.6.29. Bumps ProviderCore.version + coordinator LatestProviderVersion 0.6.29 -> 0.6.30. Re-warm/pre-compile fix tracked in DAR-378 before re-attempting compiled decode.
…491) * perf(network): streaming hot-path optimizations + dead-code cleanup Coordinator relay path (per-token costs): - Precompute the X25519 shared key once per request (chunk_key_cache) so chunk decryption is a symmetric open (~2us) instead of a fresh scalar multiplication (~40-60us) on the provider's single read goroutine. - Single-pass provider message decode: lightweight top-level type scan replaces the envelope json.Unmarshal pass (-45% decode time, -6 allocs per frame; falls back to the full envelope decode on anything ambiguous). - ChunkCh overflow now fails the request (cancel + 499, no reputation hit) instead of silently dropping tokens from a stream that is still billed. Provider WebSocket writer: - Priority control lane: attestation challenges and cancels preempt multi-MiB inference frames, so transport congestion can no longer turn into attestation-timeout reputation events or delayed cancels. - Per-frame timeout goroutine+timer replaced by one watchdog per connection with an atomic write deadline. Swift provider hot path: - Cached JSONEncoder + Data-only chunk encode path (removes the Data->String->Data round trip per token). Wire format unchanged. - Shared hex-encoding helper replaces 7 hand-rolled copies; shared ws->http URL normalization. Dead code / junk removal: - payments.Ledger write path (Deposit/PendingPayouts/SettlePayout), unused ReferralService.ledger, RequestQueue.CleanStale, dead Server setters, store sha256Hex dupe. - X25519ChaChaPoly.swift (prod crypto is NaCl box), localFirst.ts, unused currency exports, mock geo fixtures out of the prod stats route. - Stale scripts (calibrate-routing, load-test, gptoss_soak, mlx_lm_batch_bench), one-off e2e/pr311_compat_test.go. - Docs drift: AGENTS.md (bundle scripts, runbook paths, binary claim), threat-model.yaml coordinator/internal/* paths, payments/kv-cache doc paths. Verification: coordinator gofmt/vet clean, full go test green; provider swift build + protocol/crypto/coordinator-client/chunk-sender suites green; console-ui eslint/build green (4 pre-existing dashboard test failures unrelated). * fix(review): PR #491 review findings — overflow grace window, lane ordering contract, hex golden tests Addresses the two medium findings + cheap follow-ups from the multi-lens review (approve-with-nits): - api/provider.go: chunk-buffer overflow now grants a bounded 250ms grace window (sendChunkWithGrace) before failing the request, so a healthy consumer catching up after a TCP burst is not hard-killed at 256 chunks of lag; recover-guarded against a concurrent registry.Disconnect channel close (same idiom Disconnect itself uses). New test pins the grace-path delivery; overflow test's nil-deref Fatalf split. - registry/provider_writer.go: documented the real ordering contract — FIFO within a lane only, non-preemptive priority, WriteText's synchronous-completion invariant (what keeps request->cancel ordering safe), accurate lane-user lists (load/prefetch/desired-models remain on the data lane; rerouting is a follow-up). New ordering test pins WriteText-then-EnqueueText delivery order. Shared checkAccept preamble. - registry/queue.go: stale '(default 30s)' maxWait doc -> 120s. - store/memory.go: header no longer claims keys are stored unhashed. - provider-swift Util/Hex.swift: hex digit table hoisted to file scope (was rebuilt per byte); header no longer overclaims; NEW HexTests with golden vectors — hexString now backs attestation-critical hashes, so a silent change would alter every provider-reported hash fleet-wide. - provider-swift ProtocolCodec.swift: shared-encoder comment now states the defensible safety claim (configured-once + Sendable) instead of the false 'no mutable state'. Backward-compat verification (mixed-version fleet, both directions) came back COMPATIBLE: wire bytes identical (opcode .text set in untouched ChunkFrameWriter), decode fast path falls back to the exact old envelope decode, precomputed-key decrypt equals box.Open, cancels cannot overtake their own request (WriteText blocks to wire completion; every cancel site verified), old providers no-op safely on early control frames. * fix(review): address Codex P2s — scanner duplicate-key fallback, direct-mode doc - protocol/type_scan.go: scanTopLevelString now scans ALL top-level keys and bails to the envelope decode on a repeated 'type' key or any case-variant ('Type') — encoding/json is last-match-wins and case-insensitive, so the first-wins fast path could dispatch a malformed frame to the wrong handler. Divergence eliminated; the never-wrong invariant now holds unconditionally (skipReference escape hatch removed from the tests). Benchmark still ~40% faster than the double-parse baseline (2.2us vs 3.6-4.0us, 17 vs 23 allocs). - docs/provider/direct-mode.md: no longer points at the deleted console-ui/src/lib/localFirst.ts helper; documents the local-first fallback pattern inline and notes the prototype lives in git history. * docs(direct-mode): self-route header in the fallback example (Codex P2) Without X-Darkbloom-Route: self the coordinator fallback would hit the public paid fleet, contradicting the section's 'both paths are free' promise.
…ement, and false UI claims (#492) Darkbloom never uses Hypervisor.framework. SecurityChecks.isHypervisorActive() was a hardcoded-'return false' stub whose value flowed into attestation, privacy capabilities, the trust UI, and marketing copy as a memory-isolation claim that never existed. This removes the concept end to end while keeping wire + signature tolerance for the old fleet (coordinator deploys first; providers update lazily). Provider (Swift): - Delete the isHypervisorActive() stub and the ignored verifySecurityPosture parameter. - PrivacyCapabilities: hypervisorActive removed (property, CodingKeys, custom decoder that existed only to default it). - Challenge response: hypervisor_active no longer populated — omitted from both the response JSON and the signed canonical status bytes. - Registration SE blob already carried no hypervisor field; absence is now pinned by tests using the exact signing encoder settings. - com.apple.security.hypervisor entitlement removed from both plists (Hypervisor.framework is never imported; pure capability reduction — changes the signed binary's declared entitlements, nothing else). Coordinator (Go) — legacy tolerance retained: - PrivacyCapabilities.HypervisorActive removed (old providers still send the key; encoding/json drops unknown fields). - AttestationBlob.HypervisorActive bool -> *bool, emitted in the canonical re-serialization only when present, so old providers' signed blobs still verify and new blobs omitting it verify too. - AttestationResponseMessage.HypervisorActive (*bool) and its BuildStatusCanonical inclusion KEPT with deprecation comments: old providers sign hypervisor_active into the canonical status; removing it would break their StatusSignature verification. Remove once the fleet floor passes v0.6.31. - Write-only VerificationResult.HypervisorActive removed; hypervisor log fields and the SIP-refresh hypervisor half dropped. - Cross-language golden-bytes tests updated in lockstep (Go and Swift canonical bytes pinned identical); legacy shapes (hypervisor_active true/false) pinned so old-fleet signatures keep verifying. UI + docs: - Chat system context no longer claims 'Hypervisor.framework memory isolation'; trust explainer drops 'hypervisor isolation status' (and its stale Python/vllm-mlx runtime-hash list -> MLX-Swift reality). - Attestation docs describe hypervisor_active as a retired legacy field kept only for signature compat; threat-model T-028 rewritten honestly (the control never existed); README hardening table row removed; AGENTS/CLAUDE entitlements note corrected to (network, keychain). Left alone deliberately: papers/*.tex (historical document) and landing/index.html (one mention; file has in-flight edits on another branch — flagged for follow-up).
#496) 'darkbloom stop' only ran 'launchctl bootout', which deregisters the job from the current login session. The provider and watchdog plists stayed in ~/Library/LaunchAgents with RunAtLoad=true, so launchd re-bootstrapped them at the next login/reboot and restarted a provider the user had explicitly stopped. stop now also runs 'launchctl disable' (persists in launchd's per-user override database across reboots) for the provider (canonical + legacy labels) and the watchdog; start/restart re-enable before bootstrap. The plist stays on disk so 'darkbloom restart' keeps the model selection.
…edesign (#493) * feat(console-ui): move leaderboard to standalone /leaderboard tab Extract the leaderboard out of the monolithic stats page into a modular src/components/leaderboard/ package (types, useLeaderboard hook, format helpers + tests, TotalsStrip, PodiumCard, RankingsTable, Controls, thin LeaderboardContent orchestrator), add a /leaderboard route and sidebar nav item, and make the Stats page overview-only. Behavior-preserving port; same /api/leaderboard + /api/network/totals proxies. Co-authored-by: Cursor <cursoragent@cursor.com> * feat(console-ui): simplify leaderboard to ranking-first 24h design Drop the network-totals strip, window toggle, and jobs metric. Ranking is static from the last 24 hours with two views: Earnings (annualized $/yr rates with work/rewards split, no token counts) and Tokens (24h tokens served, no earnings). Adds formatAnnualizedUSD with tests. Co-authored-by: Cursor <cursoragent@cursor.com> * feat(console-ui): add per-day sub-values, drop /yr suffix from earnings Column headers ("Earnings / yr") now carry the unit; values are bare annualized dollars with the per-day rate as smaller muted text beneath each table cell. Podium breakdown inherits the suffix-free formatter (its "Annualized rate" label states the unit). Co-authored-by: Cursor <cursoragent@cursor.com> * feat(console-ui): show full dollar amounts instead of K/M abbreviations formatUSDFromMicro now renders complete numbers with thousands separators (e.g. $2,171 instead of $2.2K) across podium and table. Co-authored-by: Cursor <cursoragent@cursor.com> * feat(console-ui): medal-styled podium cards, remove Earn Now button Top-3 cards get gold/silver/bronze treatment (Crown/Medal/Award icons, rank-toned tinted borders and backgrounds, larger headline, separated work/rewards footer). Drops the Earn Now CTA and the now-unused leaderboardRankTone helper. Co-authored-by: Cursor <cursoragent@cursor.com> * feat(console-ui): add demand-fluctuation disclaimer to leaderboard header Restores a callout in the style of the old ramp-up note: annualized rates are a 24h snapshot, not a guarantee, and will shift with demand, pricing, and provider participation. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(console-ui): extract RankingsBody from LeaderboardContent Replaces the nested loading/error/empty ternary chain with early returns in a dedicated component; fixes the two sonarjs lint warnings. Co-authored-by: Cursor <cursoragent@cursor.com> * test(console-ui): add useLeaderboard hook tests, drop orphaned totals proxy Reviewer follow-ups: the /api/network/totals proxy lost its only consumer when the totals strip was removed, so delete it; add hook tests covering the fixed 24h window, metric refetch, and HTTP errors. Co-authored-by: Cursor <cursoragent@cursor.com> * feat(console-ui): fold top-3 podium into rankings table medal pills Co-authored-by: Cursor <cursoragent@cursor.com> * feat(console-ui): tint top-3 leaderboard rows to match medal tones Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Gajesh Naik <26431906+Gajesh2007@users.noreply.github.com>
…rrent codebase (#495) * docs(agents): verify and refresh AGENTS.md + CLAUDE.md against the current codebase Every factual claim in both files was checked against source; stale ones fixed with evidence, missing subsystems added, deleted things purged. Headline corrections: - Memory model: the 'estimatedMemoryGb * 3.0 headroom' rule is long gone — documented UnifiedMemoryCap (0.90 cap fraction, 2 GiB OS floor, 4 GiB load headroom, post-load measured-KV guard, env overrides) and the freeForLoadGB mirroring on the coordinator side. - Provider scoring: ScoreProvider no longer exists; selection is cost/penalty-based. WarmModels warm-bonus claim removed. - pendingModelLoads readers: now also cold-spill eligibility + warm-pool controller. - coordinator/e2e/ -> coordinator/internal/e2e/; telemetry/ vs datadog/ package descriptions swapped to reality; consumer routes (no transcriptions/images; /v1/responses exists); Solana rail removed from billing descriptions; darkbloom CLI command list (no serve). - ChatML auto-inject claim was false — replaced with TemplateRenderCheck / template_render_ok routing reality. - LatestProviderVersion is the no-release-row fallback, not the source of truth; release tag shapes; runbook paths (docs/operations/*). - CLAUDE.md store-selection claim ('Postgres not used in production yet') was flat wrong; ARV enforcement claim softened to logged-only. - 12 -> 14 E2E tests; structure maps gain apns/config/env/profilesign/ stateexport/routingsim/api/types, admin-ui/, landing/, deploy/. Added (one bullet each): type_scan single-parse decode, chunk_key_cache forget-on-terminal lifecycle, two-lane provider writer + watchdog, chunk-overflow 250ms-grace-then-499 policy, hypervisor_active legacy-compat note (#492). Open question flagged for review: CLAUDE.md Infrastructure table claims the prod console is an EigenCloud app — could not be confirmed from the repo; left unchanged. * docs(agents): tighten three claims from Codex review of #495 - metallib source is libs/mlx-swift/Source/Cmlx/mlx (the nested tree Cmlx compiles against), not the top-level libs/mlx submodule - /v1/releases/latest 404s without a release row: installs/self-updates are fixed by registering the release; LatestProviderVersion only backs the version-display fallback - template_render_ok=false fences ALL request shapes, not just tool-bearing ones; only capability version floors are tool-scoped
…ma 4 + GPT-OSS 20B) (#499) * feat(engine-v2): provider bridge for ContinuousBatchingV2 behind DARKBLOOM_ENGINE_V2 - EngineV2Config: env DARKBLOOM_ENGINE_V2 / [backend] engine_v2 key selection, gemma-4*/gpt-oss* per-model allowlist (DARKBLOOM_ENGINE_V2_MODELS override), EngineV2Factory with safe legacy fallback + WARN engine_health telemetry (operation=engine_v2_fallback) on v2 init failure - EngineV2Bridge: adapts CBv2Engine to the existing GenerationEvent surface; event framing mirrors BatchScheduler+EngineBridge (chunk/info/error, abort usage-before-error, teardown sentinel); billing-zero max() defense and the legacy decode-TPS methodology/EWMA - EngineV2Bridge+Translation: pure ChatRequest -> CBv2Request translation (temperature ?? 0.0 legacy default, logit_bias string-key id parsing, logprobs/top_logprobs mapping, seed passthrough, buildStopTokenIds-semantics stop resolution, max_tokens defaulting); CBv2KVError.capacityExhausted -> canonical token_budget_exhausted: string (429/503 classification unchanged) - EngineV2Bridge+Capacity: CBv2CapacitySnapshot -> existing BackendSlotCapacity fields (no protocol change), truthful bytes-derived token budgets, WedgeMonitor reuse + engine_v2 step_wedge transition telemetry - EngineV2Runtime: process-wide bridge registry; ProviderLoop capacity heartbeat folds v2 slots, handleCancellation fans request-id out to CBv2Engine.cancel; both no-ops when the flag is off - ChatCompletionRequest: additive optional logit_bias/logprobs/top_logprobs (v2 path only; legacy ignores them) - docs/engine-v2/CONTRACT-ISSUES-H-provider.md: contract gaps + chosen shapes - submodule: pin libs/mlx-swift-lm at the CBv2 contracts commit (fe60e17) * test(engine-v2): bridge unit suite — scripted CBv2Engine stub, no models/network 34 tests in 7 suites (swift test --filter EngineV2): - translation: field-by-field CBv2Request mapping, legacy greedy defaults, logit_bias string-key parsing, logprobs/top_logprobs mapping, buildStopTokenIds-semantics stop resolution + construction-time stamping - event framing: fixture-compare against the recorded legacy BatchScheduler+EngineBridge stream shape (chunks -> single info -> finish; abort usage-before-error; teardown sentinel; billing-zero max() defense; tokenize failure short-circuits before the engine) - error mapping: capacityExhausted -> token_budget_exhausted (retryable .tokenBudgetExhausted class), backendIneligible/unknown -> generationFailed - cancellation: provider id -> minted CBv2RequestID, unknown-id no-op, EngineV2Runtime fan-out (ProviderLoop hook path) - capacity: snapshot -> BackendSlotCapacity (bytes-derived budgets, idle/ running states, wedge counters), runtime aggregate summary - config gating: env/config flag matrix, allowlist + env override, factory legacy selection without invoking the builder, init-failure fallback with engine_v2_fallback telemetry, engine_v2 TOML key decode * fix(engine-v2): duplicate request-id guard + deterministic fan-out test - EngineV2Bridge.submitTokenized: reject a second submit under an active request id with the legacy planner's canonical message ('token_budget_exhausted: duplicate request ID' -> .requestRejected, 400) before minting an engine id, so two pumps can never share bookkeeping - test: duplicate-id rejection classification + engine isolation - test: runtimeFanOut holds the submitted stream alive for the whole test; dropping it fired onTermination(.cancelled) and raced a second (idempotent) engine cancel into the count assertions * feat(engine-v2): wire ContinuousBatchingV2 into production serving paths Closes the P1 integration gap: the EngineV2 bridge (DARKBLOOM_ENGINE_V2 / engine_v2) was implemented and tested but never instantiated outside tests — model loading always built only the legacy BatchScheduler, both request paths captured slot.scheduler directly, and the EngineV2Runtime registry stayed permanently empty. - Model load (ProviderLoop+ModelLoading -> new ProviderLoop+EngineV2): when EngineV2Config selects v2 for a (non-VLM) model, assemble the real CBv2 engine over the already-loaded container via the new EngineV2Factory.makeProductionEngine (Gemma4/GPT-OSS cbv2LayerKinds + newCacheV2, contiguous KV backend sized from the unified-memory KV budget, text detokenizer over the model tokenizer), register the bridge with EngineV2Runtime, and store it on ModelSlot alongside (never replacing) the legacy scheduler. - Routing: MultiModelBatchSchedulerEngine registry entries and AcquiredModel carry the optional bridge; streamChatCompletion submits the same tokenized prompt through it when present (translated sampling/ stop params — v2 additionally honors penalties + stop strings the legacy submit drops) and cancels through the owning engine. The coordinator inference handler and the unified local endpoint both pass slot.engineV2; standalone --local mode intentionally stays legacy. - Zero-overhead flag-off: capacity heartbeats and cancellation fan-out consult the runtime only when at least one v2 slot exists (previously an unconditional empty-registry actor hop). A v2-served slot reports capacity through its bridge ONLY (the dormant legacy scheduler is skipped so the model is never advertised twice). unloadModel unregisters and drains the bridge. - Failure semantics now exercised in production: v2 init throw -> WARN engine_health (engine_v2_fallback) + legacy fallback; VLM slots skip selection silently (no per-load WARN for permanently-unsupported shapes). - Pin libs/mlx-swift-lm to b2b6d53 (integrated CBv2 engine) — the public EngineV2 assembly the production factory constructs did not exist at the previously pinned SHA. Tests (EngineV2ProductionWiringTests, live-isolated, no weights/network): slot-factory flag-off/allowlist/VLM-gate/init-failure + runtime registration, both request-routing shapes through a scripted CBv2Engine, cancellation propagation to the engine-minted id, runtime-guard zero-consult, heartbeat fold-in with the no-double-report regression, unload retirement, and the production factory throw paths. 50 EngineV2 tests green; full suite 1214 tests / 96 suites green. * chore(submodule): bump mlx-swift-lm to engine-v2 final tip 8662159 Pulls the CBv2 contract updates the provider bridge adopts next: - CBv2Engine is now Sendable (contract-issue H§1 fix) - CBv2CapacitySnapshot.stepsExecuted monotonic step counter (H§3 fix) - CBv2Event.delta logprobs populated when sampling.topLogprobs > 0 (E§1) - CBv2Request.cacheSalt per-request prefix-cache scope (TB-007) - prefix-cache donation skip for quantized KV; shutdown timeout hardening * feat(engine-v2): adopt final CBv2 contract — drop engine box, real step counter, logprobs + cacheSalt plumbing Follow-ups on the engine-v2 bridge against mlx-swift-lm tip 8662159: - Drop EngineV2EngineBox: CBv2Engine is Sendable in the contract now, so the bridge actor holds and calls the engine directly (H§1 resolved). - Wedge signal from the engine's own counter: WedgeMonitor now samples CBv2CapacitySnapshot.stepsExecuted (published every step) instead of the bridge's event-count proxy (H§3 resolved). New capacity test pins the passthrough plus the flatline-under-hanging-admits wedge verdict. - Logprobs passthrough (coordinator path): engine delta logprobs are converted to the OpenAI streaming entry shape and ride out-of-band via a per-request EngineV2LogprobsChannel (bridge pump appends before the chunk yield); the inference handler splices them into the next content-bearing SSE chunk as choices[0].logprobs = {content: [...]} before encryption. The legacy engine NEVER emitted logprobs, so this is the OpenAI-standard shape, not a legacy match; the standalone --local path (frames served inside the upstream router) still emits none. logprobs/top_logprobs are decoded from the sealed body (the upstream request type does not model them) like reasoning_effort. - cacheSalt forward plumbing (inert): the per-tenant cache scope (SHA256(prompt_cache_key)/SHA256(user), the same value the legacy scheduler threads into the checkpoint cache) now maps onto CBv2Request.cacheSalt; "" maps to nil (cache-level salt fallback). Production still constructs the v2 engine with prefixCache: nil. Tests: scripted-stub coverage for channel ordering/conversion, SSE splice shape + skip rules, extractLogprobsSpec, cacheSalt mapping on both bridge entry points, and engine wiring threading scope+plumbing. * chore(submodule): bump mlx-swift-lm to engine-v2 final tip a0ae73b (paged eligibility guard + bench driver) * chore(submodule): bump mlx-swift-lm to engine-v2 composed tip 255b223 (compiled decode + kernel-opt + profiler) * fix(engine-v2): thread sealed-body logit_bias + seed through the coordinator v2 translation The upstream OpenAIChatCompletionRequest models neither logit_bias nor seed, so the coordinator path's internal ChatCompletionRequest always carried nil for both and EngineV2Translation silently dropped them (PR #499 review, Codex P2 #1). Recover them the same way logprobs / reasoning_effort / cache scope are recovered: decode straight from the E2E-sealed body (ProviderLoop.extractSamplingOverrides, field-independent probes) and overlay onto the v2 translation via a new EngineV2SamplingOverrides plumbing struct. v2 engine path only — the legacy engine never honored either knob and its submit call is byte-identical; the --local path still drops both (no provider seam, documented in the narrowed KNOWN DEVIATION). Tests: sealed-body extraction (both fields, absence, malformed-field independence), translate overlay → CBv2SamplingParams, and a coordinator-path wiring test proving the parsed bias + seed reach the stub CBv2Engine. * fix(engine-v2): truthful shared-KV accounting + fp16 sizing for kv_quant models Two coupled memory-accounting gaps from PR #499 review (Codex P2 #2 + #3): Fix #2 (shared-budget KV accounting): the v2 bridge bypassed the shared GlobalKVCacheBudget, so model-load admission (availableMemoryGb minus outstanding reservations) and the legacy live-KV gate under-counted when a v2 slot was active alongside a legacy slot — a concurrent legacy load could over-commit unified memory. The v2 engine runs its own optimistic + preemption admission against a private byte ledger, so a per-request GATING reservation would fight that design (and double-count against the engine's ledger). Instead each in-flight v2 request records its worst-case KV footprint (prompt + maxTokens, fp16 rate) in the shared ledger via a new bookkeeping-only recordEngineKV (never rejects — the v2 engine stays authoritative for v2 acceptance), released on every terminal/teardown path in the pump. outstandingReservedBytes() now truthfully includes v2 in-flight KV; the v2 engine's own ledger is unchanged. Fix #3 (kv_quant fp16 sizing): engine_v2 builds UNQUANTIZED CBv2LayerCache regardless of kv_quant (makeProductionEngine never builds a quantized cache), but the slot factory sized the bridge from the scheduler's live kvBytesPerToken — the QUANTIZED rate when kv_quant=true — overstating v2 token budgets/heartbeats 2–4x and under-sizing the fix-#2 reservation. EngineV2KVSizing.resolve now picks the fp16 rate (scheduler fp16KVBytesPerToken) for v2 sizing and emits a WARN engine_health telemetry (engine_v2_kv_quant_unsupported) once per load when kv_quant actually engaged for the model. Tests: sizing decision (both kv_quant configs), record+release invariant (finish and no-terminal teardown), no-op when rate unknown, and the slot factory's fp16 sizing + WARN end-to-end via a new BatchScheduler rate seam. * harden(engine-v2): request-id validation, pump lifecycle, bounded logprobs buffer, sanity guards Security/hardening findings from PR #499 review (ethenotethan): #4 request-id validation: a caller-supplied request-id is validated (non-empty, ≤256 chars, no ASCII control chars) before it becomes a dictionary key / cancel-correlation handle; nil/invalid ids normalize to a fresh req-<uuid>. Static normalizedRequestId/isValidRequestId are pure and unit-tested. This also subsumes #11 (duplicate-id): prod ids are provider-minted UUIDs, and any malformed id is now normalized to a fresh UUID before the existing duplicate guard, so an attacker cannot induce a collision — no separate change needed. #5 nextRawId overflow: += → &+= (wrapping) with a comment; 2^64 is unreachable in practice, this makes the theoretical overflow explicitly defined rather than a trap. #6 engine.submit on the actor: verified O(1) non-blocking enqueue against the CBv2Engine contract (admission check + register + enqueue, no forward pass; tokenization already off-actor) and documented it at the call site. No code change. #7 pump task lifecycle: per-request pump Tasks are now tracked in pumpTasks and cancelled by shutdown() (before the engine drain), so none outlives the bridge; AsyncStream cancellation unwinds each pump through its teardown path, releasing per-request KV. Tested: shutdown cancels a live pump and its reservation returns to 0. #8 bounded logprobs buffer: EngineV2LogprobsChannel caps at 4096 entries with drop-oldest eviction + a droppedCount and a one-line WARN on first drop, so a stalled/never-draining consumer can't grow it unboundedly. Dropping logprobs never affects content/billing/hash (out-of-band). #9 dropped logit_bias keys: parseLogitBiasCountingDropped returns the count of dropped (non-numeric/negative) keys; samplingParams emits a count-only WARN (never the keys — request content) instead of dropping silently. #10 kvBytesCapacity sanity: clampKVBytesCapacity caps the v2 admission ceiling at physical RAM so a mis-derived budget degrades to all-of-memory rather than an absurd ceiling. Tests: id validation matrix + normalization-in-submit, wrapping id, shutdown-cancels-live-pumps, logprobs cap drop-oldest + droppedCount, dropped-key count, and the capacity clamp. Full suite green (1245). * fix(engine-v2): record shared-KV reservation atomically with admission; truthful kv_quant WARN Addresses both reviewers of the PR #499 fixes: Codex (race): the shared-budget KV reservation was recorded in the detached pump task, which runs AFTER submitTokenized returns — leaving a window where a concurrent model-load gate could observe zero v2 KV between engine admission and the pump starting. Make submit/submitTokenized async (all call sites already await) and record the reservation SYNCHRONOUSLY right after engine.submit succeeds, before returning the stream. The pump now only releases. Tests assert the no-gap invariant (reservation visible the instant submit returns, no polling). Claude (truthfulness): move the engine_v2_kv_quant_unsupported WARN below the makeBridgeIfSelected guard so it fires only when the v2 bridge actually built. On a v2 init-failure fallback the model serves via legacy, which DOES honor kv_quant, so warning there was untruthful. Full suite green (1245). * chore(submodule): bump mlx-swift-lm to engine-v2 tip e4158bc (PR#62 review fixes) * fix(engine-v2): gate v2 admission on the shared KV budget before engine submission The bridge now RESERVES each request's worst-case KV footprint in the process-wide GlobalKVCacheBudget before engine.submit — atomically on the budget actor (check + record in one hop) — and rejects with the canonical token_budget_exhausted capacity error when the shared pool has no headroom, exactly like the legacy scheduler's KV-reserve rejection. Engine rejections after the gate release the reservation. The non-gating recordEngineKV is removed (the gating reserve subsumes it). Degenerate maxTokens<=0 requests skip the gate (the engine finishes them without allocating KV). Round-2 PR#499 P2: with another slot's live KV reserved, record-after-accept could overcommit unified memory on multi-slot providers. * fix(engine-v2): report committed worst-case tokens in heartbeat budget fields The coordinator's token-budget gate (activeTokenBudgetUsed + queuedTokenBudget + requestTokens <= activeTokenBudgetMax) expects 'used' to carry the committed worst-case reservation of accepted requests — the legacy scheduler's Σ(prompt + maxTokens) — and never consults max_tokens_potential. Feeding it materialized KV bytes let the coordinator over-route while accepted long-max_tokens requests still had unprotected growth. Mapping (existing wire fields only, documented in backendSlotCapacity): activeTokens stays engine truth (real KV-resident tokens); activeTokenBudgetUsed and maxTokensPotential carry Σ(prompt+maxTokens); activeTokenBudgetMax is the engine byte capacity / fp16 rate (0 when the rate is unknown, disengaging the gate). Round-2 PR#499 P2. * fix(engine-v2): stable (seed, prompt)-derived engine ids for reproducible seeded sampling The v2 sampler keys its RNG on (seed, requestID.raw, stepIndex), so minting a fresh monotonic id per submission made identical seeded requests sample differently depending on prior traffic. Seeded submissions now derive their engine id from a SplitMix64 chain over (seed, promptTokens), tagged with bit 63 to stay disjoint from the monotonic family; the id is minted in the same synchronous stretch as engine.submit so the liveness collision check cannot race a concurrent identical submission. An identical seeded request that is still live falls back to a fresh id (documented waiver); batching reproducibility remains best-effort per the engine contract. Test: stochastic sampler stub keyed exactly like SamplerV2 proves same-seed + same-prompt submissions reproduce identical output across interleaved traffic, and diverge on different seed/prompt. Round-2 PR#499 P2. * fix(engine-v2): size v2 KV admission ceilings against fleet-wide residency A v2 engine's kvBytesCapacity was derived as if only its own slot's weights were resident, so multi-slot providers could grant Σ(ceilings) far past the unified-memory KV budget. The slot factory now derives the ceiling with EngineV2KVSizing.engineKVBytesCapacity: kvBudgetBytes over ALL resident models' weights (co-resident slots + the new model) minus the construction-fixed ceilings already granted to existing v2 engines. When nothing remains the capacity is 0 and makeProductionEngine throws noKVHeadroom, so the slot serves via the legacy scheduler (whose per-request shared-budget gate — and now the v2 bridge's too — needs no static ceiling). Test hooks now receive the computed capacity ((modelId, kvBytesCapacity)) so wiring tests assert the two-slot derivation end-to-end; pure sizing tests pin the sum-within-budget invariant and degenerate clamps. Round-2 PR#499 P2. * docs(engine-v2): hardening-review rationales — pump-task bound, logprobs lock, cancel scan - pumpTasks: bounded by the engine's own admission (maxConcurrent + maxWaiting, ~68 in production) — pump tasks exist only for engine-accepted requests and self-clear on terminal; cite the contract. - EngineV2LogprobsChannel: why an uncontended NSLock beats actor isolation for a two-party bounded append/swap on the delta path. - EngineV2Runtime.cancel: O(n) scan is over v2-served MODELS (<= slot cap, single digits), not requests; an id->bridge index would add hot-path register/unregister traffic to save a 3-entry scan on a rare event. * fix(engine-v2): duplicate re-check across the gate suspension; scope KV release to the reserving request The shared-budget reserve introduced the bridge's first suspension between the duplicate-id guard and the bookkeeping install. A concurrent same-id submission that SKIPS the gate (degenerate maxTokens / unknown rate) could slip in across that gap and be overwritten; and the pump's unconditional release could drop a same-keyed reservation owned by a different submission. Now: re-check active[id] after the reserve (roll back + reject as duplicate), and the pump releases only when its own submission reserved (holdsSharedReservation). Degenerate maxTokens<=0 requests are covered by a test proving they skip the gate even on an exhausted pool. * chore(submodule): bump mlx-swift-lm to engine-v2 tip ef1ad12 (bench cells on top of e4158bc) * chore(submodule): bump mlx-swift-lm to engine-v2 tip ec541f5 (PR#62 round-2 review fixes) * feat(provider): config keys for startup preload + auto-update rollover jitter New provider.toml keys (all additive; defaults chosen to be fleet-safe): [backend] - startup_preload (default true): preload models before coordinator registration on boot. A fresh install has no persisted serving set, so out-of-the-box behavior is unchanged until a model has been served. - preload_models (default []): explicit preload list, operator order; empty means "the models served before the last restart". - startup_preload_timeout_secs (default 120): upper bound on how long registration is deferred while the preload runs. - startup_selftest (default true): 1-token greedy decode through the serving path after each preload. - startup_selftest_fail_closed (default false): retire a model whose self-test failed instead of keeping it advertised (fail-open). [provider] - update_jitter_seconds (default 300): max random delay between staging a verified auto-update bundle and beginning the drain+restart. * feat(provider): startup model preload + registration readiness gate Fixes the v0.6.30-class cold-restart failure mode: after a release restart the provider registered (and attracted routing) with NOTHING loaded, so first requests paid the full ~26 GB weight load + engine build inside a live request — and at a fleet rollover every box was cold at once (first_chunk_timeout storm). - LoadedModelsStore: persists the live serving set to ~/.darkbloom/loaded-models.json (data dir — survives auto-updates) on every load and every NON-shutdown unload (idle timeout, eviction, retirement). Shutdown teardown deliberately preserves it so a stop/update/restart remembers what was being served. - StartupPreloader: sequential, deps-injected preload driver. Memory admission never evicts — a candidate that does not fit what is free is skipped with a WARN and left to the lazy-load path; failed loads and self-tests degrade, never crash. - ProviderLoop.runStartupPreloadGate(): runs in run() BEFORE the coordinator client exists. Plan = preload_models (operator order) or the persisted set (biggest first), filtered to advertised models and capped at max_model_slots. Each preload is the full ensureModelLoaded (weights + legacy scheduler + v2 bridge/warmup when flagged). Readiness vs availability: registration is deferred until warm, at most startup_preload_timeout_secs (default 120s). On timeout the provider registers anyway — a lone provider for a model must still serve it cold, and the lazy-load path is unchanged as the fallback — while the remaining loads continue in the background. The heartbeat warm_models field (which the coordinator's warm-model bonus scores) stays truthful throughout: it is derived from live slots only. - Startup self-test (startup_selftest, default on): after each preload, a 1-token greedy decode through the REAL serving path (v2 bridge when flagged, else legacy scheduler) forces end-to-end warmth (Metal JIT, compiled buckets, chat-template render). Failure emits WARN engine_health telemetry and is fail-open by default; startup_selftest_fail_closed retires the model from this run's advertised set (registration filters through advertisedModels). Tests (live-isolated, scripted loaders — no weights, no network): persistence round-trip incl. unload removal + the shutdown-preserve guard; preloader ordering/admission-skip/failure-continuation/ fail-open/fail-closed/cancellation; plan order + unknown-id filter + dedup + slot cap; gate defers-until-warm, proceeds-at-timeout with background continuation, and fail-closed retirement. * feat(provider): rollover jitter before background auto-update install Behavior change at defaults (update_jitter_seconds = 300) — justified as the fleet-rollover fix: providers discover a release on aligned 30-minute update ticks (ticks correlate after any previous synchronized restart) and drained + restarted in near-unison, putting the entire fleet cold at the same moment (first_chunk_timeout storm). A uniform random 0-300s delay between staging the verified bundle and beginning the drain decorrelates the restarts; combined with the startup preload gate, warm boxes absorb traffic while each cold box re-warms. Placement: the delay sits strictly AFTER download + signature verification (no security-critical check is deferred; the bundle is already verified and staged, and the provider keeps serving while it waits) and BEFORE beginDraining. Forced/manual updates — the darkbloom-update command and the startup update check — do not go through AutoUpdateController and stay immediate. A 1-hour sanity cap bounds a misconfigured value. Tests: delay bounds with a seeded RNG, zero-max and forced bypasses, the sanity cap, and controller sequencing (jitter strictly between stage and drain; never invoked on up-to-date / check-failed / stage-failed paths). * fix(provider): review fixes — cancel-safe jitter, no-evict preload loads, retirement + persistence hygiene Review round (Codex + Claude) on the startup-preload/jitter feature found four real gaps; all fixed with regression tests: 1. Cancel-safe rollover jitter. The jitter closure slept with `try? await Task.sleep`, so a shutdown cancelling autoUpdateTask inside the (up to 300s) jitter window returned early and the controller proceeded to drain + commit + RESTART a shutting-down provider. AutoUpdateController now checks Task.isCancelled after waitBeforeInstall and aborts with a new .cancelled outcome (resumeServing discards the staged bundle; the next tick retries). 2. No-evict preload loads, enforced in the real load path. The preloader's freeMemoryGb pre-check was advisory only — the actual ensureModelLoaded could still slot-cap-evict or memory-evict an EARLIER preloaded (idle) model to fit a later candidate, and a local-endpoint load interleaving with the driver could make the pre-check stale. ensureModelLoaded/evictUntilAvailable now take allowEviction (default true = live-traffic behavior unchanged); the startup preload passes false, and the check runs inside the serialized isLoadingAny critical section so it cannot go stale. Tested against the production load path (fake HF snapshot + real admission gates): no-evict refuses at the slot cap and the memory gate while the default policy still evicts. 3. Post-registration fail-closed retirement now reaches the coordinator. When the gate times out, registration has already announced the model; a later self-test retirement only edited local state, so the coordinator kept routing it. Retirement now mirrors the hard-swap drop: unadvertise on the client store + forceReconnect (new provider-side CoordinatorClient method; a fresh register is the existing wire mechanism for advertised-set removals — models_update is additive, and NO protocol changes are introduced). End-to-end test: real CoordinatorClient against MockCoordinator observes the second register without the retired model. 4. Loaded-models persistence is inert until run() arms it. The persist-on-load/unload write points fired from ANY ProviderLoop — including unit tests that install/unload slots — clobbering the operator's real ~/.darkbloom/loaded-models.json (the next boot's preload plan). Persistence is now gated on loadedModelsPersistenceEnabled, set only by run() (and explicitly by the test seam with a temp path). * fix(engine-v2): honor memory_reserve_gb in v2 KV sizing; clamp heartbeat budget max to live fleet residency (PR#499 r3 P2) Two coupled capacity-truth fixes on the v2 path: * memory_reserve_gb in v2 ceilings: UnifiedMemoryCap.kvBudgetBytes gains configReserveBytes (effective cap = min(hardCap, physical - reserve), the same max(configReserve, capImplied) hold-back the load gate and the shared KV gate apply); EngineV2KVSizing.engineKVBytesCapacity threads it and the slot factory passes the operator reserve. On 16/32 GiB boxes (default 4 GiB reserve > cap-implied reserve) the bridge no longer advertises/privately admits capacity the shared gate rejects post-acceptance. * Stale activeTokenBudgetMax after later loads: v2 admission ceilings are construction-fixed, so a model loaded later (especially a legacy/non-allowlisted slot) left existing bridges advertising a stale budget the coordinator kept routing against. The heartbeat now CLAMPS the reported max to the sizing function's current answer: ProviderLoop.updateAggregateCapacity snapshots live fleet residency (all slots' weights + operator reserve) into EngineV2Runtime.capacitySummary(fleetKV:), which recomputes each bridge's budget via EngineV2KVSizing.liveEngineKVBytesBudget (min(construction grant, current fleet answer)) and passes it to backendSlotCapacity(kvBytesBudgetClamp:). The engine's private ledger keeps its grant; only the coordinator-visible figure shrinks. Heartbeat cadence only - never the submit path. Tests: kvBudgetBytes reserve regimes (binding / no-op / degenerate), pure sizing with reserve both regimes, live-clamp pure math (later legacy load shrinks by exactly its weights; co-resident v2 grants subtract; never inflates past the grant; clamps to 0), and an end-to-end wiring test: load a v2 slot, register a second slot's weights, heartbeat max shrinks by exactly weights/rate while the engine keeps its construction grant. * fix(engine-v2): preserve queue-full classification on v2 admission rejections (PR#499 r3 P2) EngineV2.submit surfaces BOTH byte-ledger capacity exhaustion and its waiting-queue-full / draining-for-shutdown guards as CBv2KVError.capacityExhausted; the bridge mapped every shape to the generic token_budget_exhausted string, so v2 queue saturation surfaced as a 503 token-budget failure instead of the legacy 429 + Retry-After queue-full path. The two families are distinguishable by shape: the slot-rejection guards throw the sentinel (needed: 1, available: 0) (no byte estimate exists), while byte-ledger rejections carry needed = a multiple of the per-token KV cost (never 1) and available = admissibleBytesCapacity (> 0 for any constructible engine). EngineV2Translation .admissionErrorMessage now maps the sentinel to the exact legacy queue-full string ('token_budget_exhausted: request queue full', BatchSchedulerTypes.RejectionReason.queueFull), which fromSchedulerMessage classifies as .queueFull -> 429. Shutdown-drain riding the same sentinel lands on queue-full too, matching the legacy handler's .queueFull("provider shutting down"). No new classification strings, no wire changes. Tests: sentinel -> queue-full string -> .queueFull -> 429; byte-figure rejection stays .tokenBudgetExhausted -> 503; near-sentinel byte figures (needed 2/available 0, needed 1/available 512) never classify as queue-full. Also carries the bridge/runtime heartbeat-clamp tests for the previous commit (same test file). * fix(engine-v2): bound pending logprobs before the first content frame (PR#499 r3 P2) The inference handler drains the capped EngineV2LogprobsChannel into a local pendingLogprobs array every frame but only clears it when injectLogprobs finds a content-bearing frame - so a stream with a long reasoning-only/tool-only prefix (GPT-OSS reasoning) re-accumulated everything the channel cap had just bounded, growing without limit and eventually serializing thousands of entries into one huge first content chunk. EngineV2LogprobsChannel.capPending mirrors the channel's own policy on the caller-held buffer: cap at maxEntries (4096), drop-OLDEST so the freshest window (the entries nearest the content that eventually renders) is kept, return the evicted count. The handler re-caps after every drain, WARNs once on first drop and logs the request's total at stream end (counts only - never token text; entries are response content and stay inside the E2E stream). Tests: drop-oldest at the real 4096 cap with count reporting and no-op below the cap; frames-loop simulation where a capped window survives reasoning-only frames and the first content chunk carries exactly the retained entries. * chore(submodule): bump mlx-swift-lm to engine-v2 tip 0c5d3a9 (round-3 review fixes) * chore(submodule): pin mlx-swift-lm to main d63b1c8 (squash-merge of engine-v2 PR #62; content-identical to 0c5d3a9) * fix(provider): startup self-test no longer poisons Qwen3.5 slots — stale VLM mrope position ids + e2e state isolation Root cause of the CI e2e regression (test 1 passes, every later test's provider dies on its first inference with '[broadcast_shapes] Shapes (1,8,L,64) and (1,1,12,64) cannot be broadcast'): 1. Qwen3.5-0.8B declares a vision_config, so the provider loads it via VLMModelFactory. The MLXVLM Qwen35 LanguageModel cached the full-prompt mrope position ids (precomputedPositionIds) on the long-lived model module and reused them for ANY later forward at cache offset 0. The startup self-test decode ('Hi', 12 prompt tokens, maxTokens=1) was the first request in every preloading provider, so the first routed request with a longer prompt sliced the stale 12-position ids (MLX clamps the out-of-range slice), producing cos/sin frozen at (1,1,12,64) against the new request's (1,8,L,64) rotary block — process-fatal in the first full-attention layer. Fixed in the mlx-swift-lm submodule (d3b1ae9): always recompute position ids at sequence start, exactly like Qwen3VL and HF transformers; precomputedPositionIds removed. Also covers Qwen35MoE (subclass). NOT self-test-specific: ANY second, longer text-only request on one scheduler crashed the same way. 2. The e2e testbed launched providers with the real user HOME, so ~/.darkbloom/loaded-models.json leaked across testbed instances: test 1's provider persisted its serving set, and test 2's freshly-booted provider startup-preloaded + self-tested it — the trigger that made every post-first test hit (1). Providers now get a per-instance temp state dir (DARKBLOOM_STATE_FILE + DARKBLOOM_LOADED_MODELS_FILE), removed on Stop. Regression test: StartupSelfTestDecodeLiveTests (weight-gated, DARKBLOOM_LIVE_MLX_TESTS) loads Qwen3.5-0.8B through the SAME VLMModelFactory selection production uses and drives the self-test-shaped request then a longer request through MultiModelBatchSchedulerEngine + MLXOpenAIService — crashes with the exact production signature without the submodule fix, passes with it. Validated end-to-end: 4 sequential e2e integration tests green locally (previously test 2+ crashed 502). * feat(provider): v0.7.0 engine posture — v2 default-on for exact prod checkpoints, legacy compiled decode opt-in, finish_reason 'length' threaded end-to-end Three coordinated changes for the v0.7.0 release posture: 1. engine_v2 default TRUE, allowlist narrowed to the EXACT production checkpoint ids (mlx-community/gpt-oss-20b-MXFP4-Q8, mlx-community/gemma-4-26b-a4b-it-8bit) — default-on scope equals the real-model-validated scope; other quantizations/families keep legacy until widened via DARKBLOOM_ENGINE_V2_MODELS. DARKBLOOM_ENGINE_V2=0 stays the absolute per-box kill switch (beats config, no release). The v2-init-failure fallback telemetry (engine_v2_fallback, WARN engine_health) now also carries the human-readable 'error' reason next to error_class + model — it is load-bearing now that v2 is the default. Exact-pattern allowlist matching compares last path components on both sides so bare ids match org-qualified patterns. Retirement (failed self-test) and hard-swap/idle/eviction paths were audited for default-on v2: every unload funnels through ProviderLoop.unloadModel, which unregisters the bridge from the runtime and drains it gracefully before scheduler teardown; the hard-swap dropAdvertisedBuild is a lazy advertised-set drop that leaves in-flight requests on the resident slot untouched. 2. Legacy compiled decode is now OPT-IN (new [backend] key legacy_compiled_decode, default false). The bumped mlx-swift-lm pin ships CompiledDecode default-ON; LegacyCompiledDecodeGate forces DARKBLOOM_COMPILED_DECODE=0 into the process env at both serve entry points (ProviderLoop.run, local standalone) BEFORE any engine/model code latches the library's static — unless the operator opted in via config or set the env var explicitly (explicit env always wins, both directions). Keeps release behavior identical to prod v0.6.30 (the compiled-decode rollback); single-stream speed ships via the v2 canary instead. 3. finish_reason 'length' is threaded end-to-end instead of being flattened to 'stop'. GenerationEvent.info gains finishReason; the legacy engine bridge passes RequestOutput.finishReason through, the v2 bridge maps CBv2FinishReason.length/.stop, the B=1 greedy fast path infers length from a full-budget decode, and MultiModelBatchSchedulerEngine forwards it as ServerGenerationInfo.stopReason (tool_calls still wins). Clients now see finish_reason 'length' on max_tokens truncation from BOTH engines. Tests: LegacyCompiledDecodeGateTests (gate matrix + real-env apply, snapshot/restore), EngineV2 gating suite updated for the exact-id allowlist + default-on selection matrix (kill switch beats default-on), fallback telemetry asserts the new 'error' field, v2 bridge test for .length/.stop/cancelled reason threading. Full provider suite green (1327 tests); live engine-path tests green (weight-gated). * docs(provider): review-nit comment fixes — GenerationEvent.info arity in docs, gate empty-env rationale
… startup warming, rollover jitter
…pt-oss-20b, gemma-4-26b-8bit, gemma-4-26b-qat-4bit), drop HF repo ids (#504) v0.7.0 gated Engine V2 on HuggingFace repo ids (mlx-community/...) that no production provider advertises — the fleet uses coordinator-catalog ids, so v2 never engaged and every box silently ran the legacy engine (confirmed against the prod provider registry). Allowlist now matches the real fleet ids; QAT-4bit included (validated: strict batch-invariance + faster).
…slots (weight-shared extraction, per-request routing) (#505) * feat(provider): engine_v2 text routing on VLM-loaded Gemma 4 slots Every prod Gemma 4 checkpoint ships a vision tower, so the provider loads it via VLMModelFactory and the slot is isVLM=true. The per-slot v2 gate (guard !isVLM else return nil) excluded 100% of Gemma traffic from engine v2 while GPT-OSS on v2 shows +78% p50 TPS. For an allowlisted VLM slot, the slot factory now extracts the CBv2-adapted MLXLLM Gemma4TextModel over the SAME weight arrays the MLXVLM wrapper holds (zero extra weight memory) and serves TEXT through engine v2; image/video requests keep the legacy VLM prepare/generate path (per-request routing, media peeled off before the bridge branch). Extraction re-applies the checkpoint's quantization structure the way loadWeights does (4bit-QAT and 8bit, incl. mixed per-layer precision), verifies with update(verify:[.all]) so any drift throws into engine_v2_fallback WARN + legacy, and runs a load-time forward parity gate (top-k containment + bounded max|Δlogit|). Telemetry logs vlm_text_routing=true. * test(provider): VLM engine_v2 extraction + per-request routing Unit (live-isolated, scripted CBv2 stub): allowlisted VLM slot builds + registers a bridge; extraction failure -> legacy + engine_v2_fallback WARN; non-allowlisted VLM slot -> silent legacy; text request routes through the bridge; image-bearing request never reaches the bridge (legacy vision path). Weight-gated live (DARKBLOOM_LIVE_MLX_TESTS + _GEMMA) on the real prod qat-4bit and 8bit checkpoints: weight-share growth ~10KB (zero copy), load-time parity gate passes, v2 == legacy greedy token-exact on the same extracted model, and a real interleaved vision request does not perturb the v2 text output. * release(provider): bump version to 0.7.2 ProviderCore.version and coordinator LatestProviderVersion -> 0.7.2 for the engine_v2 VLM text-routing capability.
Live incident: 7 providers (64GB, gemma-4-26b-8bit, v0.7.2) rejected 100%
of dispatches with token_budget_exhausted (503) from their first request
(~9,000 rejections/30min, zero successes). Their idle-looking heartbeats
kept winning the cost scheduler, and capacity-class failures are
deliberately invisible to reputation and to all three fault breakers —
sound for occasional sheds, catastrophic for a box that rejects everything.
New (provider, model)-keyed cooldown in registry/capacity_cooldown.go:
- Trips on >= N capacity rejects within window W with ZERO interleaved
accepts (defaults N=5, W=60s); any accept (first content chunk via
commitFirstContent, or clean completion via noteInferenceSuccess) resets
the streak, cooldown, and backoff — a busy-but-serving box can never trip.
- Routing gate added to providerPassesRoutingGatesLockedEx (shared by
dispatch + QuickCapacityCheck preflight, no drift).
- TTL 120s, re-probe after expiry (half-open: a still-rejecting pair
re-arms on its first post-expiry reject with exponential backoff,
capped at 10 min).
- Env tunables EIGENINFERENCE_CAPACITY_COOLDOWN_{THRESHOLD,WINDOW_SECONDS,
TTL_SECONDS,MAX_TTL_SECONDS}; THRESHOLD=0 is the kill switch.
- Distinct Datadog metric routing.capacity_cooldown_tripped
(provider_id + model tags) and a Warn log on the trip transition.
- Explicit context-overflow rejections never strike (they indict the
request, not the provider); 4xx client-shape errors never strike.
Tests: registry unit tests (trip, busy-but-serving never trips, accept
resets backoff, window slide, expiry re-probe + exponential backoff, env
tunables, kill switch, config clamps, map bounding) + api tests
(classifier table, end-to-end trip -> metric/log -> routing diverts to
healthy provider, 4xx and context-overflow never strike).
… gate+up cache; budget self-heal (#508) * fix(provider): v0.7.2 black hole — share the MoE fused gate+up cache between the VLM wrapper and the extracted text model Root cause of the 0.7.2 incident (8 boxes — 7x64GB gemma-4-26b-8bit + 1x36GB qat-4bit — rejecting 100% of requests with the shared-KV capacity string from their first request, permanently): MLXLMCommon.SwitchGLU lazily builds a fused gate+up copy of its quantized expert weights on FIRST FORWARD and retains it on the module (~540 MB/layer, 15.07 GiB model-wide on the 8-bit checkpoint, measured). The v0.7.2 VLM text extraction created a SECOND SwitchGLU tree over the same weights, and the load-time parity probe ran one forward through EACH tree — materializing TWO fused copies before the first request: 26.04 (weights) + 15.06 + 15.07 = 56.17 GiB active vs 57.6 GiB cap (0.9 x 64 GB) -> liveKVHeadroomBytes ~ 0 forever Active memory is not reclaimable by clearCache, so the KVPoolReclaimer self-heal never fired usefully, while the engine's own ledger stayed idle (nothing was ever admitted) — the two-ledgers-disagree signature. Deterministic on any box where weights + 2x fusedCache crosses the cap; reproduced locally on the real checkpoint with a simulated 64 GB memory profile, and inverted by the fix (post-fix: growth == exactly ONE shared cache, 64 GB-profile admission succeeds). Fixes: - EngineV2VLMTextExtraction now pairs every wrapper SwitchGLU with its extracted counterpart and shares ONE eagerly-built fused cache (SwitchGLU.shareFusedGateUpCache, new in mlx-swift-lm), restoring the pre-0.7.2 steady-state footprint (weights + one cache = 41.10 GiB). - The parity probe now fences the GPU stream and clears the MLX pool on exit (success and throw), so probe transients are returned before the load path's admission reads. - ensureModelLoaded re-checks serveable KV headroom AFTER the engine bridge build (the probe's materialization happens there, after the existing post-load check) and unloads + 503s instead of advertising a model whose every request the shared gate would reject. Tests (live, weight-gated): GemmaVLMParityProbeMemoryLiveTests pins the one-shared-cache growth bound, zero steady-state re-forward growth, pool hygiene, and 64 GB-profile admission; GemmaVLMEngineV2LiveTests stage (a) now asserts per-layer fused-cache array identity across the trees and awaits its scheduler unloads structurally (the deferred-Task teardown raced the next test's load). * feat(provider): GlobalKVCacheBudget audits + drops stale reservations under sustained full-rejection Defense in depth for the 0.7.2 black-hole class: when every commit has failed for 120s straight (the coordinator keeps routing, the provider keeps rejecting, and the cache-pool reclaimer isn't the binding term), the budget now logs the full reservation table + memory snapshot at ERROR severity (engine_health, operation=kv_budget_sustained_rejection) and drops any reservation older than 10 minutes with a WARN (kv_budget_stale_reservation_dropped). Live requests hold reservations for seconds-to-minutes and release on every terminal path; pending loads release once weights are resident — an entry that outlives the TTL while NOTHING is being admitted is leaked bookkeeping, and dropping it converts a permanent reject-everything wedge into a self-healed blip. Audits are rate-limited (1/min); a single successful commit resets the streak so ordinary capacity pressure can never trigger one. Reservations now carry their creation instant (ledger entries are a struct, not bare bytes); reduceReservation keeps the original instant. Thresholds and the telemetry emitter are injectable for tests; three new unit tests pin heal-after-drop, fresh-reservation immunity, and streak-reset-on-success. * release(provider): bump to v0.7.3 (coordinator LatestProviderVersion in sync) * fix(provider): review findings — allowlist the KV-budget audit telemetry fields, run the post-bridge headroom guard on every VLM slot Two blocking review findings on the v0.7.3 hotfix, both fixed: 1. The sustained-rejection audit's fields (streak_seconds, reservations, reserved_bytes, mlx_cache_bytes, system_available_bytes, reservation_count, request_id, age_seconds) were silently stripped by the telemetry field allowlist, so the ERROR event would have reached Datadog with no diagnostic payload. Added to all three mirrors (Swift TelemetryFieldFilter, Go coordinator/api/telemetry_handlers.go, TS console-ui/src/lib/telemetry-types.ts) — operational counters and reservation ids only, no prompt/response data. 2. The post-bridge measured-headroom guard only ran when the v2 bridge built. The extraction builds the shared fused MoE cache BEFORE its parity gate, so a parity/init failure that falls back to legacy (nil bridge) has still grown resident memory — the guard now runs for any VLM slot (unregister/shutdown are no-ops on the nil-bridge path). Also folded in (non-blocking findings): shareFusedGateUpCache now returns whether a cache was actually shared so sharedFusedMoELayerCount is truthful under BENCH_NO_FUSED_GATE_UP / cache-limit opt-outs, and a comment severity mismatch (CRITICAL vs .error) in GlobalKVCacheBudget. Re-verified: swift build + full swift test (1337 tests green), live regression re-run green (one shared cache 15.07 GiB, admission=true), go build + go test ./api, tsc --noEmit. * fix(provider): KV-budget audit requires a CONTINUOUS rejection streak (review finding 1) The sustained-rejection audit armed its streak at the first failed commit and reset only on a successful one, so with sparse traffic two rejections minutes apart satisfied the 120 s threshold by wall clock alone — and the audit could then drop reservations older than 10 min that back LIVE work (a 32k-token decode at 10 tps holds its reservation ~50 min; a slow pending load can exceed 10 min), silently over-admitting against memory that work is counting on. Streak semantics now require continuity and progress-awareness: - track lastRejectionAt; a gap over 30 s (constant, documented) since the previous rejection RESTARTS the streak at the newer rejection — a truly black-holed box under coordinator routing rejects many times per window - any release() of a REAL reservation resets the streak (a drain proves the table is making progress); releases of unknown ids prove nothing and do not reset, so they cannot mask a real black hole - successful commits keep resetting it; audit rate-limit unchanged Tests: idle-gap traffic never audits (and a >TTL live reservation survives), a release-interleaved rejection storm never audits, no-op releases do not suppress the audit, and the original black-hole drop-stale + fresh-reservations-survive cases still pass unchanged. * fix(provider): net the measured fused MoE cache out of the v2 KV ceiling (review finding 2) The bridge's static KV admission ceiling — which becomes heartbeat active_token_budget_max, the coordinator's admission input — was derived by EngineV2KVSizing.engineKVBytesCapacity from weights + reserves BEFORE the VLM extraction materialized the shared fused gate+up cache. Post-fix a VLM-extracted Gemma slot still retains ONE ~8-15 GiB cache that this derivation ignored, so a 64 GB box that passes the new headroom floor over-advertised ~2x -> coordinator over-routes -> the shared gate rejects the excess post-acceptance (a mild replay of the v0.7.2 incident, held down only by coordinator-side cooldowns). Measured-subtraction approach, kept consistent across all three ledgers: - EngineV2VLMTextExtraction measures the built cache post-probe (Extraction.fusedMoECacheBytes; fusedMoECacheBytes(of:) dedups the shared arrays across the wrapper+extracted trees by buffer identity) - the slot factory nets it out of the ceiling handed to makeProductionEngine (EngineV2KVSizing.netOfEngineResidentOverhead; 0 -> noKVHeadroom -> legacy fallback), so the engine ledger AND the heartbeat grant both carry the post-cache figure - the cache is recorded on the slot (ModelSlot.engineResidentOverheadBytes, re-measured from the loaded wrapper so the parity-failure fallback path is covered) and counted like weights in both residency sums: later v2 engine ceilings (co-resident loop) and the heartbeat fleet-budget clamp (ProviderLoop+Capacity -> FleetKVContext.totalResidentWeightBytes) - the shared gate already sees the cache in live MLX active, so advertised ceiling == gate headroom Tests: unit suite pins the arithmetic, the 64 GB-profile advertised token max, gate admission of a stream sized to the advertised max (and rejection of the pre-fix excess), later-engine sizing with overhead counted, and identity-deduped measurement over real quantized SwitchGLUs. Live regression (gemma-4-26b-8bit): fused cache 15.07 GiB/30 layers; advertised ceiling 13.50 GiB == shared-gate headroom exactly (pre-fix 28.56 GiB).
…eric-path accept, 404 strikes, all-cooled 429 (#510) * fix(coordinator): capacity-cooldown follow-ups — generic-path accept, true half-open probe, 404 not-loaded strikes, all-cooled surfaces 429 Fast-follow for the four P2 review findings on #507: 1. Generic-path accept (api/consumer.go): the /v1/completions and /v1/messages handlers now record the capacity-cooldown ACCEPT at first content (both the pre-accept and post-accept commit branches), matching the chat path's commitFirstContent — a long generic stream on a busy box keeps vouching for the pair while it sheds concurrent dispatches, so the shed storm can no longer read as the zero-accepts black-hole signature. 2. True half-open re-probe (registry/capacity_cooldown.go, scheduler.go): cooldown entries are now {expiry, probeAt} structs. After expiry the routing gate opens only while no probe claim is fresh; ReserveProviderEx claims the single probe at reservation commit (under the r.mu write lock, so concurrent reservations serialize) and the gate closes for everyone else until the probe's outcome lands — accept clears, reject re-arms with doubled TTL. A stale claim (probe outcome never landed) expires after 30s so a lost probe cannot wedge the pair. No thundering herd through the post-expiry window. 3. Cold 404 'model not loaded' strikes (api/consumer.go): 404 joins 429/5xx as strike-eligible, with the zero-interleaved-accepts discriminator as the safety — the normal 404-then-load-then-serve lifecycle never trips (the accept resets), only a box that 404s forever with zero accepts does. Request-shape 404s ('model not found') never strike (not capacity vocabulary). 4. All-cooled surfaces as capacity, not absence (registry/scheduler.go): both the candidate scan and the QuickCapacityCheck preflight count a pair blocked ONLY by the capacity cooldown as a TRANSIENT capacityRejection (via an ignoreCapacityCooldown re-check of the shared gate), so an all-cooled model returns 429 + Retry-After (or queue-before-shed) instead of a structural 'no providers' 503. Tests: probe-claim lifecycle + 32-way concurrent single-probe reservation (race-clean); busy box with a held-open generic stream + 5 in-window sheds never trips (real /v1/completions flow through the failover harness); 404-forever trips while 404-then-load-then-serve never does; end-to-end all-cooled replay asserts 429 + Retry-After with zero provider dispatches and no no_provider/model_unavailable misclassification. * fix(coordinator): address PR #510 Codex P2s — sweep preserves fresh probe claims; preflight vision filter on cooldown recheck 1. The >1024 bound sweep in RecordCapacityReject deleted any entry past its expiry — including half-open entries whose fresh probeAt claim is the only thing holding the gate closed while the single probe's outcome pends (expiry is deliberately in the past in that state). A reject on ANY other pair during the probe window could sweep it, reopening the gate to a thundering herd mid-probe and dropping the pair's exponential backoff. The sweep now retains entries whose probe claim is still within capacityProbeOutcomeWindow; stale claims remain sweepable. 2. quickCapacityCheck's cooldown-only recheck counted capacity-cooled pairs that could never serve the request: the requiresVision filter runs AFTER the routing gates on the main path, so a cooled TEXT-ONLY pair on a vision request read as transient capacity — surfacing a false 'at capacity' 429 (retry forever) where the vision/model-unavailable path is the truth. The recheck now applies the same vision exclusion. Both regression tests verified to fail with the fixes reverted. * fix(coordinator): cooldown preflight recheck honors thermal + hardware-fit filters (Codex round 2) The cooldown-only capacityRejection recheck ran before the thermal-critical exclusion and skipped the absolute hardware-fit gate: 1. A capacity-cooled pair that is ALSO thermally critical now stays out of the count (same exclusion the main path applies just below). 2. A cooled pair whose model can never fit the hardware (cold slot + catalog min_ram/size vs box memory) now counts as modelTooLarge, never as transient capacity — otherwise a fleet of undersized cooled boxes reads as 'busy, retry' (429) for a model that will never fit. Regression test verified to fail with the fix reverted.
* Allow self-route keys to use owned local models
* Avoid client-side provider payload parsing
* Gate off-catalog self-route on vision capability and advertised size
The owner context that admits an off-catalog model past the routable gate
now carries through the vision gate, so an owned local VLM is selectable
for image/video requests instead of being listed but never routed. The
hardware-fit and free-memory gates fall back to the provider-advertised
SizeBytes when a model has no catalog entry, so an oversized local model
is rejected deterministically as model_too_large instead of dispatching
into a provider-side load failure.
* Filter stale challenges and hidden selections in the key picker
The provider-models proxy now checks last_challenge_verified against the
coordinator's challenge_max_age_seconds instead of accepting any non-empty
timestamp, matching the eligibility the self-route paths enforce. KeyForm
submits only the allow-list entries visible in the current route mode, so
toggling "My Machine only" cannot leak hidden public model ids into a
self-route key (or vice versa); selections are kept in state so toggling
back restores them.
* Address review: keep weight-hash gate in owner context, list/retrieve parity, preserve saved allow-lists
- providerServesRoutableModelLocked / providerServesVisionModelLocked: the
owner self-route context now lifts catalog MEMBERSHIP only — a build the
catalog tracks must still pass the weight-hash tamper tripwire
(modelServableForOwnerLocked). Off-catalog local models unaffected.
- GET /v1/models/{id} honors self-route-only keys so list and retrieve agree
(OpenAI clients that validate via retrieve-model can use listed local models).
- OwnedModels: count attested providers + attestation summary (was always 0),
deterministic metadata backfill instead of map-order first-writer-wins.
- KeyForm: only drop allow-list ids that belong to the OTHER route mode's
list — saved entries in neither list (offline machine, delisted model) are
no longer silently stripped by an unrelated edit.
- Tests: owner weight-hash gate (registry + vision helper), retrieve-model
parity + paid-key off-catalog 404 (API), preserved allow-list (vitest).
* Address Codex round-2: list/route agreement + alias naming for self-route model lists
- OwnedModels filters through modelServableForOwnerLocked so a stale-hash
catalog build an owned box still advertises is never listed only to fail
every dispatch; OwnedProviderSummary mirrors it so the self-route error
paths report model_not_loaded instead of admitting an undispatchable
request.
- selfRouteModelEntries presents owned catalog builds behind an active
public alias under the alias id (the documented name inference resolves),
hiding the concrete quant builds like the public list; retrieve-by-exact-id
keeps working for both names (includeHidden), and ?include_builds=1
re-exposes them in the list (public-path parity).
- Prefer-mode + off-catalog intentionally still 404s: the paid fallback leg
can never succeed for a model only the owner's box serves, and prefer takes
an upfront paid reservation + public-capacity shed — exclusive self-route
is the correct tool there (see PR discussion).
* Address Codex round-3: picker/allow-list agreement + restricted-key privacy
- New GET /v1/me/self-route-models (Privy-authed): the alias-aware owned
live-model view shared with /v1/models for self-route keys. The console
picker proxy is now a thin passthrough to it, so the ids saved into a key's
allow-list are exactly the ids clients will list and request (aliases for
catalog builds, raw ids for off-catalog) — deriving the list client-side
from raw provider advertisements produced allow-lists holding hidden build
ids that keyModelAllowed (checked pre-alias-resolution) then rejected. Also
collapses the eligibility-filter drift (proxy re-derived freshness rules).
- Self-route /v1/models list + retrieve now respect the key's allow-list:
owned live models are private inventory; a restricted key handed out for
one local model can no longer enumerate the account's other model ids.
- KeyForm free-text fallback applies the same other-mode exclusion as the
picker: toggling 'My Machine only' with no machine models available no
longer submits the key's stale public allow-list from the text field.
* Remove now-unused providerAdvertisesModelLocked (golangci unused)
* Merge master (#510/#512); adapt new cooldown-recheck vision call site to 3-arg signature (public context)
* Address Codex round-4: header self-route discoverability + trait-aware owner preflight
- /v1/models list + retrieve now follow the request's RESOLVED self-route
policy (per-key flag OR X-Darkbloom-Route: self), matching inference: a
header-based self-route client discovers the same owned-model ids the
inference path accepts. Headerless ordinary keys keep the public view.
- OwnedProviderSummary takes the request's traits + media shape and mirrors
the dispatch gates (tools capability floor / template render, vision-capable
build): a tool call to a below-floor owned box or media to a text-only build
now fails fast with the real cause (503 model_capability_unsupported)
instead of queueing 120s into machine_busy. Queue-cleanup call site keeps
the base-shape question (unchanged behavior).
* Exclude render-broken builds from OwnedModels (list/route agreement for the template gate)
---------
Co-authored-by: Gajesh Naik <26431906+Gajesh2007@users.noreply.github.com>
…#511) * chore(trust): remove the dormant ACME device-attest-01 leg end-to-end The ACME leg was never wired end-to-end: the Swift provider never presented the device cert, and no ingress ever did mTLS client auth or forwarded the nginx-era X-Ssl-Client-* headers the coordinator read. Zero ACME verifications were ever observed in prod — hardware trust is earned exclusively via MDM SecurityInfo. Meanwhile the leg cost real money and trust: a permanent 'ACME device-attest-01' red X on every provider dashboard (DAR-394), step-ca in the prod container + persistent CA state slated for the GCP migration, an extra profile payload doing SE keygen + Apple attestation on every enrollee, and 'MDM/ACME' doctor copy that taught operators the red X gated earning. Removed: - enrollment profile payload 3 (com.apple.security.acme) — profile is now SCEP+MDM - coordinator: acme_verify.go, applyACMETrust + pending/stash/retry machinery, step-ca config (EIGENINFERENCE_STEP_CA_*), ACMEVerified through registry/ store/persistence, acme.* metrics - deploy: step-ca init/start in the container, /acme/* Caddy routes (prod+dev), acme-device.tpl, deploy-acme.sh, enroll-with-acme.mobileconfig - console-ui: the ACME red-X line in the Trust & Attestation panel, 'ACME bound' stats label, acme_verified types/fixtures; stats routability fallback no longer requires certificate/MDA proofs (they never gated routing) - provider-swift: acme proof in doctor output/score, 'MDM/ACME' trust copy (catalog still matches the old coordinator reason string for skew) Wire compat: acme_verified is still emitted (hardcoded false, deprecated) by /v1/providers/attestation, /v1/me/providers and /v1/stats because shipped provider builds decode it as a required field. The trust reason string is now 'SE attestation verified, awaiting MDM verification'. Also fixes 4 stale console tests that predated #436's mda_missing info downgrade (they used mda_verified=false as a 'degrading' fixture). Docs: trust-reliability, enrollment, threat-model (A-012/T-022/T-040), identity-binding, migration/state-export runbooks + enrollment-flow diagram regenerated. Closes DAR-394 at the root: the false-positive indicator no longer exists. If a throttle-immune trust leg is ever wanted again, do it as an in-band cert proof with fail-closed SIP/SecureBoot OID checks (see docs/provider-trust-reliability.md removal note). * fix(lint): drop test helper orphaned by the ACME test removal; close SEC-005 - rawP256PublicKeyB64ForTest was only used by the deleted TestApplyACMETrust* tests — golangci-lint 'unused' failed CI on it. - threat-model.yaml: move SEC-005 from open_findings to a resolved_findings record per the threat-model-review bot — the header-parsing surface is now deleted at the code level, not just documented as resolved.
…dates (#514) Bumps the npm_and_yarn group with 7 updates in the /console-ui directory: | Package | From | To | | --- | --- | --- | | [js-yaml](https://github.com/nodeca/js-yaml) | `4.1.1` | `4.3.0` | | [vite](https://github.com/vitejs/vite/tree/HEAD/packages/vite) | `8.0.8` | `8.1.3` | | [axios](https://github.com/axios/axios) | `1.15.0` | `1.18.1` | | [form-data](https://github.com/form-data/form-data) | `4.0.5` | `4.0.6` | | [hono](https://github.com/honojs/hono) | `4.12.12` | `4.12.27` | | [js-cookie](https://github.com/js-cookie/js-cookie) | `3.0.5` | `3.0.8` | | [undici](https://github.com/nodejs/undici) | `7.24.7` | `7.28.0` | Updates `js-yaml` from 4.1.1 to 4.3.0 - [Changelog](https://github.com/nodeca/js-yaml/blob/master/CHANGELOG.md) - [Commits](nodeca/js-yaml@4.1.1...4.3.0) Updates `vite` from 8.0.8 to 8.1.3 - [Release notes](https://github.com/vitejs/vite/releases) - [Changelog](https://github.com/vitejs/vite/blob/main/packages/vite/CHANGELOG.md) - [Commits](https://github.com/vitejs/vite/commits/v8.1.3/packages/vite) Updates `axios` from 1.15.0 to 1.18.1 - [Release notes](https://github.com/axios/axios/releases) - [Changelog](https://github.com/axios/axios/blob/v1.x/CHANGELOG.md) - [Commits](axios/axios@v1.15.0...v1.18.1) Updates `follow-redirects` from 1.15.11 to 1.16.0 - [Release notes](https://github.com/follow-redirects/follow-redirects/releases) - [Commits](follow-redirects/follow-redirects@v1.15.11...v1.16.0) Updates `form-data` from 4.0.5 to 4.0.6 - [Changelog](https://github.com/form-data/form-data/blob/master/CHANGELOG.md) - [Commits](form-data/form-data@v4.0.5...v4.0.6) Updates `hono` from 4.12.12 to 4.12.27 - [Release notes](https://github.com/honojs/hono/releases) - [Commits](honojs/hono@v4.12.12...v4.12.27) Updates `js-cookie` from 3.0.5 to 3.0.8 - [Release notes](https://github.com/js-cookie/js-cookie/releases) - [Commits](js-cookie/js-cookie@v3.0.5...v3.0.8) Updates `undici` from 7.24.7 to 7.28.0 - [Release notes](https://github.com/nodejs/undici/releases) - [Commits](nodejs/undici@v7.24.7...v7.28.0) --- updated-dependencies: - dependency-name: js-yaml dependency-version: 4.3.0 dependency-type: indirect dependency-group: npm_and_yarn - dependency-name: vite dependency-version: 8.1.3 dependency-type: indirect dependency-group: npm_and_yarn - dependency-name: axios dependency-version: 1.18.1 dependency-type: indirect dependency-group: npm_and_yarn - dependency-name: follow-redirects dependency-version: 1.16.0 dependency-type: indirect dependency-group: npm_and_yarn - dependency-name: form-data dependency-version: 4.0.6 dependency-type: indirect dependency-group: npm_and_yarn - dependency-name: hono dependency-version: 4.12.27 dependency-type: indirect dependency-group: npm_and_yarn - dependency-name: js-cookie dependency-version: 3.0.8 dependency-type: indirect dependency-group: npm_and_yarn - dependency-name: undici dependency-version: 7.28.0 dependency-type: indirect dependency-group: npm_and_yarn ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps the go_modules group with 1 update in the / directory: [golang.org/x/net](https://github.com/golang/net). Updates `golang.org/x/net` from 0.51.0 to 0.55.0 - [Commits](golang/net@v0.51.0...v0.55.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-version: 0.55.0 dependency-type: indirect dependency-group: go_modules ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…nputs, realistic assumptions (console + landing) (#502) The old calculator buried the estimate under five input groups (Mac type, chip, memory, model multi-select, electricity) and priced a saturated best case (80% util × 4× batch). Redesign both the console /earn page and the landing-page mirror around setting honest expectations: - Results-first hero: annual floor→estimate range with monthly beneath, base-reward + best-model chips, and a projection disclaimer - Inputs reduced to chip + unified memory; electricity baked in at $0.15/kWh - Read-only "What your Mac can run" list; estimate always uses best earner - Base rewards + "How we estimate this" become collapsed accordions with a readable six-step derivation (was a dense mono formula block) - Assumptions recalibrated to healthy demand: 60% utilization, 2.5× avg concurrency (was 80% × 4×) - No-fit machines get a "notify me when smaller models launch" CTA Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Gajesh Naik <26431906+Gajesh2007@users.noreply.github.com>
…; drain logic re-ported in follow-up commits) Co-authored-by: gajcodex <gajcodex@users.noreply.github.com>
… split - ProviderLoop drain logic now lives in a dedicated ProviderLoop+Shutdown.swift (memoized startShutdown + runShutdownSequence + cancelBackgroundWorkAndPreloads + shutdownShouldAbortLoad) instead of a monolithic run() epilogue. - CoordinatorClient draining (beginDraining / flushOutbound / drain-cancel and drain-disconnect routing) ported onto the NWConnection transport across the CoordinatorClient split files; outbound write confirmation now rides the contentProcessed completion (OutboundRouter.markWritten). - handleInferenceRequest: authoritative shutdown re-check + detached cold-load task with cancel mapping (inflightModelLoadTasks). - ensureModelLoaded / refreshWeightHash abort gates use shutdownShouldAbortLoad so accepted requests finish cold loads during a drain. - CLI: stop/restart/start drain via launchd-reported PIDs before manipulating the job (ProviderDrainHelper); stop message preserves master's auto-start wording. - Deleted the unused onActiveRequests callback and re-privatized readPID. Co-authored-by: gajcodex <gajcodex@users.noreply.github.com>
- ensureModelLoaded: a cancelled loader no longer poisons healthy co-waiters of the same model (retry instead of propagating the loader's CancellationError); shutdown/self-cancel paths still terminate. - waitForInflightDrain: sleep on an unstructured task when polling through a cancelled context so a shutdown mid-update-drain can't hot-spin the actor. - commitStagedUpdateBundle: abort the staged-update commit once shutdown has begun so a concurrent 'darkbloom stop' can never trigger restartAfterUpdate. - handleDrainDisconnect: finish the OutboundRouter so flushOutbound exits immediately after a mid-drain socket drop instead of burning its timeout. - Non-drain shutdown branch: restore the bounded in-flight wait so local endpoint streams aren't unloaded mid-stream on coordinator disconnect. - CLI drain timeout + launchd ExitTimeOut raised to 660s (drain window + 60s teardown margin) so a full-window drain isn't SIGKILLed mid-teardown. - Tests: OutboundRouter pending-write accounting, DaemonState inflight_request_count round-trip + legacy decode, and a MockCoordinator integration test pinning the 503 reject + drain cancel routing. Co-authored-by: gajcodex <gajcodex@users.noreply.github.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rebases #392 onto current master and re-ports the graceful-drain feature onto the architecture master moved to while the PR was open, then fixes the edge cases a two-reviewer pass found, and adds the drain tests that were missing.
Why the rebase was invasive
#392 was written against a monolithic
ProviderLoop.swift(3,600 lines) andCoordinatorClient.swift(1,300 lines) with aURLSessionWebSocketTasktransport. Master has since:ProviderLoop+*.swift/CoordinatorClient+*.swiftextension modules (refactor: tech-debt cleanup + partial modularization (coordinator + provider) #444/refactor: split coordinator + provider god-files into focused modules (stacked on #444) #445),NWConnectionwith non-blocking sends and a direct inference-chunk fast path that bypasses the outboundAsyncStream(perf(provider): migrate WS transport to NWConnection (52→130 TPS) #479/perf(provider): bypass AsyncStream for inference chunks — direct-to-kernel send (370x faster) #480),A textual merge was unresolvable, so this branch merges master (taking master's file structure) and re-ports every drain behavior into the right module. All 15 hard-won edge cases from #392's review rounds were verified present by two independent review agents.
What changed relative to #392
Structure (behavior-preserving)
ProviderLoop+Shutdown.swift(memoizedstartShutdown→runShutdownSequence→cancelBackgroundWorkAndPreloads+shutdownShouldAbortLoad) instead of a 120-linerun()epilogue;ProviderLoop+Serve.swiftkeeps only the two funnel points.beginDraining,flushOutbound, drain-cancel/disconnect routing) is ported across theCoordinatorClientsplit files. Outbound write confirmation now rides theNWConnection.contentProcessedcompletion (sendTextFrame(onProcessed:)→OutboundRouter.markWritten) instead of the removedawait ws.sendreturn. Chunk ordering vs the terminal is preserved bySendHandle.send's existing flush barrier.Garbage removed
onActiveRequestscallback onstopProcessGracefully(dead since the drain messaging moved to the daemon state file).ProcessLifecycle.readPIDre-privatized (its external caller was removed in a feat(provider): graceful drain for darkbloom stop/start/restart #392 iteration; the> 0validation is kept).AGENTS.md/CLAUDE.mdedits dropped — master already carries a stronger version of the same PR-diagram rule.pid > 0filter and a stale "reads the PID from DaemonStateFile" doc comment inProviderDrainHelper.Review fixes (new in this branch)
ensureModelLoaded's waiter branch retries the load instead of propagating the loader'sCancellationError(a local request or preload co-waiting on the model a cancelled request was cold-loading would previously fail silently).waitForInflightDrainsleeps on an unstructured task when polling through a cancelled context, so adarkbloom stoplanding mid-update-drain can't hot-spin the actor for up to 120 s.commitStagedUpdateBundleaborts once shutdown has begun, so a concurrent stop can never end inrestartAfterUpdaterelaunching the service the user just stopped.handleDrainDisconnectfinishes theOutboundRouter, soflushOutboundexits immediately after a mid-drain socket drop instead of burning its 5 s timeout.ExitTimeOutraised 600 s → 660 s: the daemon's own drain window is 600 s, and the outer killers previously had zero margin for the post-drain teardown (outbound flush, preload waits, model unloads) — a full-window drain would be SIGKILLed mid-teardown and reported as force-killed.Tests (new)
OutboundRouterpending-write accounting (yield/markWritten/underflow/activate-reset/finish-reset).DaemonState.inflightRequestCountround-trip + legacy (pre-field) decode.beginDraining→ new inference request is 503-rejected at the receive path with the socket kept open → a subsequentcancelreaches the drain handler.ProcessLifecycleTests(graceful stop / force-kill / nonpositive PIDs) and theExitTimeOutplist assertion carried over (updated to 660).Before / After
Behavior ("Before" = #392 as-is on its stale base):
flowchart LR subgraph Before["Before (PR #392, conflicts with master)"] A1[darkbloom stop / restart / start] --> B1[textual merge conflict<br/>in 6 files - unmergeable] C1[cancel of a cold-loading request] --> D1[co-waiters of the same model<br/>poisoned with CancellationError] E1[stop during auto-update drain] --> F1[hot-spin on the actor +<br/>commit + relaunch after stop] G1[drain uses most of 600s window] --> H1[CLI/launchd SIGKILL<br/>mid-teardown - force-kill warning] end subgraph After["After (this branch)"] A2[darkbloom stop / restart / start] --> B2[clean merge on master:<br/>SIGTERM to launchd PIDs, daemon drains,<br/>503 reroutes, tail flushed, then bootout] C2[cancel of a cold-loading request] --> D2[co-waiters retry the load;<br/>only the cancelled request aborts] E2[stop during auto-update drain] --> F2[250ms poll cadence kept;<br/>staged commit aborted, no relaunch] G2[drain uses most of 600s window] --> H2[660s outer window:<br/>60s teardown margin, clean exit] endCode (where the drain logic lives):
flowchart LR subgraph BeforeCode["Before (#392 file layout)"] R1["ProviderLoop.swift run() epilogue<br/>(monolithic, 3.6k-line file)"] --> S1[drain + teardown inline] T1["CoordinatorClient.swift<br/>(URLSessionWebSocketTask)"] --> U1["pendingWrites decremented<br/>after await ws.send"] end subgraph AfterCode["After (master's modular split)"] R2["ProviderLoop+Serve.swift<br/>onCancel + fall-through"] --> S2["ProviderLoop+Shutdown.swift<br/>startShutdown (memoized)<br/>runShutdownSequence<br/>cancelBackgroundWorkAndPreloads<br/>shutdownShouldAbortLoad"] S2 --> V2["+Cancellation / +Preload /<br/>+ModelLoading / +InferenceHandler<br/>(load-task cancel, waiter sparing,<br/>per-model abort gates)"] T2["CoordinatorClient.swift drain state<br/>+Inbound: 503 reject, cancel routing<br/>+Connection: drain-disconnect break"] --> U2["OutboundRouter.markWritten via<br/>NWConnection .contentProcessed;<br/>flushOutbound before shutdown()"] endTesting
swiftc -parsepasses on every changed file (Linux box — fullswift build/swift testneed macOS + MLX; CI runs them).Merging this into #392 makes that PR mergeable against master with the improvements above.
Need help on this PR? Tag
/codesmithwith what you need. Autofix is disabled.