feat(registry): v2 version floors + tripwire, prefill-honest TTFT (absorbs #453), satisficing utilization band [stacked on per-model caps]#526
Open
Gajesh2007 wants to merge 28 commits into
Conversation
…r 6) The quality-concurrency cap consumed resolvedDecodeTPS(p) — the provider- LEVEL registration benchmark — for every model on the box. Mixed boxes benchmarked on gpt-oss (58-93 tok/s) therefore granted gemma caps of 12-23 where gemma's real solo rate (10-18 tok/s) warrants 1-2, and the coordinator marched 8-11 concurrent gemma requests onto exactly the boxes that collapse past batch 3 (2026-07-06 postmortem §3.2). The cap now resolves each model's OWN static solo rate: - Solo-gated sample store (solo_tps.go): RecordSolo/SoloMedian beside the existing 50-sample ring. Heartbeat ingest records a slot EWMA as solo ONLY when the whole box has ≤1 running+waiting request across ALL slots (the allowance is the sample-generating request itself) — mixed-box samples stay honest. The existing Record/Median (fleetMedianTPS → TTFT estimation) keeps its deliberately load-inclusive semantics, untouched. - Resolver resolvedSoloModelTPSLocked, fallback chain: per-(model,chip) solo median at ≥ EIGENINFERENCE_QUALITY_CAP_SOLO_MIN_SAMPLES (default 5) → cross-chip pooled solo median → EIGENINFERENCE_MODEL_SOLO_TPS_SEED (model=tok/s CSV; cold-start answer, the registry is restart-wiped) → resolvedDecodeTPS(p) unchanged. Rates stay STATIC (never under-load EWMA) so the cap cannot feedback-collapse; solo gating preserves that. - Kill switch EIGENINFERENCE_QUALITY_CAP_PER_MODEL_TPS (default true); false restores resolvedDecodeTPS(p) at every wired site exactly. - The DecodeTPS<=0 dedicated-only guard now applies only to the provider- level fallback: a per-model rate (solo median/seed) is model-specific by construction and caps even without a registration benchmark. resolvedDecodeTPS call sites, each decided deliberately: SWITCHED to the solo resolver (quality-cap inputs, all through the single entry hasConcurrencyHeadroomForModelCapResolvedLocked): - scheduler.go routing-snapshot headroom (snap.hasHeadroom) - scheduler.go quickCapacityCheck preflight concurrency gate - scheduler.go providerCanAdmitLockedEx final admit re-check (already Resolved; now per-model internally) - registry.go ModelCapacitySnapshot public-capacity headroom (already Resolved; now per-model internally) - warm_pool_controller.go warmSaturated gate (already Resolved) - warm_pool_controller.go warm-target decode samples: soloDecodeTPS now comes from the solo resolver so warm targets and admission caps cannot disagree; the observed-EWMA chain moved to a new serviceDecodeTPS that feeds only the E[S] service-time estimate (load-inclusive on purpose). KEPT provider-level (TTFT-estimation / cost-model / scoring semantics): - scheduler.go snap.decodeTPS (TTFT + cost estimates, effectiveDecodeTPS) - scheduler.go resolvedModelTPSLocked base (observed-EWMA cost chain) - scheduler.go resolvedPrefillTPS fallback (decode × ratio-12) - registry.go ModelCapacitySnapshot effectiveTPS (capacity-feed rate) - utilization.go FleetCapacity.DecodeTPS aggregate - warm_pool_controller.go cold-candidate score heuristic The unused explicit-rate hasConcurrencyHeadroomForModelCapLocked was deleted (golangci-lint unused); effectiveMaxConcurrencyForModelLocked stays as the explicit-rate wrapper. Regression tests (fail with the resolver mutation reverted; verified): the exact postmortem mixed box (benchmark 93, gemma solo 14, floor 15 → gemma cap 2, gpt-oss cap 23; kill switch off reproduces the old 23), solo ingest gating through real heartbeats, fallback-chain order, min-sample floor, seed parsing (malformed entries skipped), restart cold-start via seed, warm-pool solo/service rate split. New envs (documented in docs/operations/coordinator-deploy.md): EIGENINFERENCE_QUALITY_CAP_PER_MODEL_TPS, EIGENINFERENCE_QUALITY_CAP_SOLO_MIN_SAMPLES, EIGENINFERENCE_MODEL_SOLO_TPS_SEED.
… the shared budget freeMemoryAdmits admitted against the per-slot ActiveTokenBudgetMax with only SAME-MODEL coordinator pending counted. But a slot max is not a private budget: every slot on a box reports its own committed tokens plus the SAME shared KV headroom (see providerTokenBudget). Two co-resident models could therefore double-spend the one shared pool inside the ~30s heartbeat gap — a burst admitted against model A's slot was invisible to model B's stale slot max until the next heartbeat. Fix (kept minimal around freeMemoryAdmits for the pending budget-clamp branch to merge next to): - pooled_admission.go: providerPooledTokenBudget reconstructs the whole-box pool from the slots via the proven providerTokenBudget (committed sums across slots; shared headroom counted exactly once), plus an all-slots committed baseline that mirrors committedTokenBudget (max(used+queued, maxTokensPotential) per slot) so heartbeat-reflected requests are not double-counted against coordinator pending. - routingSnapshot gains pendingMaxTokensAllModels (the pending-token accumulator WITHOUT the model filter) and the pooled budget, populated by both snapshot builders (dispatch + preflight). - Budget admission is now: fits the per-slot budget (unchanged — the slot max still encodes the model's own context limits) AND fits the pool with every model's coordinator-pending tokens charged. Single-model providers are arithmetically unchanged (the pool reduces to the slot's own budget; boundary pinned byte-for-byte in TestFreeMemoryAdmitsSingleModelUnchanged, including the maxTokensPotential-dominates case). Legacy providers without token budgets never reach the pooled check. Inert under DEDICATED partitioning (no co-resident dispatch); live before the open-pool flip. Regression tests (fail with pooledBudgetAdmits mutated out; verified): burst model A to the whole pool through the REAL reservation path, then a model B request within the heartbeat gap is rejected; co-resident request WITHIN pool headroom still admits; 6th same-model request past the slot budget still rejects; pure-function double-spend + boundary tables; pooled-budget reconstruction arithmetic (shared headroom once, negative floors, budgetless slots ignored).
…g-prompt ttft_429 over-shed
No provider reports observed_prefill_tps (0/313 warm slots), so resolvePrefillTPS
falls back to sqrt(bandwidth)x12 = ~280 tok/s — ~23x below the measured real prefill
(empirical idle p50 6,523 tok/s, p90 17,707, recovered by inverting the router's
own stored ttft_ms). That pessimistic estimate makes the TTFT gate reject every
prompt above ~550 tokens (the live ~41% ttft_429 over-shed).
- Add an env-gated, data-derived prefill fallback (default 6500 tok/s = measured
p50) that replaces the sqrt(bandwidth)xratio estimate when no measurement
exists. Staged shadow->enforce via EIGENINFERENCE_PREFILL_FALLBACK_MODE
(default off, behavior-neutral). prefill_fallback.go is the self-contained config
+ helper module.
- Raise maxPrefillTPS 5000->20000 (above measured p90) so a correctly-measured
value from a fixed provider is neither zeroed at ingest nor capped in routing.
Tunable via EIGENINFERENCE_MAX_PREFILL_TPS for instant rollback.
- shadow mode emits routing.prefill_fallback{would_admit|would_shed} from the
preflight so the projected ttft_429 recovery is measured before enforcing.
Cost-ranking is untouched (still on the static estimate); only the TTFT admission
gate moves. Orthogonal to the durable provider-side measurement fix (separate PR):
once providers report a real observed_prefill_tps, that measured value wins and
this fallback is never consulted.
ABSORBED from open PR #453 (branch prefill-fallback-recalibration) into the
deploy-batch-B stack: the ceiling raise is a hard prerequisite for the
per-(model,chip) observed-prefill medians landing on top of this commit — the
old 5000 tok/s ingest clamp would zero the majority of REAL v0.7.5
observed_prefill_tps reports (measured p50 6,523 > 5000) before they could
ever feed a median. PR #453 should be closed in favor of this stack.
…re + per-model version routing floors The v0.7.5 release deletes the legacy scheduler: a >=0.7.5 box truthfully reports its engine cap (<=4) and can never silently legacy-serve at cap 24 again. But the fleet migrates over hours-to-days, and the 2026-07-06 gemma postmortem showed what one silently-degraded box does to a whole model pool. Two coordinator-side defenses, both inert until their envs are set: 1. Version-keyed heartbeat sanity clamp (v2_capacity_clamp.go). Providers at/above EIGENINFERENCE_V2_VERSION_FLOOR heartbeating a chat-slot max_concurrency above EIGENINFERENCE_V2_MAX_CONCURRENCY_CEILING (default 4) are clamped down to the ceiling, logged at ERROR, and counted on a new provider.v2_concurrency_tripwire Datadog counter (registry hook -> api.emitV2ClampTripwire, the SetHardUntrustHook pattern). On a >=floor box that report means the silent-legacy-fallback bug RESURFACED — this is a permanent audit of the release's fail-loud promise, not a rollout metric. The v2 pass runs BEFORE the general clampBackendCapacity pass so the tripwire records the provider's original report; below-floor providers (and empty-version providers — below any floor, trait-floor convention) keep today's clamp-24 behavior byte-for-byte. 2. Per-model provider-version routing floors (model_version_floors.go). EIGENINFERENCE_MODEL_VERSION_FLOORS="gemma-4=0.7.5" fences a model whose resolved build id contains the pattern (case-insensitive substring, the dedicatedPatternForLocked semantics) to providers at/above the paired version. Enforced in the single routing gate (providerPassesRoutingGatesLockedEx, beside the trait version floors — so dispatch, the capacity preflight, and the final admit re-check all agree) AND in the warm-pool candidate gate (new warmColdBelowVersionFloor disqualification reason: never warm a box we won't route to, and the reason tally says so on the dashboard). Empty-version providers fail every floor. Applied to self-route too: a below-floor binary would MIS-SERVE the model, which is just as true on the owner's own machine. Malformed CSV entries degrade to no-floor, never to a wrong floor. Regression tests (verified to fail with the gate check neutralized): >=floor 24-report clamped to 4 + tripwire with the original report (incl. a 30-report bypassing the legacy 24 stage), below-floor/empty-version/ floor-unset boxes byte-identical to today, floored model excluded from below-floor + empty-version boxes in routing AND preflight AND warm-pool candidacy while unfloored models on the same box keep routing, parse tables, set-then-clear restores defaults. New envs documented in docs/operations/coordinator-deploy.md: EIGENINFERENCE_V2_VERSION_FLOOR, EIGENINFERENCE_V2_MAX_CONCURRENCY_CEILING, EIGENINFERENCE_MODEL_VERSION_FLOORS.
…ill medians
The TTFT estimate derives an unmeasured provider's prefill rate from the
static EIGENINFERENCE_PREFILL_DECODE_RATIO chain (sqrt-bandwidth × 12 ≈ 280
tok/s) — ~23x below the measured fleet reality (p50 6,523 tok/s, p90 17,707;
docs/reports/2026-06-22-live-prefill-tps-check.md). gpt-oss's p90 prompt is
13,711 tokens, so its predicted TTFT is ~38s against the 5s deadline and
18.2k requests/day die on ttft_too_slow MISpredictions while the fleet idles.
This makes the estimate measurement-driven: a per-(model, chip family) median
ring of the observed_prefill_tps EWMAs v0.7.5 providers report in heartbeats
(prefill_tps.go, the third TPSRegistry ring beside the load-inclusive and
solo decode rings — same 50-sample FIFO shape). The TTFT prefill resolution
(prefillTPSForSnapshot) becomes, most- to least-specific:
own slot measurement → trusted per-(model,chip) median
→ (enforce mode) #453 fallback anchor → static decode×ratio chain
with the median trusted at >= EIGENINFERENCE_TTFT_PREFILL_MIN_SAMPLES
(default 5) samples, applied in BOTH the dispatch snapshot and the capacity
preflight (shed/Retry-After agree with routing), and the #512 online TTFT
calibrator kept downstream as the corrector. No cross-chip pooled fallback on
purpose: prefill is compute-bound and spreads 3-4x across chip tiers, so a
cross-chip transfer could over-admit a slow tier at the hard gate — with too
few same-chip samples the estimate falls back to the anchor/ratio instead.
Relationship to open PR #453 (prefill-fallback-recalibration): ABSORBED, not
superseded — cherry-picked one commit below as the no-measurement bridge.
Its maxPrefillTPS raise (5000 → 20000) is a hard prerequisite here: the old
ingest clamp would zero the MAJORITY of real v0.7.5 prefill reports (p50
6,523 > 5,000) before they could feed a median
(TestPrefillIngestZeroedUnderOldCeiling pins this). Its flat 6500 anchor
(staged off/shadow/enforce) remains the fallback layer for (model, chip)
pairs without enough samples; the medians supersede the anchor wherever real
samples exist, and both retire naturally per pair as boxes report their own
measurement (layer 1).
Ingest gating decision (documented in prefill_tps.go): NOT solo-gated,
unlike the decode solo ring — prefill is per-request compute-bound work, not
the shared-decode-loop rate the solo gate protects. The known garbage mode
from #453's analysis (prefix-cache-hit EWMA overflow) is zeroed upstream by
clampBackendCapacity; residual noise is absorbed by median rank-robustness,
the min-sample floor, and the downstream calibrator.
Kill switch EIGENINFERENCE_TTFT_PREFILL_MEDIANS (default true, live-read):
false removes the median layer and restores the pure ratio path exactly
(byte-identical with the #453 mode also off).
Regression tests (verified to fail with the median layer neutralized): the
production overshoot — 13.7k-token prompt at real prefill 6,500 admitted
(TTFT ~2.1s), ratio-derived ~38s rejected — through the REAL
ReserveProviderEx path incl. kill switch and min-sample floor; preflight
bestTTFT consistency; resolution-order table (own observation > median >
anchor > static, median may lower, cap applies); heartbeat ingest (contended
box records, out-of-range zeroed, old-ceiling starvation); ring
median/FIFO/invalid-sample tables; cross-box convergence.
New envs documented in docs/operations/coordinator-deploy.md:
EIGENINFERENCE_TTFT_PREFILL_MEDIANS, EIGENINFERENCE_TTFT_PREFILL_MIN_SAMPLES
(plus rows for the absorbed #453 trio).
…d off the fastest tier (dormant) Candidate selection is strictly cheapest-cost with near-tie randomization, so the fastest boxes win everything: top-10% of boxes take 79% of traffic, the median gpt-oss box serves 9 req/day, ~70% of online boxes serve nothing in a given hour, and 41 online 24GB boxes advertising gpt-oss served 0 — despite solo decode (19 tok/s) that clears the 15 tok/s quality floor. Consumers never experience "cheapest cost"; they experience "met the SLO or not". Behind EIGENINFERENCE_SATISFICING_BAND (bool, default FALSE — fully dormant; flag-off selection is byte-identical and pinned by test): - Band membership reuses the EXACT predicates admission already computes, no new estimators: the candidate's calibrated breakdown.TTFTMs (the value the hard MaxTTFTMs ceiling gates on) must clear the request deadline minus EIGENINFERENCE_SATISFICING_TTFT_MARGIN_MS (default 1000), and projectedPerRequestDecodeTPS (the decode-floor preference's projection) must clear pr.MinDecodeTPS. No-deadline/no-floor requests satisfy the respective criterion vacuously; a deadline request with no reliable TTFT estimate (no BackendCapacity) is not a band member. - Selection within the band is weighted random, weight = 1/(1+recentServes): recentServes is a registry-local, in-memory, exponentially-decaying per-provider serve counter (half-life 5 min ≈ a sliding ~10-min window; chosen over bucketed windows for O(1) state and exact decay testability via an injectable clock). Fed on EVERY successful reservation regardless of the flag so weights are warm at flip-time; restart-wiped like the TPS registries (converges in minutes); size-bounded by opportunistic pruning of decayed entries. - Cache affinity keeps its pin WITHIN the band (a pinned consumer's turns are serial — honoring the prefix-cache pin costs no spread; evicting it would regress TTFT p90, the rollout guard metric). - Candidates outside the band are selected ONLY when the band is empty — and then via exactly today's cheapest-cost path. Speculative/backup dispatch, queueing, retries, admission gates: untouched. The band only reorders WHICH eligible candidate is picked first, inside selectBestCandidateScanLocked. Tests (verified to fail with the selection hook neutralized): flag-off pinned-order table (50 trials, deterministic cheapest); flag-on statistical spread through the REAL ReserveProviderEx path (300 trials — every band member selected >=1, cheapest not always); empty-band fallback (all candidates pass the 3s hard gate but miss the margin -> deterministic cheapest); weight shift with recentServes (~9 recent serves -> ~1/10 the picks of never-served peers, no starvation); half-life decay with injected clock; counter fed on the real path with the flag off; affinity pin honored within the band. New envs documented in docs/operations/coordinator-deploy.md: EIGENINFERENCE_SATISFICING_BAND, EIGENINFERENCE_SATISFICING_TTFT_MARGIN_MS.
…ummary, calibrated shadow TTFT Two substantive fixes from the quality-gate review of this stack, plus test hermeticity pins: - OwnedProviderSummary now applies providerBelowModelVersionFloorLocked, the same per-model version floor the dispatch gate enforces (self-route included). Without it a below-floor OWNED box read as "serves model" in the self-route preflight, queued for up to 120s, and died as machine_busy instead of failing fast with the real cause. Regression test (TestModelVersionFloorOwnedSummaryConsistency) fails without the check and pins that only the floored model is fenced. - estimatedRecalibratedTTFTFromSnapshot (the prefill-fallback SHADOW estimate) now applies the #512 online calibration exactly like the live preflight path does. An enforce flip gates on calibrate(recalibrated-raw), so the shadow would_admit/would_shed split must compare calibrated-vs-calibrated — with a learned ratio < 1 (the norm on the over-predicting legacy chain) the uncalibrated shadow under-reported the projected ttft_429 recovery. Observability-only; identity when the calibrator is cold, so all existing tests hold unchanged. - Tests that depend on env DEFAULTS (band off, medians on, min-samples 5) now pin them via t.Setenv so ambient operator environments cannot flip test behavior; stale comment on V2ConcurrencyClampConfig corrected. Reviewer note kept as-is (documented, not changed): with EIGENINFERENCE_LONG_PROMPT_TOKENS > 0 (default 0 = off), a trusted prefill median or enforce-mode anchor also shifts long-prompt COST ranking via the longPromptPenalty's resolvePrefillTPS input — intentional per that knob's design ("the bias follows real measured prefill"), but flag interactions should be noted when enabling both.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
|
Found 1 test failure on Blacksmith runners: Failure
|
Gajesh2007
marked this pull request as ready for review
July 8, 2026 06:28
ethenotethan
requested changes
Jul 8, 2026
ethenotethan
left a comment
ethenotethan
left a comment
Contributor
There was a problem hiding this comment.
Automated Code Review — Layr-Labs/d-inference#
Verdict: REQUEST_CHANGES
Security — 1 finding(s) (1 blocking)
- 🟡 [MEDIUM]
coordinator/registry/satisficing_band.go:138— Using math/rand for weighted selection in production routing- Suggestion: Use crypto/rand for security-sensitive randomness or document that this is intentionally using deterministic PRNG for reproducible testing
Performance — 3 finding(s) (2 blocking)
- 🟡 [MEDIUM]
coordinator/registry/prefill_fallback.go:183-199— Unbounded goroutines in recalibrated TTFT estimation without context cancellation- Suggestion: Add context cancellation or use errgroup to bound concurrent TTFT calculations in QuickCapacityCheckRecalibratedTTFT
- 🔵 [INFO]
coordinator/registry/satisficing_band.go:135-159— O(n) weighted random selection could be O(log n) with alias method for large bands- Suggestion: Consider using alias method or binary search for weighted selection when band size exceeds ~100 candidates
- 🟡 [MEDIUM]
coordinator/registry/model_version_floors.go:126-133— Repeated string lowercasing and contains checks in hot routing path- Suggestion: Cache lowercased model strings or pre-compute pattern matches to avoid repeated string operations per request
Type_diligence — ✅ No issues found
Additive_complexity — ✅ No issues found
4 finding(s) total, 3 blocking. Verdict: REQUEST_CHANGES.
🤖 Automated review by Centaur · DAR-186
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 083279bac5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
strconv.ParseFloat accepts NaN/+Inf/Infinity spellings, and NaN slips past the v <= 0 filter (NaN comparisons are always false). A seed or overcommit entry like "gemma=NaN" then flows into qualityConcurrency / int(math.Ceil(qc * overcommit)) as an implementation-defined integer conversion, silently strangling the model to cap 1. parseModelFloatMap now skips non-finite values like every other malformed entry.
…ve request The heartbeat ingest gated solo sampling on the BOX being uncontended (soloSampleEligible, total running+waiting <= 1) but then recorded EVERY slot with a nonzero ObservedDecodeTPS. An idle co-resident slot keeps re-reporting its stale decayed EWMA, so with model A serving the box's one request, model B minted a duplicate 'solo' sample per ~30s heartbeat from a single long-past observation — reaching the min-sample trust floor with no real measurement and deriving B's quality cap from a rate no request produced. A fully idle box likewise minted samples from thin air. Now a slot is sampled only when it owns the box's one active request (NumRunning+NumWaiting > 0); a fully idle box records nothing. The load-inclusive Record path (TTFT estimation) is unchanged.
…s co-resident models The pooled gate reconstructed the shared KV pool by summing slot TOKEN counts, but co-resident models spend the ONE byte pool at different KVBytesPerToken rates (a 26B model's token costs ~10x a small model's), so tokens are not a common unit: the token pool is denominated by the largest per-slot free-token view (the smallest-KV model's), and a small-KV pending burst that fits token-wise can exhaust the byte pool while a big-KV co-resident request still passes the gate (90k small-KV tokens = 0.9 GB pending, +3k big-KV tokens = 0.3 GB reads as 93k <= 100k and admits into a 1 GB pool). providerPooledTokenBudget now additionally reconstructs the pool in bytes (per-slot tokens x that slot's KV rate, shared free byte headroom counted once), the routing snapshot carries byte-normalized all-models pending, and pooledBudgetAdmits checks in bytes when every unit is normalizable: all budget slots report a KV rate, every pending request's model has one, and the incoming request's own rate is known. Any gap (legacy provider, cold pending model) falls back to token accounting byte-for-byte — the single-model and legacy boundary tests pin that. Snapshot pending/pool assembly is deduplicated into fillSnapshotPendingAndPool, shared by the dispatch snapshot and the queue preflight.
…path The pooled gate only ran inside the resident-slot budget branch (activeTokenBudgetMax > 0). A request for a model with NO budget slot on the box (cold, not loaded) skipped it entirely and fell through to the memory-estimation path — so in-gap pending on a resident model was invisible to it, and the cold request double-spent the same shared KV its post-load serving will occupy. freeMemoryAdmits now also runs pooledBudgetAdmits on the budget-less path whenever any resident slot reports a token budget (pool.total > 0). A cold model has no reported KV rate, so the check runs in token units; legacy providers with no budget slots at all are untouched (pool.total == 0 admits unconditionally, pinned by TestFreeMemoryAdmitsLegacyProviderUnchanged).
…ission gate enforces ModelCapacitySnapshot summed per-slot budget headroom straight off the heartbeat, so a co-resident box whose shared pool was fully coordinator-pending to model A still advertised model B's stale slot (used 0 / max 10k) as remaining budget and counted the provider routable — capacity the dispatch path's pooledBudgetAdmits rejects, luring upstream routers into 429s during the heartbeat gap. Phase 1 now reconstructs each provider's pooled remaining (token units, mirroring the admission gate's committed-baseline charge of all-models pending); phase 2 clamps every per-slot headroom contribution by it and treats an exhausted pool (0, distinct from the -1 legacy sentinel) as no budget headroom for every model on the box — cold ones included, matching the cold-slot pooled gate.
…e queue-driven swap planner Review fixes for PR #526 (Codex X1, X5). The floor gate existed only in the dispatch gate and the warm-pool candidate gate; two sibling predicates could still disagree with routing about who serves a floored model: - providerCanRouteBuildLocked (alias routability): a below-floor box advertising an alias's Desired build made ResolveModel resolve to Desired, the candidate scan then found zero eligible providers, and the request queued/died against a build the fleet cannot serve instead of falling back to the Previous build. RoutableProviderIDsForBuild shares the predicate, so rollout/drop measurement now agrees too. - providerHasWarmModelLocked: a below-floor WARM box suppressed load planning for the model even though routing would never use its warm slot. - modelLoadCandidatePendingLocked: an idle below-floor box could be picked as a load_model target, burning GPU memory on an unroutable warm slot. Regression tests: alias fallback to Previous, planner ignoring a below-floor warm box, and planner never targeting a below-floor box (each fails without its check).
…e concurrency ceiling Review fix for PR #526 (Codex X4). The v2 tripwire clamped each SLOT's max_concurrency to the ceiling, but the provider-LEVEL valve (pendingCount < maxConcurrency, aggregated across ALL models) still used the legacy token-budget safety valve of 24 — so a >=floor v2 box with multiple warm models could admit slots x ceiling requests in aggregate, exceeding the box-wide cap the one-engine v2 design promises (productionMaxConcurrentRequests). Provider.maxConcurrency now bounds the legacy cap chain with v2BoxCeilingClamp for >=EIGENINFERENCE_V2_VERSION_FLOOR providers. Tightening only: a small box's hardware-derived cap of 2 stays 2, below-floor providers and the floor-unset default are byte-for-byte unchanged. Regression tests: two warm token-budget slots on a v2 box -> MaxConcurrency 4 and zero aggregate headroom at 2+2 pending (fails without the clamp); plus the no-behavior-change pins (below floor, floor disabled, never-raise).
Review fix for PR #526 (Codex X3). Providers resend a PERSISTENT per-slot observed_prefill_tps EWMA on every 30s heartbeat, and the ingest recorded a sample on every tick — so ONE cold-prefill measurement satisfied the prefill-median ring's min-sample trust floor (EIGENINFERENCE_TTFT_PREFILL_MIN_SAMPLES=5) after five idle ticks of the same value, defeating the floor's purpose of delaying trust until several real observations exist. Heartbeat now records a sample only when the reported value CHANGED since the connection's last recorded one (Provider.lastRecordedPrefillTPS, per-model, p.mu-guarded). Value change is the honest new-measurement signal on the wire: the per-slot first-token counters advance on prefix-cache hits too, which the provider deliberately does not sample, so they would over-count. Regression test: five identical heartbeats -> 1 sample (fails without the gate), genuine changes -> counted, per-model tracking independent.
…uest has no TTFT deadline Review fix for PR #526 (Codex X2, plus Centaur C1 doc note). In the default soft TTFT mode public requests carry MaxTTFTMs == 0, so band membership had NO TTFT criterion at all — a COLD (slotState "unknown") candidate passing the decode floor entered the band and could weighted-randomly beat a warm provider, forcing avoidable 15-60s model loads the deadline branch would have priced in via the slot-state penalty inside breakdown.TTFTMs. candidateInSatisficingBand now restricts the band to WARM (weights-resident) candidates when no deadline exists; cold boxes stay reachable via the cheapest-cost fallback, where statePenalty already makes them a last resort. Growing warm capacity remains the warm-pool controller's job. Also documents the deliberate math/rand use at the weighted-selection site: load-spreading jitter among fully-gated candidates, matching the existing near-tie rand.Intn in selectBestCandidateScanLocked — not a security boundary. Regression test: warm + cold candidates, MaxTTFTMs=0, band on -> the cold box is never selected across 100 trials (fails without the modelLoaded check).
…ty-caps # Conflicts: # coordinator/registry/scheduler.go
… live-used byte pool total Final Codex round on #524 — three real bugs in the byte-mode pooled-admission and solo-TPS code this PR adds: - ModelCapacitySnapshot clamped pooled budget in TOKEN units while the admission gate (pooledBudgetAdmits) enforces BYTES on mixed-KV providers, so /v1/models[/capacity] advertised headroom routing then refused. The snapshot now routes through a shared byte-aware helper (pooledRemainingTokens) computed per-model, keeping the feed's verdict identical to the gate's. - Solo TPS samples were recorded when a slot had running+waiting > 0, so a box with one QUEUED (not decoding) request minted its retained EWMA as a stale "solo" sample every heartbeat. Tightened to require an actual running decode (NumRunning > 0) plus the box-wide uncontended rule; preserves the prior round's owner-slot-only behavior. - The byte-mode pool total added committedBytes (which carries MaxTokensPotential worst-case growth) to the shared free headroom, double-counting a co-resident slot's not-yet-materialized growth as physical KV capacity. Now built from live used bytes + shared free, mirroring the token path (providerTokenBudget). Each fix ships a fails-without-fix regression test (revert-verified). All prior-round tests stay green (pooled admission, byte-normalization, cold-model pool charge, snapshot clamp, solo gating, postmortem cap). go test ./... -race -count=1 green across all 21 coordinator packages.
…snapshot) Merge origin/fix/per-model-quality-caps (cd7039c) into the version-floors branch so #526 is stacked on the current #524 tip (which itself carries master), making the PR mergeable against master. Clean merge — no textual conflicts. The ModelCapacitySnapshot composition auto-merged into disjoint regions of the per-model loop: #526's per-model version-floor gate (providerBelowModelVersionFloorLocked continue) at the top, #524's byte-aware per-model pooled remaining (pooledRemainingTokens over the fillSnapshotPendingAndPool byte pool) before the snap build. Both properties hold: floor-excluded providers are dropped from the snapshot AND included providers' pooled remaining is computed byte-aware. The merged function is byte-identical to the merge-train rehearsal's verified resolution. gofmt clean, go vet clean, go test ./... -race -count=1 green (21/21 packages, 0 data races).
#524 (byte-aware pooled remaining) and #526 (per-model version-floor gating) both edit ModelCapacitySnapshot; they auto-merge into the correct composition (below-floor boxes dropped AND byte-aware pooled remaining). This guard locks that down on one mixed-KV floored fleet — fails if either property regresses.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2494a31b09
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
+1125
to
+1127
| if r.providerBelowModelVersionFloorLocked(p, model) { | ||
| return false | ||
| } |
There was a problem hiding this comment.
With EIGENINFERENCE_MODEL_VERSION_FLOORS enabled for a dedicated model and only below-floor providers online, this gate makes QuickCapacityCheck report structural absence, but runInferenceAdmission then calls HasProviderForModel, which still counts any online provider advertising the build and ignores the new floor. That returns a transient machine_busy 429/Retry-After even though no eligible provider can serve until a binary upgrade; mirror the floor in HasProviderForModel or in that dedicated-model branch so the request gets the real no-provider verdict.
Useful? React with 👍 / 👎.
Comment on lines
1673
to
1675
| func resolvePrefillTPS(snap routingSnapshot) float64 { | ||
| tps := snap.prefillTPS | ||
| if snap.observedPrefillTPS > 0 { | ||
| tps = snap.observedPrefillTPS | ||
| } | ||
| if tps > maxPrefillTPS { | ||
| tps = maxPrefillTPS | ||
| } | ||
| return tps | ||
| return prefillTPSForSnapshot(snap, prefillFallbackMode == PrefillFallbackEnforce) | ||
| } |
There was a problem hiding this comment.
When prefill medians have samples or PREFILL_FALLBACK_MODE=enforce is enabled, dispatch/preflight now use this resolved prefill rate, but ModelCapacitySnapshot still computes estimated_ttft_ms from only the provider/static prefillTPS (or the slot's own observed value). For unmeasured slots this leaves /v1/models/capacity advertising the old pessimistic ~280 tok/s TTFT while the scheduler admits with the median/6500-tok/s estimate, so upstream routers that poll estimated_ttft_ms can keep deranking capacity that this coordinator would actually serve; populate the capacity snapshot from the same prefill resolution path.
Useful? React with 👍 / 👎.
Comment on lines
+162
to
+164
| case ttftPrefillMediansEnabled() && snap.prefillMedianTPS > 0 && | ||
| snap.prefillMedianSamples >= ttftPrefillMinSamples(): | ||
| tps = snap.prefillMedianTPS |
There was a problem hiding this comment.
When the prefill-median layer first crosses its sample floor in a long-running coordinator, this branch can replace the raw TTFT estimator with measured median prefill while calibratedTTFTMs continues applying calibration ratios learned against the old static estimate. If the old estimator was over-predicting and the learned ratio is near the 0.2 floor, the now-measured median TTFT is multiplied down and the hard TTFT gate/satisficing band can admit long-prompt routes that are actually over target until the calibration window turns over; key or reset the calibration by prefill source (static vs median/fallback) before trusting the new raw estimate.
Useful? React with 👍 / 👎.
ethenotethan
reviewed
Jul 9, 2026
ethenotethan
left a comment
ethenotethan
left a comment
Contributor
There was a problem hiding this comment.
Automated Code Review — Layr-Labs/d-inference#
Verdict: COMMENT
Security — ⚠️ SKIPPED
Error: OpenRouter HTTP 402: {"error":{"message":"Insufficient credits. Add more using https://openrouter.ai/settings/credits","code":402}}
Performance — ⚠️ SKIPPED
Error: OpenRouter HTTP 402: {"error":{"message":"Insufficient credits. Add more using https://openrouter.ai/settings/credits","code":402}}
Type_diligence — ⚠️ SKIPPED
Error: OpenRouter HTTP 402: {"error":{"message":"Insufficient credits. Add more using https://openrouter.ai/settings/credits","code":402}}
Additive_complexity — ⚠️ SKIPPED
Error: OpenRouter HTTP 402: {"error":{"message":"Insufficient credits. Add more using https://openrouter.ai/settings/credits","code":402}}
✅ All four passes clean. No issues found.
🤖 Automated review by Centaur · DAR-186
ethenotethan
reviewed
Jul 9, 2026
ethenotethan
left a comment
ethenotethan
left a comment
Contributor
There was a problem hiding this comment.
Automated Code Review — Layr-Labs/d-inference#
Verdict: COMMENT
Security — ⚠️ SKIPPED
Error: OpenRouter HTTP 402: {"error":{"message":"Insufficient credits. Add more using https://openrouter.ai/settings/credits","code":402}}
Performance — ⚠️ SKIPPED
Error: OpenRouter HTTP 402: {"error":{"message":"Insufficient credits. Add more using https://openrouter.ai/settings/credits","code":402}}
Type_diligence — ⚠️ SKIPPED
Error: OpenRouter HTTP 402: {"error":{"message":"Insufficient credits. Add more using https://openrouter.ai/settings/credits","code":402}}
Additive_complexity — ⚠️ SKIPPED
Error: OpenRouter HTTP 402: {"error":{"message":"Insufficient credits. Add more using https://openrouter.ai/settings/credits","code":402}}
✅ All four passes clean. No issues found.
🤖 Automated review by Centaur · DAR-186
Byte-mode pooled admission multiplies heartbeat token counts (clamped to ~10B) by slot.KVBytesPerToken, which is unbounded. A garbage/absurd rate overflows int64 to negative used/total bytes, breaking admission (a negative pool total rejects everything, or with a negative left side admits everything). Add maxKVBytesPerToken (16 MiB/token, ~160x gemma's real rate so legitimate rates pass untouched) and clampKVBytesPerToken; apply it in providerPooledTokenBudget (map store + all three byte products) and at the two scheduler snapshot sites that read slot.KVBytesPerToken. The pending-byte normalization reads from the now-clamped pool map, no change needed there. Regression test: a slot with KVBytesPerToken=MaxInt64/2 no longer wraps used/total bytes negative and admission fails closed.
…fallback The solo quality-cap resolver keyed by ChipFamily alone, lumping an M4 Max with an M4 Pro (3-4x apart in decode throughput), and SoloMedianAllChips pooled every chip's samples into one median. A fast, sample-heavy chip could then raise a slow box's cap — over-admission, the exact collapse this PR prevents. Add chipClassKey (family|tier, byte-for-byte #526's prefillChipClass), key RecordSolo ingest and the resolver's primary lookup by class (the load-inclusive Record stays family-keyed for fleetMedianTPS). Rewrite SoloMedianAllChips to return the MIN of per-class medians (never a fast-chip-dominated pooled median) plus the total sample count, so the cross-class fallback can never exceed the slowest class's typical rate. SAFETY INVARIANT: the resolver never hands a slow box a rate faster than its own class demonstrated; worst case it under-caps a fast box (safe). Tests (revert-verified): min-of-class semantics, same-class primary, cross-tier isolation, conservative fallback, cold-start, and an end-to-end heartbeat-driven no-cross-tier-over-cap check.
…lback On a byte-reconstructable mixed-KV box, pooledBudgetAdmits and pooledRemainingTokens collapsed to TOKEN accounting whenever the incoming request's own KV rate was 0 (a cold/absent slot), even though the resident pool is byte-reconstructable. A large-KV cold request was then under-charged against the small-KV token view of the pool and could double-spend the shared KV. Track pool.maxKVBytesPerToken (max clamped resident rate) and, in the byteMode && pendingBytesKnown branch, price a cold request (rate 0) at that max rate instead of falling through to tokens. Fall through to token accounting ONLY when !byteMode or !pendingBytesKnown. Single-KV and no-byte-mode paths are arithmetically unchanged. pooledRemainingTokens applies the SAME cold-rate substitution so the capacity feed stays equivalent to the gate. Tests (revert-verified): cold mixed-KV request charged in bytes (rejects an over-pool burst the token fallback admitted), single-KV boundary unchanged, and an admits<->remaining equivalence sweep across byte/cold/ token/pending-unknown paths.
The cold-slot pooled gate (scheduler.go) and the capacity feed (registry.go) comments still described the pre-fix behavior where a cold request (no reported KV rate) fell to token accounting. After the Fix 3 cold-rate substitution, a byte-reconstructable pool prices cold requests in bytes at the box's max resident rate in both the gate and the feed. Comment-only; no behavior change. (Flagged by both PR #524 reviewers.)
Refresh #526's stacked base onto the final-round #524 (fix/per-model-quality-caps): KV-rate overflow clamp, chip-CLASS solo keying + min-of-class cross-chip fallback, byte-priced cold requests, and cold-path docs. registry.go ModelCapacitySnapshot composition holds both concerns: below-floor providers are dropped from the public capacity feed AND per-model pooled remaining is byte-aware with cold-byte pricing. scheduler.go auto-merged clean. Composition guard TestModelCapacitySnapshotFlooredByteModeComposition still green under #524's cold-byte pricing.
ethenotethan
reviewed
Jul 9, 2026
ethenotethan
left a comment
ethenotethan
left a comment
Contributor
There was a problem hiding this comment.
Automated Code Review — Layr-Labs/d-inference#
Verdict: COMMENT
Security — ⚠️ SKIPPED
Error: OpenRouter HTTP 402: {"error":{"message":"Insufficient credits. Add more using https://openrouter.ai/settings/credits","code":402}}
Performance — ⚠️ SKIPPED
Error: OpenRouter HTTP 402: {"error":{"message":"Insufficient credits. Add more using https://openrouter.ai/settings/credits","code":402}}
Type_diligence — ⚠️ SKIPPED
Error: OpenRouter HTTP 402: {"error":{"message":"Insufficient credits. Add more using https://openrouter.ai/settings/credits","code":402}}
Additive_complexity — ⚠️ SKIPPED
Error: OpenRouter HTTP 402: {"error":{"message":"Insufficient credits. Add more using https://openrouter.ai/settings/credits","code":402}}
✅ All four passes clean. No issues found.
🤖 Automated review by Centaur · DAR-186
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5b4c035afb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
+20
to
+24
| // Enforcement points (all four, so alias resolution, routing, and warming can | ||
| // never disagree about who serves a floored model): | ||
| // - the single routing gate providerPassesRoutingGatesLockedEx (shared by the | ||
| // dispatch hot path, the capacity preflight, and the final admit re-check), | ||
| // beside the trait version floors; |
There was a problem hiding this comment.
When EIGENINFERENCE_MODEL_VERSION_FLOORS is enabled and only below-floor providers advertise a build, the new routing/preflight gates reject every request, but the public listing surfaces still aggregate those providers: ListModels only checks catalog/trust/private-text readiness, and ModelCountryCodes likewise counts providers by advertised model. That leaves /v1/models and OpenRouter datacenters advertising a model/location that has no routable provider, so upstreams can keep sending traffic that immediately fails; include these listing predicates in the floor enforcement set too.
Useful? React with 👍 / 👎.
Comment on lines
+128
to
+130
| func SetMaxPrefillTPS(tps float64) { | ||
| if tps > 0 { | ||
| maxPrefillTPS = tps |
There was a problem hiding this comment.
If EIGENINFERENCE_MAX_PREFILL_TPS is set to +Inf/Inf, strconv.ParseFloat accepts it and this setter installs an infinite maxPrefillTPS because it only checks > 0. With an infinite ceiling, clampBackendCapacity no longer zeroes +Inf observed prefill reports (+Inf > maxPrefillTPS is false), allowing a broken provider to seed infinite prefill medians and drive TTFT estimates toward zero for long prompts; reject non-finite values before updating the ceiling.
Useful? React with 👍 / 👎.
Comment on lines
+807
to
+809
| if satisficingBandEnabled() { | ||
| if band := satisficingBandMembers(pool, pr); len(band) > 0 { | ||
| winner := r.pickSatisficingBandLocked(band) |
There was a problem hiding this comment.
When EIGENINFERENCE_SATISFICING_BAND=true, this path picks from every TTFT/decode-clearing band member using only inverse recent serves. That bypasses cost penalties already computed on each candidate, including the gray-box capacityRateMs derater for providers that recently returned material capacity 503s, so a degraded pair that the normal cheapest-cost path would sink can be randomly selected again as long as its TTFT/decode estimates clear the band. Filter or weight band members by those health/capacity penalties so enabling the canary does not undo the existing derating.
Useful? React with 👍 / 👎.
Gajesh2007
added a commit
that referenced
this pull request
Jul 9, 2026
…(postmortem layer 6) (#524) * fix(registry): per-model solo-TPS quality caps (gemma postmortem layer 6) The quality-concurrency cap consumed resolvedDecodeTPS(p) — the provider- LEVEL registration benchmark — for every model on the box. Mixed boxes benchmarked on gpt-oss (58-93 tok/s) therefore granted gemma caps of 12-23 where gemma's real solo rate (10-18 tok/s) warrants 1-2, and the coordinator marched 8-11 concurrent gemma requests onto exactly the boxes that collapse past batch 3 (2026-07-06 postmortem §3.2). The cap now resolves each model's OWN static solo rate: - Solo-gated sample store (solo_tps.go): RecordSolo/SoloMedian beside the existing 50-sample ring. Heartbeat ingest records a slot EWMA as solo ONLY when the whole box has ≤1 running+waiting request across ALL slots (the allowance is the sample-generating request itself) — mixed-box samples stay honest. The existing Record/Median (fleetMedianTPS → TTFT estimation) keeps its deliberately load-inclusive semantics, untouched. - Resolver resolvedSoloModelTPSLocked, fallback chain: per-(model,chip) solo median at ≥ EIGENINFERENCE_QUALITY_CAP_SOLO_MIN_SAMPLES (default 5) → cross-chip pooled solo median → EIGENINFERENCE_MODEL_SOLO_TPS_SEED (model=tok/s CSV; cold-start answer, the registry is restart-wiped) → resolvedDecodeTPS(p) unchanged. Rates stay STATIC (never under-load EWMA) so the cap cannot feedback-collapse; solo gating preserves that. - Kill switch EIGENINFERENCE_QUALITY_CAP_PER_MODEL_TPS (default true); false restores resolvedDecodeTPS(p) at every wired site exactly. - The DecodeTPS<=0 dedicated-only guard now applies only to the provider- level fallback: a per-model rate (solo median/seed) is model-specific by construction and caps even without a registration benchmark. resolvedDecodeTPS call sites, each decided deliberately: SWITCHED to the solo resolver (quality-cap inputs, all through the single entry hasConcurrencyHeadroomForModelCapResolvedLocked): - scheduler.go routing-snapshot headroom (snap.hasHeadroom) - scheduler.go quickCapacityCheck preflight concurrency gate - scheduler.go providerCanAdmitLockedEx final admit re-check (already Resolved; now per-model internally) - registry.go ModelCapacitySnapshot public-capacity headroom (already Resolved; now per-model internally) - warm_pool_controller.go warmSaturated gate (already Resolved) - warm_pool_controller.go warm-target decode samples: soloDecodeTPS now comes from the solo resolver so warm targets and admission caps cannot disagree; the observed-EWMA chain moved to a new serviceDecodeTPS that feeds only the E[S] service-time estimate (load-inclusive on purpose). KEPT provider-level (TTFT-estimation / cost-model / scoring semantics): - scheduler.go snap.decodeTPS (TTFT + cost estimates, effectiveDecodeTPS) - scheduler.go resolvedModelTPSLocked base (observed-EWMA cost chain) - scheduler.go resolvedPrefillTPS fallback (decode × ratio-12) - registry.go ModelCapacitySnapshot effectiveTPS (capacity-feed rate) - utilization.go FleetCapacity.DecodeTPS aggregate - warm_pool_controller.go cold-candidate score heuristic The unused explicit-rate hasConcurrencyHeadroomForModelCapLocked was deleted (golangci-lint unused); effectiveMaxConcurrencyForModelLocked stays as the explicit-rate wrapper. Regression tests (fail with the resolver mutation reverted; verified): the exact postmortem mixed box (benchmark 93, gemma solo 14, floor 15 → gemma cap 2, gpt-oss cap 23; kill switch off reproduces the old 23), solo ingest gating through real heartbeats, fallback-chain order, min-sample floor, seed parsing (malformed entries skipped), restart cold-start via seed, warm-pool solo/service rate split. New envs (documented in docs/operations/coordinator-deploy.md): EIGENINFERENCE_QUALITY_CAP_PER_MODEL_TPS, EIGENINFERENCE_QUALITY_CAP_SOLO_MIN_SAMPLES, EIGENINFERENCE_MODEL_SOLO_TPS_SEED. * fix(registry): pooled-KV admission — stop co-resident double-spend of the shared budget freeMemoryAdmits admitted against the per-slot ActiveTokenBudgetMax with only SAME-MODEL coordinator pending counted. But a slot max is not a private budget: every slot on a box reports its own committed tokens plus the SAME shared KV headroom (see providerTokenBudget). Two co-resident models could therefore double-spend the one shared pool inside the ~30s heartbeat gap — a burst admitted against model A's slot was invisible to model B's stale slot max until the next heartbeat. Fix (kept minimal around freeMemoryAdmits for the pending budget-clamp branch to merge next to): - pooled_admission.go: providerPooledTokenBudget reconstructs the whole-box pool from the slots via the proven providerTokenBudget (committed sums across slots; shared headroom counted exactly once), plus an all-slots committed baseline that mirrors committedTokenBudget (max(used+queued, maxTokensPotential) per slot) so heartbeat-reflected requests are not double-counted against coordinator pending. - routingSnapshot gains pendingMaxTokensAllModels (the pending-token accumulator WITHOUT the model filter) and the pooled budget, populated by both snapshot builders (dispatch + preflight). - Budget admission is now: fits the per-slot budget (unchanged — the slot max still encodes the model's own context limits) AND fits the pool with every model's coordinator-pending tokens charged. Single-model providers are arithmetically unchanged (the pool reduces to the slot's own budget; boundary pinned byte-for-byte in TestFreeMemoryAdmitsSingleModelUnchanged, including the maxTokensPotential-dominates case). Legacy providers without token budgets never reach the pooled check. Inert under DEDICATED partitioning (no co-resident dispatch); live before the open-pool flip. Regression tests (fail with pooledBudgetAdmits mutated out; verified): burst model A to the whole pool through the REAL reservation path, then a model B request within the heartbeat gap is rejected; co-resident request WITHIN pool headroom still admits; 6th same-model request past the slot budget still rejects; pure-function double-spend + boundary tables; pooled-budget reconstruction arithmetic (shared headroom once, negative floors, budgetless slots ignored). * fix(registry): reject non-finite values in per-model tunable CSV parsing strconv.ParseFloat accepts NaN/+Inf/Infinity spellings, and NaN slips past the v <= 0 filter (NaN comparisons are always false). A seed or overcommit entry like "gemma=NaN" then flows into qualityConcurrency / int(math.Ceil(qc * overcommit)) as an implementation-defined integer conversion, silently strangling the model to cap 1. parseModelFloatMap now skips non-finite values like every other malformed entry. * fix(registry): solo TPS samples only from the slot that owns the active request The heartbeat ingest gated solo sampling on the BOX being uncontended (soloSampleEligible, total running+waiting <= 1) but then recorded EVERY slot with a nonzero ObservedDecodeTPS. An idle co-resident slot keeps re-reporting its stale decayed EWMA, so with model A serving the box's one request, model B minted a duplicate 'solo' sample per ~30s heartbeat from a single long-past observation — reaching the min-sample trust floor with no real measurement and deriving B's quality cap from a rate no request produced. A fully idle box likewise minted samples from thin air. Now a slot is sampled only when it owns the box's one active request (NumRunning+NumWaiting > 0); a fully idle box records nothing. The load-inclusive Record path (TTFT estimation) is unchanged. * fix(registry): byte-normalize pooled-KV admission across heterogeneous co-resident models The pooled gate reconstructed the shared KV pool by summing slot TOKEN counts, but co-resident models spend the ONE byte pool at different KVBytesPerToken rates (a 26B model's token costs ~10x a small model's), so tokens are not a common unit: the token pool is denominated by the largest per-slot free-token view (the smallest-KV model's), and a small-KV pending burst that fits token-wise can exhaust the byte pool while a big-KV co-resident request still passes the gate (90k small-KV tokens = 0.9 GB pending, +3k big-KV tokens = 0.3 GB reads as 93k <= 100k and admits into a 1 GB pool). providerPooledTokenBudget now additionally reconstructs the pool in bytes (per-slot tokens x that slot's KV rate, shared free byte headroom counted once), the routing snapshot carries byte-normalized all-models pending, and pooledBudgetAdmits checks in bytes when every unit is normalizable: all budget slots report a KV rate, every pending request's model has one, and the incoming request's own rate is known. Any gap (legacy provider, cold pending model) falls back to token accounting byte-for-byte — the single-model and legacy boundary tests pin that. Snapshot pending/pool assembly is deduplicated into fillSnapshotPendingAndPool, shared by the dispatch snapshot and the queue preflight. * fix(registry): charge the shared KV pool on the cold-model admission path The pooled gate only ran inside the resident-slot budget branch (activeTokenBudgetMax > 0). A request for a model with NO budget slot on the box (cold, not loaded) skipped it entirely and fell through to the memory-estimation path — so in-gap pending on a resident model was invisible to it, and the cold request double-spent the same shared KV its post-load serving will occupy. freeMemoryAdmits now also runs pooledBudgetAdmits on the budget-less path whenever any resident slot reports a token budget (pool.total > 0). A cold model has no reported KV rate, so the check runs in token units; legacy providers with no budget slots at all are untouched (pool.total == 0 admits unconditionally, pinned by TestFreeMemoryAdmitsLegacyProviderUnchanged). * fix(registry): clamp /v1/models capacity by the pooled budget the admission gate enforces ModelCapacitySnapshot summed per-slot budget headroom straight off the heartbeat, so a co-resident box whose shared pool was fully coordinator-pending to model A still advertised model B's stale slot (used 0 / max 10k) as remaining budget and counted the provider routable — capacity the dispatch path's pooledBudgetAdmits rejects, luring upstream routers into 429s during the heartbeat gap. Phase 1 now reconstructs each provider's pooled remaining (token units, mirroring the admission gate's committed-baseline charge of all-models pending); phase 2 clamps every per-slot headroom contribution by it and treats an exhausted pool (0, distinct from the -1 legacy sentinel) as no budget headroom for every model on the box — cold ones included, matching the cold-slot pooled gate. * fix(registry): byte-unit capacity snapshot, running-decode solo gate, live-used byte pool total Final Codex round on #524 — three real bugs in the byte-mode pooled-admission and solo-TPS code this PR adds: - ModelCapacitySnapshot clamped pooled budget in TOKEN units while the admission gate (pooledBudgetAdmits) enforces BYTES on mixed-KV providers, so /v1/models[/capacity] advertised headroom routing then refused. The snapshot now routes through a shared byte-aware helper (pooledRemainingTokens) computed per-model, keeping the feed's verdict identical to the gate's. - Solo TPS samples were recorded when a slot had running+waiting > 0, so a box with one QUEUED (not decoding) request minted its retained EWMA as a stale "solo" sample every heartbeat. Tightened to require an actual running decode (NumRunning > 0) plus the box-wide uncontended rule; preserves the prior round's owner-slot-only behavior. - The byte-mode pool total added committedBytes (which carries MaxTokensPotential worst-case growth) to the shared free headroom, double-counting a co-resident slot's not-yet-materialized growth as physical KV capacity. Now built from live used bytes + shared free, mirroring the token path (providerTokenBudget). Each fix ships a fails-without-fix regression test (revert-verified). All prior-round tests stay green (pooled admission, byte-normalization, cold-model pool charge, snapshot clamp, solo gating, postmortem cap). go test ./... -race -count=1 green across all 21 coordinator packages. * fix(registry): clamp KV byte rates before byte-pool multiplication Byte-mode pooled admission multiplies heartbeat token counts (clamped to ~10B) by slot.KVBytesPerToken, which is unbounded. A garbage/absurd rate overflows int64 to negative used/total bytes, breaking admission (a negative pool total rejects everything, or with a negative left side admits everything). Add maxKVBytesPerToken (16 MiB/token, ~160x gemma's real rate so legitimate rates pass untouched) and clampKVBytesPerToken; apply it in providerPooledTokenBudget (map store + all three byte products) and at the two scheduler snapshot sites that read slot.KVBytesPerToken. The pending-byte normalization reads from the now-clamped pool map, no change needed there. Regression test: a slot with KVBytesPerToken=MaxInt64/2 no longer wraps used/total bytes negative and admission fails closed. * fix(registry): key solo caps by chip CLASS + conservative cross-chip fallback The solo quality-cap resolver keyed by ChipFamily alone, lumping an M4 Max with an M4 Pro (3-4x apart in decode throughput), and SoloMedianAllChips pooled every chip's samples into one median. A fast, sample-heavy chip could then raise a slow box's cap — over-admission, the exact collapse this PR prevents. Add chipClassKey (family|tier, byte-for-byte #526's prefillChipClass), key RecordSolo ingest and the resolver's primary lookup by class (the load-inclusive Record stays family-keyed for fleetMedianTPS). Rewrite SoloMedianAllChips to return the MIN of per-class medians (never a fast-chip-dominated pooled median) plus the total sample count, so the cross-class fallback can never exceed the slowest class's typical rate. SAFETY INVARIANT: the resolver never hands a slow box a rate faster than its own class demonstrated; worst case it under-caps a fast box (safe). Tests (revert-verified): min-of-class semantics, same-class primary, cross-tier isolation, conservative fallback, cold-start, and an end-to-end heartbeat-driven no-cross-tier-over-cap check. * fix(registry): price cold unknown-KV requests in bytes, not token fallback On a byte-reconstructable mixed-KV box, pooledBudgetAdmits and pooledRemainingTokens collapsed to TOKEN accounting whenever the incoming request's own KV rate was 0 (a cold/absent slot), even though the resident pool is byte-reconstructable. A large-KV cold request was then under-charged against the small-KV token view of the pool and could double-spend the shared KV. Track pool.maxKVBytesPerToken (max clamped resident rate) and, in the byteMode && pendingBytesKnown branch, price a cold request (rate 0) at that max rate instead of falling through to tokens. Fall through to token accounting ONLY when !byteMode or !pendingBytesKnown. Single-KV and no-byte-mode paths are arithmetically unchanged. pooledRemainingTokens applies the SAME cold-rate substitution so the capacity feed stays equivalent to the gate. Tests (revert-verified): cold mixed-KV request charged in bytes (rejects an over-pool burst the token fallback admitted), single-KV boundary unchanged, and an admits<->remaining equivalence sweep across byte/cold/ token/pending-unknown paths. * docs(registry): update cold-path comments for byte-priced cold requests The cold-slot pooled gate (scheduler.go) and the capacity feed (registry.go) comments still described the pre-fix behavior where a cold request (no reported KV rate) fell to token accounting. After the Fix 3 cold-rate substitution, a byte-reconstructable pool prices cold requests in bytes at the box's max resident rate in both the gate and the feed. Comment-only; no behavior change. (Flagged by both PR #524 reviewers.) * fix(registry): bound cross-class solo rates by seed * fix(registry): keep cold pooled KV accounting in bytes * fix(registry): honor authoritative zero KV budgets
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR body —
fix/version-floors-ttft-band@083279baProblem
Three problems, one migration-defense stack:
docs/reports/2026-07-06-gemma4-serving-failure-postmortem.md§3.1) showed what silently-degraded boxes do to a model pool: 54% of open-pool gemma traffic ran on the legacy engine at 2–3 tok/s. The coordinator needs (a) a permanent tripwire that catches the silent-legacy signature (a ≥0.7.5 box heartbeatingmax_concurrency24) and (b) a way to fence gemma onto re-slice-capable binaries during the migration.sqrt(bandwidth) × EIGENINFERENCE_PREFILL_DECODE_RATIO(12)≈ 280 tok/s, ~23× below measured reality (fleet p50 6,523 tok/s, p90 17,707 — recovered from the router's own storedttft_ms). gpt-oss's p90 prompt is 13,711 tokens, so predicted TTFT ≈ 38s against a 5s deadline and 18.2k requests/day die onttft_too_slowmispredictions (~41% long-prompt over-shed) while the fleet idles.What changed (per commit)
81bba621— absorbed PR fix(coordinator): recalibrate prefill-TPS fallback — cut the 41% long-prompt ttft_429 over-shed #453: prefill-TPS fallback recalibration (registry/prefill_fallback.go,api/prefill_fallback_metrics.go): env-gated, data-derived prefill fallback anchor —EIGENINFERENCE_PREFILL_FALLBACK_MODEoff|shadow|enforce(default off, behavior-neutral),EIGENINFERENCE_PREFILL_FALLBACK_TPS(default 6500 = measured p50).shadowemitsrouting.prefill_fallback{would_admit|would_shed}from the preflight so the projectedttft_429recovery is measured before enforcing. Also raises the prefill sanity ceilingmaxPrefillTPSfrom the hardcoded constant 5000 to env-tunableEIGENINFERENCE_MAX_PREFILL_TPS(default 20000, above measured p90) — a hard prerequisite for commit 3: the old 5000 ingest clamp would zero the majority of real v0.7.5observed_prefill_tpsreports (p50 6,523 > 5,000) before they could ever feed a median. Cost ranking untouched; only the TTFT admission gate moves.66f12329— mixed-fleet defense (plan 2.4): (a) version-keyed heartbeat sanity clamp (registry/v2_capacity_clamp.go,api/v2_clamp_metrics.go): providers at/aboveEIGENINFERENCE_V2_VERSION_FLOOR(default empty = off) heartbeating a chat-slotmax_concurrencyaboveEIGENINFERENCE_V2_MAX_CONCURRENCY_CEILING(default 4) are clamped, logged at ERROR, and counted on theprovider.v2_concurrency_tripwireDatadog counter — on a ≥floor box that report means the silent-legacy bug resurfaced; this is a permanent audit of the release's fail-loud promise, not a rollout metric. The v2 pass runs BEFORE the generalclampBackendCapacitypass so the tripwire records the original report; below-floor and empty-version providers keep today's clamp-24 behavior byte-for-byte. (b) per-model provider-version routing floors (registry/model_version_floors.go):EIGENINFERENCE_MODEL_VERSION_FLOORS="gemma-4=0.7.5"(pattern=version CSV, case-insensitive substring likeDEDICATED_MODELS) enforced in the single routing gate (providerPassesRoutingGatesLockedEx— dispatch, preflight, and final admit re-check all agree) AND the warm-pool candidate gate (newwarmColdBelowVersionFloorreason) AND self-route. Empty-version providers fail every floor; malformed CSV degrades to no-floor, never a wrong floor.b97c32ce— prefill-honest TTFT (plan 2.7;registry/prefill_tps.go): a third TPSRegistry ring (RecordPrefill/PrefillMedian, same 50-sample FIFO shape) of theobserved_prefill_tpsEWMAs v0.7.5 providers report in heartbeats. TTFT prefill resolution (prefillTPSForSnapshot, consumed byresolvePrefillTPS) becomes, most- to least-specific: own slot measurement → trusted per-(model, chip) median (≥EIGENINFERENCE_TTFT_PREFILL_MIN_SAMPLES, default 5) → (enforce mode) the fix(coordinator): recalibrate prefill-TPS fallback — cut the 41% long-prompt ttft_429 over-shed #453 anchor → static decode×ratio chain, applied in BOTH the dispatch snapshot and the capacity preflight, with the fix(coordinator): seven routing fixes from prod-DB analysis — TTFT gate calibration, reconnect-proof breakers, dedicated queueing, decode floor, servability #512 online calibrator kept downstream as corrector. Deliberately NO cross-chip pooling (prefill is compute-bound, spreads 3–4× across tiers) and NOT solo-gated (prefill is per-request compute work; garbage EWMAs are zeroed at ingest by the clamp). Kill switchEIGENINFERENCE_TTFT_PREFILL_MEDIANS(default true, live-read) — false restores the pure ratio path exactly.87d045c1— satisficing utilization band (plan 2.8;registry/satisficing_band.go), behindEIGENINFERENCE_SATISFICING_BAND(default FALSE — fully dormant, flag-off selection byte-identical and pinned by test): band = candidates whose calibratedbreakdown.TTFTMsclears the request deadline minusEIGENINFERENCE_SATISFICING_TTFT_MARGIN_MS(default 1000) AND whoseprojectedPerRequestDecodeTPSclears the decode floor — the exact predicates admission already computes, no new estimators. Within the band: weighted-random, weight1/(1+recentServes)from a registry-local exponentially-decaying per-provider counter (half-life 5 min), fed on every successful reservation regardless of the flag so weights are warm at flip time. Cache affinity keeps its pin within the band; an empty band falls through to exactly today's cheapest-cost path; speculative dispatch/queueing/retries untouched. Hooked inselectBestCandidateScanLocked.083279ba— quality-gate review fixes:OwnedProviderSummarynow applies the model version floor (a below-floor OWNED box previously read "serves model" in the self-route preflight, queued 120s, died asmachine_busy); the prefill-fallback SHADOW estimate now applies fix(coordinator): seven routing fixes from prod-DB analysis — TTFT gate calibration, reconnect-proof breakers, dedicated queueing, decode floor, servability #512 online calibration sowould_admit|would_shedcompares calibrated-vs-calibrated (an enforce flip gates on calibrated values; the uncalibrated shadow under-reported the projected recovery); tests pin env defaults viat.Setenv. Documented-not-changed reviewer note: withEIGENINFERENCE_LONG_PROMPT_TOKENS > 0(default 0 = off), a trusted median/anchor also shifts long-prompt cost ranking vialongPromptPenalty— intentional per that knob's design.Before / After
flowchart TB subgraph Before["Before — trusting heartbeats, static prefill guess, winner-takes-all"] A1["0.7.4 mixed box, silent legacy fallback<br/>heartbeats max_concurrency 24"] --> B1[clampBackendCapacity accepts ≤24] --> C1[coordinator trusts it<br/>→ traffic marched into the trap] D1[13.7k-token gpt-oss prompt] --> E1["resolvePrefillTPS: no measurement →<br/>sqrt(bandwidth)×12 ≈ 280 tok/s"] --> F1["predicted TTFT ~38s > 5s deadline<br/>→ ttft_too_slow 429 (18.2k/day false)"] G1[selection] --> H1[strict cheapest-cost + near-tie<br/>→ top-10% boxes take 79%] end subgraph After["After — floors + tripwire, measured prefill, SLO band (dormant)"] A2["box ≥ V2_VERSION_FLOOR<br/>heartbeats max_concurrency 24"] --> B2["clampV2MaxConcurrency FIRST:<br/>clamp→4, ERROR log,<br/>provider.v2_concurrency_tripwire"] --> C2[general clamp pass second] A3[gemma request] --> B3{"MODEL_VERSION_FLOORS<br/>gemma-4=0.7.5: provider ≥ floor?"} -- no --> C3[not routed, not warmed, not self-routed<br/>warmColdBelowVersionFloor tallied] B3 -- yes --> D3[normal gates] D2[13.7k-token prompt] --> E2["prefillTPSForSnapshot:<br/>own slot → (model,chip) median n≥5<br/>→ enforce? 6500 anchor → ratio chain"] --> F2["honest TTFT → routes to<br/>high-prefill boxes instead of shed"] G2[selection] --> H2{SATISFICING_BAND=true?} H2 -- "off (default)" --> I2[byte-identical cheapest-cost] H2 -- on --> J2["band = deadline−margin & decode-floor<br/>members → weighted random 1/(1+recentServes),<br/>affinity pin kept; empty band → cheapest-cost"] endNew envs
EIGENINFERENCE_V2_VERSION_FLOORmax_concurrencyabove the ceiling is clamped + tripwired (provider.v2_concurrency_tripwire)0.7.5at Deploy B once ≥70% of online ≥36GB boxes run 0.7.5; permanent audit thereafterEIGENINFERENCE_V2_MAX_CONCURRENCY_CEILING4productionMaxConcurrentRequests); must be ≥1EIGENINFERENCE_MODEL_VERSION_FLOORSpattern=versionCSV; floored models route/warm/self-route only onto ≥floor providersgemma-4=0.7.5with the V2 floor; retire after fleet convergenceEIGENINFERENCE_TTFT_PREFILL_MEDIANStrue(live-read)observed_prefill_tps; kill switch if misbehavingEIGENINFERENCE_TTFT_PREFILL_MIN_SAMPLES5(live-read)EIGENINFERENCE_PREFILL_FALLBACK_MODEoffoff|shadow|enforce— the #453 anchor for UNMEASURED (model, chip) pairs; shadow = metrics onlyshadowat Deploy B;enforceat runbook step 7 after shadow validatesEIGENINFERENCE_PREFILL_FALLBACK_TPS6500EIGENINFERENCE_MAX_PREFILL_TPS20000(was hardcoded 5000)EIGENINFERENCE_SATISFICING_BANDfalse(live-read)EIGENINFERENCE_SATISFICING_TTFT_MARGIN_MS1000(live-read)Test evidence
TestPrefillMedianLongPromptRegression(prefill_tps_test.go) — the production overshoot: a 13,711-token prompt on a warm box whose fleet-measured prefill is ~6,500 tok/s is admitted with medians on and shed with them off (fails with the median layer neutralized).TestPrefillIngestZeroedUnderOldCeiling— pins why the fix(coordinator): recalibrate prefill-TPS fallback — cut the 41% long-prompt ttft_429 over-shed #453 ceiling raise is a prerequisite: real reports (6,523 tok/s) are zeroed at ingest under the old 5000 cap.TestV2ClampAtOrAboveFloorClampsAndTrips(incl. a 30-report that bypasses the legacy-24 stage),TestV2ClampOriginalReportSurvivesLegacyClamp(ordering),TestV2ClampBelowFloorKeepsLegacyBehavior+TestV2ClampEmptyVersionIsBelowFloor+TestV2ClampDisabledWhenFloorEmpty(flag-off/below-floor byte-identical pins).TestModelVersionFloorRoutingGate,TestModelVersionFloorPreflightConsistency(shed/Retry-After agree with routing),TestModelVersionFloorWarmPoolCandidacy,TestModelVersionFloorOwnedSummaryConsistency(the 083279b review fix — fails without it),TestModelVersionFloorDisabledByDefault,TestParseModelVersionFloors.TestLongPromptAdmittedUnderEnforceFallback,TestPrefillFallbackShadowIsBehaviorNeutral(shadow = byte-identical routing),TestEmitPrefillFallbackShadowWouldAdmit/WouldShed,TestMaxPrefillCeilingRaiseKeepsHighObserved,TestPrefillTPSForSnapshotResolutionOrder,TestPrefillTPSForSnapshotRecalibration(calibrated shadow, the other 083279b fix).TestSatisficingBandOffSelectionPinned(50 trials, flag-off deterministic-cheapest — the byte-identical pin),TestSatisficingBandSpreadsAcrossMembers(300 trials through the REALReserveProviderExpath),TestSatisficingBandNonMembersOnlyWhenBandEmpty,TestSatisficingBandWeightsShiftWithRecentServes(~9 recent serves → ~1/10 the picks, no starvation),TestRecentServeCounterDecay(injected clock),TestSatisficingBandServeCounterFedOnRealPath(flag off),TestSatisficingBandCacheAffinityPinnedWithinBand.Suite status: re-verified 2026-07-07 by ship-prep:
go test ./registry/... ./api/...ok in the worktree.Review status
Codex rescue: FAIL → fixed in
083279ba(self-route preflight missing the version floor; uncalibrated shadow TTFT), then independent reviewer: PASS on the fixed stack (dual-reviewed per JOURNAL.md Foundry row).Rollout notes
observed_prefill_tps; theMAX_PREFILL_TPSceiling raise is behavior-neutral on today's non-reporting fleet.PREFILL_FALLBACK_MODE=shadowat Deploy B → floors (V2_VERSION_FLOOR=0.7.5,MODEL_VERSION_FLOORS=gemma-4=0.7.5) at ≥70% online adoption →enforceat step 7 if shadow numbers validate →SATISFICING_BAND=truecanary last, after gemma stabilizes.MODEL_VERSION_FLOORSretires after fleet convergence;V2_VERSION_FLOORstays (permanent fail-loud audit). The fix(coordinator): recalibrate prefill-TPS fallback — cut the 41% long-prompt ttft_429 over-shed #453 anchor and the ratio chain both retire naturally per (model, chip) pair as real measurements take over.cmd/coordinator/main.go,registry/concurrency_cap.go,registry/warm_pool_controller.go, docs): whichever of coord-b/coord-c merges second takes the rebase.Need help on this PR? Tag
/codesmithwith what you need. Autofix is disabled.