fix(registry): per-model solo-TPS quality caps + pooled-KV admission (postmortem layer 6)

[Gajesh2007]

Per-model solo-TPS quality caps and pooled KV admission

Base: master
Final head: 85817ad2 - 17 commits, 12 files, 2,600 additions, 104 deletions
Release role: Merge group A only. Do not deploy the coordinator from this PR by itself.

Problem

The 2026-07-06 Gemma serving incident exposed two coordinator-side admission failures.

First, the quality-concurrency cap used resolvedDecodeTPS(p), the provider-level registration benchmark, for every model on a mixed-model box. A box benchmarked on gpt-oss at 58-93 tok/s therefore granted Gemma caps of 12-23 even though Gemma's own 10-18 tok/s solo rate warrants approximately 1-2 concurrent decodes. The coordinator concentrated 8-11 Gemma requests on boxes that collapse beyond small batches.

Second, freeMemoryAdmits treated each model slot's token budget as private and counted only same-model pending work. In reality, every slot exposes its own committed tokens plus the same physical KV-cache headroom. Co-resident models could spend that shared pool twice during the heartbeat gap, and token-only accounting could not compare models with different KV bytes per token.

Review of the accumulated byte and chip-class follow-ups found three final gaps:

• A cold model with no resident slot was priced only from resident KV rates. A large-KV cold model could therefore be undercharged both as the incoming request and later as coordinator-pending work.
• For an unsampled chip class, organic samples from a faster class were consulted before the operator seed. Fast-class data could override a deliberately conservative cold-start seed and widen the slow class's hard cap.
• Engine V2 can truthfully report a positive KV rate with a zero token budget after its live fleet clamp. The coordinator treated that model-local zero as an omitted legacy budget, so co-resident pooled headroom could admit and advertise a model the provider had declared full.

What changed

Per-model quality caps

• Added a solo-gated store to TPSRegistry. Heartbeats record a solo sample only when the whole provider has at most one running-or-waiting request, and only for the slot that has an actual running decode. Idle slots and queued-only slots cannot repeatedly contribute retained EWMA values.
• Keyed solo samples by chip class (ChipFamily|ChipTier) so an M4 Max rate is not treated as an M4 Pro rate. Cross-class transfer uses the minimum per-class median rather than a sample-weighted fleet median.
• Added resolvedSoloModelTPSLocked as the static rate source for quality caps. The final resolver order is: trusted same-class median; minimum cross-class median bounded above by the configured seed when one exists; the seed when no cross-class median is trusted; then the prior provider-level benchmark behavior. Same-class organic evidence still supersedes the seed, while faster-class data cannot widen an unsampled class above its configured cold-start estimate.
• Kept load-inclusive TPS observations separate. They still feed TTFT and service-time estimation; they never feed the hard quality cap.
• Applied the same resolved cap at routing snapshot selection, quick-capacity preflight, final reservation re-check, public model-capacity snapshots, warm-pool saturation, and warm-target quality math.
• Rejected malformed, non-positive, NaN, and infinite values in the shared per-model float-map parser.
• Preserved EIGENINFERENCE_QUALITY_CAP_PER_MODEL_TPS=false as the kill switch that restores the provider-level benchmark behavior.

Shared KV-pool admission

• Added providerPooledTokenBudget to reconstruct one provider-wide pool from all backend slots. Live used capacity adds across slots; shared free headroom is counted once; committed potential is retained only as the pending de-duplication baseline.
• Normalized the pool and charges into bytes whenever the provider reports KV bytes per token. Per-token rates are clamped before multiplication to prevent malformed heartbeat values from overflowing int64 accounting.
• Added all-model pending totals to both routing snapshot builders. pooledBudgetAdmits now charges every coordinator-pending model before admitting a new request.
• Kept the existing per-slot check for model context limits, then required the request to fit the provider-wide pool as a second condition.
• Applied the pooled check to cold requests as well as resident-slot requests.
• The final readiness fix routes every pooled byte charge through resolvedPooledKVBytesPerToken. A resident model keeps its clamped reported rate; a cold model with no resident slot uses max(400,000 B/token coordinator default, highest resident rate), clamped by the existing 16 MiB/token overflow ceiling. The same policy prices both the incoming cold request and already-pending cold work, so an unknown large-KV model cannot borrow a smaller resident model's rate or disable byte accounting.
• Added saturating pending-byte accumulation with addPooledKVByteCharge, and division-based admission comparisons, so a large pending or incoming charge rejects instead of wrapping an int64 product.
• Distinguished authoritative Engine V2 zero budgets from legacy omission. A positive KV rate preserves a zero maximum as model-local known-full capacity in both freeMemoryAdmits and ModelCapacitySnapshot; another model's pooled headroom cannot widen it. Positive-budget co-residents remain usable, and zero-budget slot rates remain available for pending and capacity math.
• Added pooledRemainingTokens so /v1/models and /v1/models/capacity expose the same byte-aware pooled headroom that admission enforces.
• Preserved legacy behavior for providers reporting neither token budgets nor KV rates, and preserved the arithmetic boundary for single-model providers.

Commit history

Commit	Change
9b69fee7	Per-model solo-TPS quality caps
c397dc46	Provider-wide pooled KV admission
ac482fe0	Reject non-finite per-model tunable values
ef179d08	Sample only the slot that owns the active request
3ba518cc	Normalize heterogeneous co-resident KV use into bytes
ba8ad368	Charge the shared pool on the cold-model path
5edbe165	Clamp public capacity by the pooled admission budget
942429fa	Merge the then-current master into the branch
cd7039cc	Align byte snapshots, running-decode gating, and live-used pool totals
1ddb69db	Clamp KV byte rates before byte multiplication
34b73308	Key solo caps by chip class and make cross-class transfer conservative
8f0a28b8	Keep cold unknown-KV requests in byte accounting
1a4d25e8	Correct cold-path documentation
96b3dc29	Bound cross-class solo-rate transfer by the operator seed
8872e744	Keep incoming and pending cold pooled-KV accounting in bytes
5e5024f8	Merge current master, including #527 capacity-rate accounting
85817ad2	Honor authoritative zero Engine V2 KV budgets

The final merge preserves #527's capacity-rate lifecycle accounting and this PR's pooled-admission and capacity-snapshot behavior without conflicts.

Behavior: Before and after

flowchart TB
  subgraph Before["Before"]
    A1["Gemma request reaches mixed-model provider"] --> B1["Cap uses provider-level gpt-oss benchmark"]
    B1 --> C1["Gemma receives an over-wide concurrency cap"]
    C1 --> D1["Batch grows and per-request decode collapses"]

    E1["Co-resident model requests"] --> F1["Per-slot token checks count same-model pending only"]
    F1 --> G1["Both models spend the same KV headroom"]

    H1["Cold large-KV model has no resident slot"] --> I1["Request and pending work borrow resident KV rates"]
    I1 --> J1["Shared pool is undercharged"]

    K1["Slow chip class has no samples"] --> L1["Fast-class samples run before conservative seed"]
    L1 --> M1["Slow class can receive a fast-class cap"]

    N1["V2 model reports positive KV rate and zero budget"] --> O1["Zero is treated as legacy omission"]
    O1 --> P1["Co-resident pooled headroom admits a known-full model"]
  end

  subgraph After["After"]
    A2["Gemma request reaches mixed-model provider"] --> B2["Cap uses static model and chip-class solo rate"]
    B2 --> C2["Concurrency stays near the model quality floor"]

    E2["Co-resident model requests"] --> F2["Per-slot limit plus provider-wide byte pool"]
    F2 --> G2["All-model pending work is charged once"]

    H2["Cold model has no resident slot"] --> I2["Resolve conservative cold-model KV rate"]
    I2 --> J2["Use the same rate for incoming and pending work"]

    K2["Chip class has no trusted same-class median"] --> L2["Bound cross-class transfer by the operator seed"]
    L2 --> M2["Fallback remains conservative until local samples exist"]

    N2["V2 model reports positive KV rate and zero budget"] --> O2["Treat zero as authoritative model-local capacity"]
    O2 --> P2["Admission and capacity reject it without blocking positive-budget models"]
  end

Loading

Code: Before and after

flowchart LR
  subgraph BeforeCode["Before"]
    H1["Heartbeat"] --> R1["Record load-inclusive TPS"]
    R1 --> D1["resolvedDecodeTPS provider rate"]
    D1 --> C1["effectiveMaxConcurrencyForModelLocked"]

    S1["snapshotProviderLockedEx or quickCapacityCheck"] --> F1["freeMemoryAdmits"]
    F1 --> P1["Per-slot budget plus same-model pending"]
    Z1["Positive KV rate plus zero max"] --> U1["Falls through as budget unavailable"]
    U1 --> F1
  end

  subgraph AfterCode["After"]
    H2["Heartbeat"] --> G2["soloSampleEligible plus NumRunning"]
    G2 --> R2["TPSRegistry.RecordSolo by model and chip class"]
    R2 --> D2["resolvedSoloModelTPSLocked"]
    D2 --> C2["hasConcurrencyHeadroomForModelCapResolvedLocked"]
    C2 --> W2["routing, preflight, final admit, capacity feed, warm pool"]

    S2["snapshotProviderLockedEx or quickCapacityCheck"] --> A2["fillSnapshotPendingAndPool"]
    A2 --> B2["providerPooledTokenBudget"]
    A2 --> K2["resolvedPooledKVBytesPerToken"]
    K2 --> O2["addPooledKVByteCharge saturates pending bytes"]
    B2 --> F2["freeMemoryAdmits plus pooledBudgetAdmits"]
    K2 --> F2
    B2 --> P2["pooledRemainingTokens"]
    K2 --> P2
    P2 --> M2["ModelCapacitySnapshot"]
    Z2["Positive KV rate plus zero max"] --> Q2["knownZeroTokenBudget"]
    Q2 --> F2
    Q2 --> M2
  end

Loading

Configuration

Environment variable	Default	Meaning
EIGENINFERENCE_QUALITY_CAP_PER_MODEL_TPS	true	Use each model's static solo rate for hard quality caps. false restores the provider-level rate at every cap site.
EIGENINFERENCE_QUALITY_CAP_SOLO_MIN_SAMPLES	5	Same-class solo samples required before a median is trusted.
EIGENINFERENCE_MODEL_SOLO_TPS_SEED	empty	Cold-start build-id=tok/s values, matched case-insensitively. For an unsampled class, broader organic data is capped at this seed; trusted same-class samples still take precedence.

These settings are parsed at coordinator startup. Pooled admission has no feature flag.

At the single integrated coordinator deployment, the planned seed remains:

gemma-4-26b-qat-4bit=14,gpt-oss-20b=30

Remove the interim EIGENINFERENCE_QUALITY_CONCURRENCY_OVERCOMMIT_BY_MODEL=gemma-4-26b-qat-4bit=0.15 at that same integrated deployment; this PR supersedes it.

Test coverage

The current diff contains focused regression coverage for:

• The exact mixed-box postmortem case: provider benchmark 93, Gemma solo rate 14, quality floor 15, Gemma cap 2, and gpt-oss cap 23.
• Kill-switch restoration and model-specific capping without a registration benchmark.
• Whole-box solo gating, owner-slot-only recording, running-decode-only recording, chip-class keying, conservative cross-class medians, seed cold start, and sample floors.
• Shared-pool reconstruction, single-model equivalence, legacy-provider behavior, heterogeneous KV rates, live-used byte totals, rate clamping, cold admission, and real reservation-path double-spend prevention.
• Admission and ModelCapacitySnapshot equivalence in token and byte modes.
• Warm-pool separation between static solo rates and load-inclusive service rates.

Readiness regressions added to the final head:

• TestPooledFirstColdRequestUsesConservativeDefault: an incoming cold large-KV request uses max(400,000 B/token default, highest resident rate) rather than the maximum resident rate alone; public pooled remaining uses the same policy.
• TestPooledPendingColdRequestKeepsByteAccounting: already-pending cold work is normalized with the same conservative default policy and cannot force token-mode or optimistic resident-rate accounting for the next request.
• TestQualityCapSeedBoundsCrossClassTransfer: an unsampled slow chip class remains capped by the operator seed even when a faster class has enough organic samples.
• TestResolvedSoloModelTPSFallbackChain: trusted same-class organic samples still supersede the cold-start seed after reaching the sample floor.
• TestPooledKnownZeroBudgetRejects: a single Engine V2 positive-rate/zero-max slot is known-full, while a true legacy slot remains unconstrained.
• TestPooledZeroBudgetResidentRateStaysSymmetric: a mixed provider retains the zero-budget resident's reported rate across pending, admission, and capacity; that model rejects even with abundant pooled headroom while its positive-budget co-resident still admits.

Final verification:

• PASS focused merged regression lane under -race
• PASS go test ./... -race -count=1
• PASS go vet ./...
• PASS go build ./cmd/coordinator
• PASS gofmt -l on changed Go files and git diff --check
• PASS all required GitHub checks on the final PR head: Coordinator Tests, Coordinator Lint, Provider Tests, and Console UI Lint & Build
• NON-BLOCKING INFRA FAILURE the optional E2E workflow hit the known Qwen3.5 MLX concurrent-batching crash (broadcast_shapes: incompatible (5,1) / (20,1) and (3,1) / (6,1) shapes). Current master at 9b1d3b02 failed the same test with the same MLX fatal error in run 29045836408; the PR run is 29048151439. This coordinator-only PR does not modify the provider, MLX submodules, or E2E tests.

Merge and release sequence

This PR is merge group A, not a coordinator deployment unit.

1. Merge fix(registry): per-model solo-TPS quality caps + pooled-KV admission (postmortem layer 6) #524 after its readiness fixes, latest-master sync, review threads, and final checks are complete.
2. Prepare and merge the next stacked coordinator PRs in order: feat(registry,api): harden warm-pool recovery, admission, and runtime shedding #525, then feat(registry): v2 version floors + tripwire, prefill-honest TTFT (absorbs #453), satisficing utilization band [stacked on per-model caps] #526.
3. Run the integrated release gates against the complete coordinator and provider set.
4. Publish and validate the complete one-engine v0.7.5 provider release, including full legacy-engine removal.
5. Canary that complete v0.7.5 release and confirm inference, capacity, upgrade, and rollback gates.
6. Only after the provider canary passes, deploy the complete coordinator once.

Do not partially deploy #524, #525, or #526. Do not deploy the new coordinator before the complete one-engine v0.7.5 provider release has been published, validated, and canaried.

At the eventual integrated deployment, verify mixed-box Gemma caps, per-chip-class sample counts, pooled-budget rejection reasons, public capacity/admission symmetry, and gpt-oss TTFT/TPS. The quality-cap rollback is EIGENINFERENCE_QUALITY_CAP_PER_MODEL_TPS=false; pooled admission requires rolling back the integrated coordinator artifact.

Review notes

• The cold-model KV-rate, seed-precedence, and Engine V2 known-zero findings are fixed with fails-before-fix regressions and an independent final PASS.
• Current master, including fix(routing): correct capacity-rate window accounting #527, is merged at 5e5024f8; final verification ran on descendant 85817ad2.
• This PR must merge before the stacked warm/open-pool and version-floor/TTFT work that depends on these registry seams.

---

^{Need help on this PR? Tag /codesmith with what you need. Autofix is disabled.}

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
d-inference	Ready	Preview	Jul 9, 2026 8:31pm
d-inference-console-ui-dev	Ready	Preview	Jul 9, 2026 8:31pm
d-inference-landing	Ready	Preview	Jul 9, 2026 8:31pm

[github-actions]

This diff makes purely capacity-accounting changes to the provider registry's TPS sampling and token-budget headroom logic; it does not materially strengthen or weaken any security control in the threat model.

---

Trust boundaries touched

• TB-002 (Coordinator → Provider WebSocket): registry.go is the registry admission and routing layer; changes here affect what providers are routable and how capacity is surfaced.
• TB-007 (Provider Inference Engine): ModelCapacitySnapshot feeds the routing/dispatch layer that sits between consumers and providers.

---

Threat assessment

Threat	Finding
T-006 — Unauthenticated WS floods registry	ℹ️ No change to registration admission path or pre-auth gating.
T-007 / T-012 / T-027 — Weight-hash enforcement (fail-open, SEC-007)	ℹ️ Untouched. knownZeroTokenBudget and soloSampleEligible are new capacity helpers; no code path in this diff adds or removes weight-hash checks. Fail-open remains as documented.
T-009 / T-033 / T-034 — Attestation chain / code-identity gate	ℹ️ providerSupportsPrivateTextLocked / CodeAttested gate not modified. The new pooledBudgetRemaining field and tokenBudgetKnownZero flag are only read in the hasBudgetHeadroom expression (line ~4706) and the headroom clamp (line ~4689). Neither touches the 11-flag privacy routing gate.
T-015 / T-036 — Trust level elevation / SIP bypass	ℹ️ Trust state machine (MarkUntrusted, TrustHardware, challenge loop) is not modified.

---

New attack surface / observations

1. pooledRemainingTokens return value of -1 flows into hasBudgetHeadroom — confirm sentinel handling is correct (low risk, worth a unit-test)

At line ~4706:

s.pooledBudgetRemaining != 0

The comment documents that -1 means "legacy, no pooled budget". With the new expression, a legacy provider (pooledBudgetRemaining == -1) satisfies != 0, so the pooled-budget check passes — intentionally. A provider whose pool is genuinely at 0 (exhausted) is correctly blocked. This is the intended semantics per the comment, but the sentinel value -1 meaning "unconstrained" is a fragile convention: if pooledRemainingTokens ever returns -1 due to an arithmetic underflow on a non-legacy box, that box would be treated as unconstrained rather than exhausted. Recommend a named constant or a dedicated bool field alongside pooledBudgetRemaining to make the sentinel contract explicit (similar to the existing tokenBudgetKnownZero bool pattern used in the same diff). This is not a threat-model gap but is a correctness/auditability concern.

2. soloSampleEligible contention check is evaluated once per heartbeat, outside the per-slot loop

soloEligible is computed once for the whole BackendCapacity snapshot, then applied to each slot. If soloSampleEligible inspects fleet-wide or multi-slot state, a provider could potentially craft heartbeat messages to selectively pass the soloEligible gate while one slot is running and others are draining (influencing the resolvedSoloModelTPSLocked quality cap). This affects scheduling fairness, not security posture — not a STRIDE gap but worth confirming soloSampleEligible only reads provider-local data.

---

SEC-* findings resolved by this PR

None. Open findings SEC-007, SEC-034, SEC-011, SEC-004, SEC-012, SEC-017, SEC-021, and others remain unaddressed.

---

Summary

No security mitigations are added, removed, or weakened. The diff is a scheduling/capacity correctness fix. The only items worth a second look are the -1 sentinel convention for pooledBudgetRemaining (prefer an explicit bool or named constant) and confirming soloSampleEligible is purely local-state to prevent a provider from gaming the solo-TPS quality cap via crafted heartbeats.

---

🔐 Threat model: docs/threat-model.yaml · Updates on each push to this PR

[blacksmith-sh]

Found 2 test failures on Blacksmith runners:

Failures

Test	View Logs
github.com/eigeninference/d-inference/e2e/TestIntegration_ConcurrentRequests	View Logs
github.com/eigeninference/d-inference/e2e/TestProfile_SingleProviderNonStreaming	View Logs

^{Need help on this PR? Tag /codesmith with what you need.}

[ethenotethan]

Automated Code Review — Layr-Labs/d-inference#

Verdict: REQUEST_CHANGES

Security — 2 finding(s)

• 🔵 [INFO] coordinator/registry/concurrency_cap.go:46-65 — Environment variable parsing lacks validation for malicious input
  • Suggestion: Add input validation and sanitization for environment variables before parsing
• 🔵 [INFO] coordinator/registry/concurrency_cap.go:143-162 — CSV parsing vulnerable to injection through model names
  • Suggestion: Validate and sanitize model names in CSV parsing to prevent injection attacks

Performance — 1 finding(s) (1 blocking)

• 🟡 [MEDIUM] coordinator/registry/scheduler.go:1079-1089 — Unbounded token accumulation in pendingMaxTokensAllModels without pre-allocation
  • Suggestion: Pre-allocate pendingMaxTokensAllModels based on len(p.pendingReqs) to avoid repeated slice growth: snap.pendingMaxTokensAllModels = make([]int, 0, len(p.pendingReqs)) or accumulate directly as int without intermediate slice operations

Type_diligence — 1 finding(s) (1 blocking)

• 🟡 [MEDIUM] coordinator/registry/concurrency_cap.go:139 — map[string]float64 lookup without ok check in qualityCapOvercommitForModelLocked
  • Suggestion: Use two-value assignment: if overcommit, ok := qualityCapOvercommitByModel[strings.ToLower(model)]; ok { return overcommit }

Additive_complexity — 3 finding(s) (2 blocking)

• 🔵 [INFO] coordinator/registry/concurrency_cap.go:37-40 — qualityCapOvercommitByModel and modelSoloTPSSeed both use parseModelFloatMap but have separate variables
  • Suggestion: Consider consolidating the parsing logic or using a generic map type to reduce duplication of similar functionality
• 🟡 [MEDIUM] coordinator/registry/pooled_admission.go:1-77 — New pooled admission system adds significant complexity with overlapping responsibility to existing per-slot checks
  • Suggestion: Consider if this could be integrated into existing admission logic rather than adding a parallel system
• 🟡 [MEDIUM] coordinator/registry/solo_tps.go:1-107 — Entire new solo TPS tracking system duplicates existing TPS registry functionality with different gating logic
  • Suggestion: Evaluate if the existing TPS registry could be extended with gating flags rather than creating a parallel tracking system

7 finding(s) total, 4 blocking. Verdict: REQUEST_CHANGES.

🤖 Automated review by Centaur · DAR-186

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c397dc46ea

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[Gajesh2007]

Review round addressed @ethenotethan — ready for re-review.

• Fixed: non-finite env values rejected in per-model tunable parsing (ac482fe); solo-TPS samples now only from the slot owning the box's one active request — no stale-EWMA contamination (ef179d0); pooled-KV admission byte-normalized across heterogeneous co-resident KV rates, token-fallback for legacy (3ba518c); cold-model path now charges the shared pool (ba8ad36); /v1/models capacity snapshot clamped by the same pooled budget the admission gate enforces (5edbe16). Every fix carries a regression test verified failing pre-fix; full go test ./... -race green.
• Declined with evidence (per-thread replies): pendingMaxTokensAllModels is an int, not a slice; both map lookups already use comma-ok; the two tunables already share one parser; pooled admission is wired into the existing freeMemoryAdmits seam (per-slot fields are structurally blind to the shared pool — the postmortem double-spend); solo store is a deliberate second keyed store on the same TPSRegistry (solo-condition vs under-load semantics).
• CI: E2E TestIntegration_ConcurrentRequests = known master-reproducing MLX broadcast_shapes flake; Provider Tests = known StartupPreloadTests timing flake (2.87s vs 2.8s expectation) — this branch touches no provider-swift code. The codex check failure is workflow infra (the codex bot actor lacks write permission for the review workflow), not this branch.

[ethenotethan]

Automated Code Review — Layr-Labs/d-inference#

Verdict: REQUEST_CHANGES

Security — 1 finding(s)

• 🔵 [INFO] coordinator/registry/concurrency_cap.go:164 — Missing input validation on math.IsNaN and math.IsInf checks
  • Suggestion: The parseModelFloatMap function checks for NaN and Inf values but doesn't validate that strconv.ParseFloat succeeded before calling these functions on potentially invalid float64 values

Performance — 1 finding(s) (1 blocking)

• 🟡 [MEDIUM] coordinator/registry/pooled_admission.go:73-116 — providerPooledTokenBudget creates unbounded map allocation inside loop without pre-allocation
  • Suggestion: Pre-allocate kvBytesPerToken map with capacity len(slots) before the loop to avoid repeated map growth

Type_diligence — 1 finding(s) (1 blocking)

• 🟡 [MEDIUM] coordinator/registry/concurrency_cap.go:164 — math.IsNaN and math.IsInf calls missing import for math package
  • Suggestion: Add 'import "math"' to the imports section or use qualified calls if math is imported elsewhere

Additive_complexity — ✅ No issues found

3 finding(s) total, 2 blocking. Verdict: REQUEST_CHANGES.

🤖 Automated review by Centaur · DAR-186

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 942429fa98

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[ethenotethan]

Automated Code Review — Layr-Labs/d-inference#

Verdict: COMMENT

Security — ⚠️ SKIPPED

Error: OpenRouter HTTP 402: {"error":{"message":"Insufficient credits. Add more using https://openrouter.ai/settings/credits","code":402}}

Performance — ⚠️ SKIPPED

Error: OpenRouter HTTP 402: {"error":{"message":"Insufficient credits. Add more using https://openrouter.ai/settings/credits","code":402}}

Type_diligence — ⚠️ SKIPPED

Error: OpenRouter HTTP 402: {"error":{"message":"Insufficient credits. Add more using https://openrouter.ai/settings/credits","code":402}}

Additive_complexity — ⚠️ SKIPPED

Error: OpenRouter HTTP 402: {"error":{"message":"Insufficient credits. Add more using https://openrouter.ai/settings/credits","code":402}}

✅ All four passes clean. No issues found.

🤖 Automated review by Centaur · DAR-186

[ethenotethan]

Automated Code Review — Layr-Labs/d-inference#

Verdict: COMMENT

Security — ⚠️ SKIPPED

Error: OpenRouter HTTP 402: {"error":{"message":"Insufficient credits. Add more using https://openrouter.ai/settings/credits","code":402}}

Performance — ⚠️ SKIPPED

Error: OpenRouter HTTP 402: {"error":{"message":"Insufficient credits. Add more using https://openrouter.ai/settings/credits","code":402}}

Type_diligence — ⚠️ SKIPPED

Error: OpenRouter HTTP 402: {"error":{"message":"Insufficient credits. Add more using https://openrouter.ai/settings/credits","code":402}}

Additive_complexity — ⚠️ SKIPPED

Error: OpenRouter HTTP 402: {"error":{"message":"Insufficient credits. Add more using https://openrouter.ai/settings/credits","code":402}}

✅ All four passes clean. No issues found.

🤖 Automated review by Centaur · DAR-186

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1a4d25e83c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[ethenotethan]

Automated Code Review — Layr-Labs/d-inference#

Verdict: REQUEST_CHANGES

Security — 4 finding(s) (1 blocking)

• 🟡 [MEDIUM] coordinator/registry/pooled_admission.go:47 — maxKVBytesPerToken constant may be insufficient for legitimate use cases
  • Suggestion: Consider making this configurable or document the rationale for 16 MiB limit
• 🔵 [INFO] coordinator/registry/pooled_admission.go:95-105 — Integer overflow protection relies on saturation at MaxInt64
  • Suggestion: Add explicit overflow checks before multiplication
• 🔵 [INFO] coordinator/registry/concurrency_cap.go:164-170 — parseModelFloatMap filters NaN/Inf but may miss edge cases
  • Suggestion: Consider using math.IsNaN and math.IsInf consistently throughout codebase
• 🔵 [INFO] coordinator/registry/solo_tps.go:41-44 — RecordSolo accepts empty chipClass parameter
  • Suggestion: Add validation to reject empty chipClass parameter

Performance — 3 finding(s) (2 blocking)

• 🟡 [MEDIUM] coordinator/registry/pooled_admission.go:101-107 — Potential overflow in addPooledKVByteCharge multiplication without bounds checking
  • Suggestion: Add explicit overflow check before multiplication: if tokens > (math.MaxInt64-total)/rate { return math.MaxInt64 }
• 🟡 [MEDIUM] coordinator/registry/pooled_admission.go:195-210 — Multiple overflow-prone multiplications in pooled budget reconstruction
  • Suggestion: Use addPooledKVByteCharge helper for all token*rate calculations to prevent overflow
• 🔵 [INFO] coordinator/registry/concurrency_cap.go:164 — Map allocation without size hint in parseModelFloatMap
  • Suggestion: Pre-allocate map with estimated size: out := make(map[string]float64, len(pairs))

Type_diligence — 4 finding(s) (4 blocking)

• 🟡 [MEDIUM] coordinator/registry/concurrency_cap.go:164-167 — math.IsNaN and math.IsInf checks without proper error handling for strconv.ParseFloat
  • Suggestion: Add explicit type assertion check for the ParseFloat result before using math.IsNaN/IsInf, or handle the error case more explicitly to ensure v is a valid float64
• 🟡 [MEDIUM] coordinator/registry/pooled_admission.go:104-107 — Integer overflow check uses division without verifying divisor is non-zero
  • Suggestion: Add explicit check for rate > 0 before the division: if rate <= 0 { return total }; if tokens > (math.MaxInt64-total)/rate { return math.MaxInt64 }
• 🟡 [MEDIUM] coordinator/registry/pooled_admission.go:271-274 — Map access without checking if key exists in kvBytesPerToken
  • Suggestion: Use safe map access pattern: if rate, ok := pool.kvBytesPerToken[pr.Model]; ok { ... } else { rate = resolvedPooledKVBytesPerToken(...) }
• 🟡 [MEDIUM] coordinator/registry/pooled_admission.go:315-318 — Map access without checking if modelRate key exists
  • Suggestion: Add bounds check or use safe map access pattern for pool.kvBytesPerToken[modelRate] lookup

Additive_complexity — 5 finding(s) (2 blocking)

• 🔵 [INFO] coordinator/registry/concurrency_cap.go:37-80 — Multiple new environment variables added for per-model TPS configuration
  • Suggestion: Consider consolidating these configuration options or providing a single configuration file approach instead of multiple environment variables
• 🟡 [MEDIUM] coordinator/registry/pooled_admission.go:1-331 — Entire new file with complex pooled admission logic duplicating existing admission patterns
  • Suggestion: Consider refactoring existing admission logic to accommodate pooled budgets rather than creating parallel admission paths
• 🟡 [MEDIUM] coordinator/registry/solo_tps.go:1-156 — New solo TPS tracking system parallel to existing TPS registry
  • Suggestion: Consider extending the existing TPSRegistry with solo tracking capabilities rather than creating a separate parallel system
• 🔵 [INFO] coordinator/registry/scheduler.go:1234-1268 — fillSnapshotPendingAndPool duplicates pending request aggregation logic
  • Suggestion: Extract common pending request aggregation into a shared helper method
• 🔵 [INFO] coordinator/registry/concurrency_cap.go:185-241 — soloModelTPS struct with single field and complex resolver chain
  • Suggestion: Consider simplifying the resolver chain or using a more direct approach for rate resolution

16 finding(s) total, 9 blocking. Verdict: REQUEST_CHANGES.

🤖 Automated review by Centaur · DAR-186

Dismissed after all associated threads were addressed or resolved on later commits. The PR is now at 85817ad, independently reviewed, and the final coordinator race suite, vet, build, and required checks pass.