feat(coordinator): route Gemma 4 only to dedicated providers; 429 shed#434
Conversation
Restrict a dedicated model family (default: gemma-4, via EIGENINFERENCE_DEDICATED_MODELS) to providers whose entire advertised catalog is that family. Enforced via one shared predicate (providerExcludedByDedicatedRuleLocked) at every public selection site: the routing gate (providerPassesRoutingGatesLockedEx, covering both the dispatch hot path and the OpenRouter capacity preflight), cold-spill, the warm-pool target picker, and the model-swap planner/warm-detection — so routing, shedding, and pre-warming can never drift. Self-route to one's own machine is exempt. When the fleet serves the model but no dedicated box is available, shed to OpenRouter as a transient 429 + Retry-After instead of 503.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
This PR introduces dedicated-box routing isolation for specific model families (e.g. Gemma-4); it is operationally significant but has no material security impact on the trust model — one minor trust-boundary interaction deserves a brief note. Trust Boundaries Touched
Threat Assessment
New Attack Surface —
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a5fc81f361
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…ore-shed) Dedicated-family models (e.g. Gemma 4) bypass the queue-before-shed window when their dedicated boxes are at capacity: holding an OpenRouter request up to 120s would blow its TTFT SLA, so shed immediately with a 429 + Retry-After for a clean failover. Applied in both handleChatCompletions and handleGenericInference. Non-dedicated models keep queue-before-shed unchanged. Addresses Codex PR review (saturated-dedicated queue-before-shed). The structurally-unroutable 429-vs-503 nit is intentionally left at 429 per product decision (clean OpenRouter failover over an honest 503).
|
Thanks @chatgpt-codex-connector — addressed both findings in ① Shed saturated dedicated models before queueing — ✅ fixed. Dedicated-family models now bypass queue-before-shed when their dedicated boxes are saturated ( ② Apply routing gates before treating a provider as available — intentionally left at 429 (product decision). For a dedicated model, we prefer a transient Net effect: every "can't serve a dedicated model right now" case (no dedicated box, or dedicated boxes saturated) now sheds as 429 + Retry-After; only a model genuinely absent from the fleet returns 503. |
Summary
Testing
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f28e89a94b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
providerCanRouteBuildLocked (used by alias resolution, constrained resolution, and RoutableProviderIDsForBuild) now applies providerExcludedByDedicatedRuleLocked, so alias routability matches dispatch routability. Fixes staged-rollout failover: an alias whose Desired build is advertised only by a mixed box now correctly resolves to a Previous build on a dedicated box instead of resolving Desired and 429ing at dispatch. Owner self-route (allowPrivate) stays exempt. Addresses Codex PR review (alias routability gate).
|
@chatgpt-codex-connector finding ③ (alias routability gate) — ✅ fixed in
All three review threads are now addressed — ① fast-429 when saturated ( |
Summary
Testing / Checks
|
ethenotethan
left a comment
ethenotethan
left a comment
There was a problem hiding this comment.
Automated Code Review — Layr-Labs/d-inference#
Verdict: REQUEST_CHANGES
Security — ✅ No issues found
Performance — 4 finding(s) (2 blocking)
- 🟡 [MEDIUM]
coordinator/api/consumer.go:2104-2141— Duplicate rejection recording and response writing logic between chat completions and generic inference handlers- Suggestion: Extract the dedicated model 429 response logic into a shared helper function to eliminate code duplication and reduce maintenance burden
- 🟡 [MEDIUM]
coordinator/api/consumer.go:4931-4968— Identical rejection recording and response writing logic duplicated from chat completions handler- Suggestion: Create a shared helper function for dedicated model 429 responses that both handlers can call, reducing code duplication by ~40 lines
- 🔵 [INFO]
coordinator/registry/dedicated_models.go:122-141— O(n) linear scan through all providers for each HasProviderForModel call- Suggestion: Consider adding an index by model ID to avoid scanning all providers, or cache results if this method is called frequently during request processing
- 🔵 [INFO]
coordinator/registry/dedicated_models.go:123-127— Map allocation for allowedSerials set on every call without pre-sizing- Suggestion: Pre-allocate the map with make(map[string]struct{}, len(allowedSerials)) to avoid potential rehashing during insertion
Type_diligence — ✅ No issues found
Additive_complexity — 5 finding(s) (2 blocking)
- 🟡 [MEDIUM]
coordinator/api/consumer.go:2104-2141— Identical 38-line dedicated model shed logic duplicated between handleChatCompletions and handleGenericInference- Suggestion: Extract the dedicated model 429 shed logic into a shared helper method to eliminate duplication and ensure consistent behavior
- 🟡 [MEDIUM]
coordinator/api/consumer.go:4931-4968— Second instance of identical dedicated model shed logic - exact copy of lines 2104-2141- Suggestion: Use the same extracted helper method for both endpoints to maintain DRY principle
- 🔵 [INFO]
coordinator/registry/dedicated_models.go:17-28— ParseDedicatedModels reimplements basic CSV parsing with manual string splitting and trimming- Suggestion: Consider using encoding/csv package or at minimum extract this as a generic utility function if the simple splitting is intentional
- 🔵 [INFO]
coordinator/registry/dedicated_models.go:122-142— HasProviderForModel method has complex logic but only used for distinguishing 429 vs 503 error codes- Suggestion: Consider inlining this logic at the call site or simplifying the interface since it's a single-purpose helper
- 🔵 [INFO]
coordinator/registry/dedicated_models_test.go:1-400— 400-line test file with 15+ test functions for a relatively simple substring matching feature- Suggestion: Consider consolidating related test cases into table-driven tests to reduce test code volume while maintaining coverage
9 finding(s) total, 4 blocking. Verdict: REQUEST_CHANGES.
🤖 Automated review by Centaur · DAR-186
ethenotethan
left a comment
ethenotethan
left a comment
There was a problem hiding this comment.
Automated Code Review — Layr-Labs/d-inference#
Verdict: REQUEST_CHANGES
Security — ✅ No issues found
Performance — 4 finding(s) (1 blocking)
- 🔵 [INFO]
coordinator/api/consumer.go:2105-2141— Duplicate rejection recording and response writing logic- Suggestion: Extract common rejection handling into a shared function to avoid code duplication between handleChatCompletions and handleGenericInference
- 🔵 [INFO]
coordinator/api/consumer.go:4932-4968— Duplicate rejection recording and response writing logic- Suggestion: Extract common rejection handling into a shared function to avoid code duplication between handleChatCompletions and handleGenericInference
- 🟡 [MEDIUM]
coordinator/registry/dedicated_models.go:122-141— HasProviderForModel iterates all providers with nested locking- Suggestion: Consider caching provider-model mappings or using a more efficient lookup structure to avoid O(n) iteration with nested locks on hot path
- 🔵 [INFO]
coordinator/registry/dedicated_models.go:17-27— ParseDedicatedModels creates slice without pre-allocation- Suggestion: Pre-allocate slice with capacity len(strings.Split(csv, ',')) to avoid repeated allocations during append operations
Type_diligence — ✅ No issues found
Additive_complexity — 4 finding(s) (2 blocking)
- 🟡 [MEDIUM]
coordinator/api/consumer.go:2105-2141— Identical 37-line rejection handling block duplicated between chat completions and generic inference- Suggestion: Extract the dedicated model rejection logic into a shared helper method like handleDedicatedModelRejection(w, r, model, publicModel, retryAfter, refundReservation)
- 🟡 [MEDIUM]
coordinator/api/consumer.go:4932-4968— Second instance of identical 37-line rejection handling block- Suggestion: Use the same extracted helper method as suggested for the first instance
- 🔵 [INFO]
coordinator/registry/dedicated_models.go:73-85— providerDedicatedToPatternLocked could be simplified by combining the loop logic- Suggestion: Consider inlining this into the single caller or simplifying the advertised counter logic
- 🔵 [INFO]
coordinator/registry/dedicated_models.go:122-142— HasProviderForModel in registry layer duplicates routing logic that belongs in scheduler- Suggestion: Move this method to scheduler.go or make it use existing scheduler methods to avoid duplicating provider filtering logic
8 finding(s) total, 3 blocking. Verdict: REQUEST_CHANGES.
🤖 Automated review by Centaur · DAR-186
| } else if s.registry.IsDedicatedModel(model) && s.registry.HasProviderForModel(model, allowedProviderSerials...) { | ||
| // Dedicated-box model (e.g. Gemma 4): the fleet DOES serve this | ||
| // model, but no provider DEDICATED to it can take the request right | ||
| // now — either none are dedicated, or the dedicated ones are busy/ | ||
| // cooling. That is transient capacity pressure, not an absent model, | ||
| // so shed to OpenRouter as a 429 + Retry-After (clean failover) | ||
| // rather than a 503 (which can get the endpoint marked unhealthy / | ||
| // deranked). Mirrors the capacity_429 path above. | ||
| retryAfter := s.estimateRetryAfter(model) | ||
| w.Header().Set("Retry-After", strconv.Itoa(retryAfter)) | ||
| refundReservation() | ||
| s.ddIncr("routing.decisions", []string{"model:" + model, "model_type:" + s.registry.ModelType(model), "outcome:dedicated_capacity_429"}) | ||
| s.recordRejection(rejectionInfo{ | ||
| r: r, | ||
| stage: "preflight_capacity", | ||
| reasonCode: "machine_busy", | ||
| httpStatus: http.StatusTooManyRequests, | ||
| keyID: keyIDFromContext(r.Context()), | ||
| consumerKeyHash: store.HashKey(consumerKeyFromContext(r.Context())), | ||
| requestedModel: publicModel, | ||
| resolvedModel: model, | ||
| stream: stream, | ||
| estimatedPromptTokens: estimatedPromptTokens, | ||
| requestedMaxTokens: requestedMaxTokens, | ||
| requiresVision: requiresVision, | ||
| hasTools: hasTools, | ||
| retryAfterMs: retryAfter * 1000, | ||
| params: rejectionSamplingParams(parsed), | ||
| servabilityComputed: true, | ||
| candidateCount: candidateCount, | ||
| capacityRejections: capacityRejections, | ||
| modelTooLargeRejections: modelTooLarge, | ||
| bestTTFTMs: ttftMsForRejection(bestTTFT, hasTTFT), | ||
| }) | ||
| writeJSON(w, http.StatusTooManyRequests, errorResponse("rate_limit_exceeded", | ||
| fmt.Sprintf("no provider dedicated to model %q is available right now — retry after %ds", publicModel, retryAfter), | ||
| withCode("rate_limit_exceeded"))) |
There was a problem hiding this comment.
🔵 [INFO] ⚡ Duplicate rejection recording and response writing logic
💡 Suggestion: Extract common rejection handling into a shared function to avoid code duplication between handleChatCompletions and handleGenericInference
📊 Score: 2×3 = 6 · Category: repeated_work
| } else if s.registry.IsDedicatedModel(model) && s.registry.HasProviderForModel(model, allowedProviderSerials...) { | ||
| // Dedicated-box model served by the fleet but no dedicated box is | ||
| // available right now: transient capacity, so shed with 429 + | ||
| // Retry-After (not 503). Mirrors the chat-completions preflight so | ||
| // /v1/completions and /v1/messages classify the shed identically. | ||
| retryAfter := s.estimateRetryAfter(model) | ||
| w.Header().Set("Retry-After", strconv.Itoa(retryAfter)) | ||
| refundReservation() | ||
| s.ddIncr("routing.decisions", []string{"model:" + model, "model_type:" + s.registry.ModelType(model), "outcome:dedicated_capacity_429"}) | ||
| s.recordRejection(rejectionInfo{ | ||
| r: r, | ||
| stage: "preflight_capacity", | ||
| reasonCode: "machine_busy", | ||
| httpStatus: http.StatusTooManyRequests, | ||
| keyID: keyIDFromContext(r.Context()), | ||
| consumerKeyHash: store.HashKey(consumerKeyFromContext(r.Context())), | ||
| requestedModel: publicModel, | ||
| resolvedModel: model, | ||
| stream: stream, | ||
| estimatedPromptTokens: estimatedPromptTokens, | ||
| requestedMaxTokens: requestedMaxTokens, | ||
| requiresVision: requiresVision, | ||
| hasTools: hasTools, | ||
| retryAfterMs: retryAfter * 1000, | ||
| params: rejectionSamplingParams(parsed), | ||
| servabilityComputed: true, | ||
| candidateCount: candidateCount, | ||
| capacityRejections: capacityRejections, | ||
| modelTooLargeRejections: modelTooLarge, | ||
| bestTTFTMs: ttftMsForRejection(bestTTFT, hasTTFT), | ||
| }) | ||
| writeJSON(w, http.StatusTooManyRequests, errorResponse("rate_limit_exceeded", | ||
| fmt.Sprintf("no provider dedicated to model %q is available right now — retry after %ds", publicModel, retryAfter), | ||
| withCode("rate_limit_exceeded"))) | ||
| return | ||
| } else { | ||
| // Queueing cannot help — fail fast with a retryable 503 instead of |
There was a problem hiding this comment.
🔵 [INFO] ⚡ Duplicate rejection recording and response writing logic
💡 Suggestion: Extract common rejection handling into a shared function to avoid code duplication between handleChatCompletions and handleGenericInference
📊 Score: 2×3 = 6 · Category: repeated_work
| func (r *Registry) HasProviderForModel(model string, allowedSerials ...string) bool { | ||
| allowedSet := make(map[string]struct{}, len(allowedSerials)) | ||
| for _, s := range allowedSerials { | ||
| allowedSet[s] = struct{}{} | ||
| } | ||
| r.mu.RLock() | ||
| defer r.mu.RUnlock() | ||
| for _, p := range r.providers { | ||
| if len(allowedSet) > 0 && !providerMatchesAllowedSerial(p, allowedSet) { | ||
| continue | ||
| } | ||
| p.mu.Lock() | ||
| eligible := p.Status != StatusOffline && p.Status != StatusUntrusted && | ||
| r.providerServesCatalogModelLocked(p, model) | ||
| p.mu.Unlock() | ||
| if eligible { | ||
| return true | ||
| } | ||
| } | ||
| return false |
There was a problem hiding this comment.
🟡 [MEDIUM] ⚡ HasProviderForModel iterates all providers with nested locking
💡 Suggestion: Consider caching provider-model mappings or using a more efficient lookup structure to avoid O(n) iteration with nested locks on hot path
📊 Score: 3×4 = 12 · Category: blocking_io
| // or all-blank input yields a nil slice (feature disabled). | ||
| func ParseDedicatedModels(csv string) []string { | ||
| var out []string | ||
| for _, p := range strings.Split(csv, ",") { | ||
| p = strings.ToLower(strings.TrimSpace(p)) | ||
| if p == "" { | ||
| continue | ||
| } | ||
| out = append(out, p) | ||
| } | ||
| return out |
There was a problem hiding this comment.
🔵 [INFO] ⚡ ParseDedicatedModels creates slice without pre-allocation
💡 Suggestion: Pre-allocate slice with capacity len(strings.Split(csv, ',')) to avoid repeated allocations during append operations
📊 Score: 2×2 = 4 · Category: unbounded_allocations
| } else if s.registry.IsDedicatedModel(model) && s.registry.HasProviderForModel(model, allowedProviderSerials...) { | ||
| // Dedicated-box model (e.g. Gemma 4): the fleet DOES serve this | ||
| // model, but no provider DEDICATED to it can take the request right | ||
| // now — either none are dedicated, or the dedicated ones are busy/ | ||
| // cooling. That is transient capacity pressure, not an absent model, | ||
| // so shed to OpenRouter as a 429 + Retry-After (clean failover) | ||
| // rather than a 503 (which can get the endpoint marked unhealthy / | ||
| // deranked). Mirrors the capacity_429 path above. | ||
| retryAfter := s.estimateRetryAfter(model) | ||
| w.Header().Set("Retry-After", strconv.Itoa(retryAfter)) | ||
| refundReservation() | ||
| s.ddIncr("routing.decisions", []string{"model:" + model, "model_type:" + s.registry.ModelType(model), "outcome:dedicated_capacity_429"}) | ||
| s.recordRejection(rejectionInfo{ | ||
| r: r, | ||
| stage: "preflight_capacity", | ||
| reasonCode: "machine_busy", | ||
| httpStatus: http.StatusTooManyRequests, | ||
| keyID: keyIDFromContext(r.Context()), | ||
| consumerKeyHash: store.HashKey(consumerKeyFromContext(r.Context())), | ||
| requestedModel: publicModel, | ||
| resolvedModel: model, | ||
| stream: stream, | ||
| estimatedPromptTokens: estimatedPromptTokens, | ||
| requestedMaxTokens: requestedMaxTokens, | ||
| requiresVision: requiresVision, | ||
| hasTools: hasTools, | ||
| retryAfterMs: retryAfter * 1000, | ||
| params: rejectionSamplingParams(parsed), | ||
| servabilityComputed: true, | ||
| candidateCount: candidateCount, | ||
| capacityRejections: capacityRejections, | ||
| modelTooLargeRejections: modelTooLarge, | ||
| bestTTFTMs: ttftMsForRejection(bestTTFT, hasTTFT), | ||
| }) | ||
| writeJSON(w, http.StatusTooManyRequests, errorResponse("rate_limit_exceeded", | ||
| fmt.Sprintf("no provider dedicated to model %q is available right now — retry after %ds", publicModel, retryAfter), | ||
| withCode("rate_limit_exceeded"))) |
There was a problem hiding this comment.
🟡 [MEDIUM] 🧩 Identical 37-line rejection handling block duplicated between chat completions and generic inference
💡 Suggestion: Extract the dedicated model rejection logic into a shared helper method like handleDedicatedModelRejection(w, r, model, publicModel, retryAfter, refundReservation)
📊 Score: 3×4 = 12 · Category: duplicate logic
| } else if s.registry.IsDedicatedModel(model) && s.registry.HasProviderForModel(model, allowedProviderSerials...) { | ||
| // Dedicated-box model served by the fleet but no dedicated box is | ||
| // available right now: transient capacity, so shed with 429 + | ||
| // Retry-After (not 503). Mirrors the chat-completions preflight so | ||
| // /v1/completions and /v1/messages classify the shed identically. | ||
| retryAfter := s.estimateRetryAfter(model) | ||
| w.Header().Set("Retry-After", strconv.Itoa(retryAfter)) | ||
| refundReservation() | ||
| s.ddIncr("routing.decisions", []string{"model:" + model, "model_type:" + s.registry.ModelType(model), "outcome:dedicated_capacity_429"}) | ||
| s.recordRejection(rejectionInfo{ | ||
| r: r, | ||
| stage: "preflight_capacity", | ||
| reasonCode: "machine_busy", | ||
| httpStatus: http.StatusTooManyRequests, | ||
| keyID: keyIDFromContext(r.Context()), | ||
| consumerKeyHash: store.HashKey(consumerKeyFromContext(r.Context())), | ||
| requestedModel: publicModel, | ||
| resolvedModel: model, | ||
| stream: stream, | ||
| estimatedPromptTokens: estimatedPromptTokens, | ||
| requestedMaxTokens: requestedMaxTokens, | ||
| requiresVision: requiresVision, | ||
| hasTools: hasTools, | ||
| retryAfterMs: retryAfter * 1000, | ||
| params: rejectionSamplingParams(parsed), | ||
| servabilityComputed: true, | ||
| candidateCount: candidateCount, | ||
| capacityRejections: capacityRejections, | ||
| modelTooLargeRejections: modelTooLarge, | ||
| bestTTFTMs: ttftMsForRejection(bestTTFT, hasTTFT), | ||
| }) | ||
| writeJSON(w, http.StatusTooManyRequests, errorResponse("rate_limit_exceeded", | ||
| fmt.Sprintf("no provider dedicated to model %q is available right now — retry after %ds", publicModel, retryAfter), | ||
| withCode("rate_limit_exceeded"))) | ||
| return | ||
| } else { | ||
| // Queueing cannot help — fail fast with a retryable 503 instead of |
There was a problem hiding this comment.
🟡 [MEDIUM] 🧩 Second instance of identical 37-line rejection handling block
💡 Suggestion: Use the same extracted helper method as suggested for the first instance
📊 Score: 3×4 = 12 · Category: duplicate logic
| // Caller holds r.mu AND p.mu (mirrors providerServesCatalogModelLocked). | ||
| func (r *Registry) providerDedicatedToPatternLocked(p *Provider, pattern string) bool { | ||
| advertised := 0 | ||
| for _, m := range p.Models { | ||
| if !r.modelAllowedByCatalogLocked(m) { | ||
| continue | ||
| } | ||
| advertised++ | ||
| if !strings.Contains(strings.ToLower(m.ID), pattern) { | ||
| return false | ||
| } | ||
| } | ||
| return advertised > 0 |
There was a problem hiding this comment.
🔵 [INFO] 🧩 providerDedicatedToPatternLocked could be simplified by combining the loop logic
💡 Suggestion: Consider inlining this into the single caller or simplifying the advertised counter logic
📊 Score: 2×3 = 6 · Category: over-abstraction
| func (r *Registry) HasProviderForModel(model string, allowedSerials ...string) bool { | ||
| allowedSet := make(map[string]struct{}, len(allowedSerials)) | ||
| for _, s := range allowedSerials { | ||
| allowedSet[s] = struct{}{} | ||
| } | ||
| r.mu.RLock() | ||
| defer r.mu.RUnlock() | ||
| for _, p := range r.providers { | ||
| if len(allowedSet) > 0 && !providerMatchesAllowedSerial(p, allowedSet) { | ||
| continue | ||
| } | ||
| p.mu.Lock() | ||
| eligible := p.Status != StatusOffline && p.Status != StatusUntrusted && | ||
| r.providerServesCatalogModelLocked(p, model) | ||
| p.mu.Unlock() | ||
| if eligible { | ||
| return true | ||
| } | ||
| } | ||
| return false | ||
| } |
There was a problem hiding this comment.
🔵 [INFO] 🧩 HasProviderForModel in registry layer duplicates routing logic that belongs in scheduler
💡 Suggestion: Move this method to scheduler.go or make it use existing scheduler methods to avoid duplicating provider filtering logic
📊 Score: 2×2 = 4 · Category: misplaced responsibility
2 participants
What & why
Gemma 4 destabilizes when it shares a Mac with other models/work (memory contention, OOM/jetsam, ragged-batch NaN, Metal resource-count leaks). This restricts a dedicated model family (default
gemma-4) to providers whose entire advertised catalog is that family — i.e. machines dedicated to Gemma 4. Those dedicated boxes are the only routing candidates and the only providers counted when deciding whether to accept or shed a request to OpenRouter.Coordinator-only — no provider/Swift or protocol changes.
Routing behavior: before → after
Before — a Gemma 4 request could land on any box serving it (including mixed boxes where it contends), and "no provider" shed as
503:flowchart TD Req["Gemma 4 request"] --> Gate{"Provider serves gemma-4?"} Gate -->|"Dedicated box ✓<br/>(gemma-4 only)"| Cand["Candidate pool"] Gate -->|"Mixed box ✓<br/>(gemma-4 + qwen/gpt-oss)"| Cand Cand --> Cap{"Capacity?"} Cap -->|"Yes"| Route["Route to best score —<br/>⚠️ often a MIXED box →<br/>Gemma 4 contends → OOM / NaN / leak"] Cap -->|"All busy"| R429["429 rate_limit"] Cap -->|"None eligible"| R503["❌ 503 model_unavailable<br/>(OpenRouter may derank us)"]After — mixed boxes are filtered out at the single shared gate (routing and capacity preflight), so only dedicated boxes are eligible; "no dedicated box" sheds as
429:flowchart TD Req["Gemma 4 request"] --> Gate{"Provider serves gemma-4?"} Gate -->|"Dedicated box ✓"| Ded{"Dedicated to gemma-4?<br/>(advertised catalog == gemma-4 only)"} Gate -->|"Mixed box ✓"| Ded Ded -->|"No — mixed"| Drop["🚫 Excluded from routing<br/>AND capacity count<br/>(self-route to own box exempt)"] Ded -->|"Yes — dedicated"| Cand["Candidate pool =<br/>dedicated boxes only"] Cand --> Cap{"Capacity?"} Cap -->|"Yes"| Route["✅ Route to a DEDICATED box →<br/>Gemma 4 isolated, no contention"] Cap -->|"No dedicated box free"| R429["✅ 429 + Retry-After →<br/>clean OpenRouter failover"]How
One shared predicate,
registry.providerExcludedByDedicatedRuleLocked(p, model)(model is a dedicated family AND the provider isn't dedicated to it), applied at every public provider-selection site so routing, shedding, and pre-warming can never drift:providerPassesRoutingGatesLockedEx— the documented single gate shared by the dispatch hot path (snapshotProviderLockedEx) and the OpenRouter capacity preflight (quickCapacityCheck).coldSpillProviderEligibleLocked,warmPoolCandidateLocked,modelLoadCandidatePendingLocked— so the autoscaler/swap planner never pre-warms a dedicated model onto a mixed box.providerHasWarmModelLocked— so a warm mixed box isn't counted as warm capacity for a dedicated model (which would suppress warming a real dedicated box).A caller self-routing to its own machine is exempt (
!selfRouteOwner). Eligibility is catalog-based, not load-based, so an idle dedicated box stays eligible (cold-loads on demand).429, not 503
When the fleet serves the model but no dedicated box is available (none exist, or all busy), the preflight
no_eligible_providershed now returns429 rate_limit_exceeded+Retry-Afterinstead of503— for OpenRouter this is a clean transient failover rather than an "unhealthy endpoint" signal. Applied in bothhandleChatCompletionsand the generic/v1/completions+/v1/messagespath. A genuinely-absent model still503s (guarded byIsDedicatedModel(model) && HasProviderForModel(model, serials...)).Config & rollout
EIGENINFERENCE_DEDICATED_MODELS— comma-separated, case-insensitive substring patterns. Unset → defaultgemma-4(ON); set tonone/empty → disabled.registry.New) are unaffected — the rule only activates viacmd/coordinator/main.goat startup.Important
Default is ON. Once deployed, if the fleet has no Gemma-4-only boxes, all Gemma 4 traffic sheds to OpenRouter (429). Ensure dedicated machines exist (operationally: only Gemma 4 downloaded on them) or set
EIGENINFERENCE_DEDICATED_MODELS=none. Coordinator deploy is human-only.Testing
/v1/chat/completionsand/v1/completions, with a 503 control for a truly-absent model. Tests fail without the change; baselines confirm the dedicated rule is the differentiator.go build ./..., linux cross-compile, and the full coordinator suite pass;gofmtclean.Known follow-ups (non-blocking, cosmetic)
ModelCapacitySnapshot//v1/models/ the OpenRouter feed are display-only and left ungated, so they over-report Gemma 4 capacity by counting mixed boxes. No routing/admission impact.🤖 Generated with Claude Code