feat(proxy): proactively compact over-window sessions instead of failing - #644
Merged
Conversation
A long Claude Code session (~1M+ tokens) could dead-end in the scorer: the
context-window pre-filter excluded every model whose window couldn't fit the
request, and combined with an org's excluded-models list this emptied the
eligible pool, returning ErrNoEligibleProvider — surfaced to the client as a
misleading "register a BYOK key" 400. Root cause: the router only reacted at
overflow (>100% of every window), too late for any model to ingest the history
and summarize it.
Mirror Claude Code's compaction: trigger BELOW the window (default 85%, via
ROUTER_COMPACTION_PCT) so the pre-summary history still fits a summarizer, then
run a tiered cascade before routing:
1. Tier 1 — ClearOldToolResults: clear all but the most recent tool results
(local, no model call); tool output dominates agentic-session tokens.
2. Tier 3 — structured 9-section summary via a window-aware Anthropic-family
summarizer (haiku when the history fits, claude-fable-5 for larger),
rewriting history to [summary + recent 12 turns] (tool pairing preserved).
3. Rescue — progressive TrimLastNMessages until it fits.
If even the trimmed floor overflows the largest eligible window, return a new
ErrContextWindowExceeded → HTTP 413 with an actionable message, replacing the
misleading ErrNoEligibleProvider mapping for the overflow case. The summary
call is billed as a `_precompaction_summary` ledger row. Wired into both
ProxyMessages and ProxyOpenAIChatCompletion.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Claude finished @steventohme's task —— View job Comment-length nits (advisory, won't block)The inline comment tool is unavailable in this environment, so flagging here instead.
Current (6 lines): // ErrContextWindowExceeded is returned when a request's estimated context —
// even after the full compaction cascade (tool-result cleanup, structured
// summarization, and recent-turn trimming) — still exceeds the largest context
// window any eligible model offers. It maps to HTTP 413, distinguishing a
// genuinely-too-large request from the "no provider keys" flavor of
// ErrNoEligibleProvider.Suggestion: // ErrContextWindowExceeded is returned after the full compaction cascade
// (tool-result cleanup, summarization, trim) still can't fit any eligible
// model's window. Maps to HTTP 413, distinct from ErrNoEligibleProvider.Was 6 lines; the cascade enumeration and HTTP 413 distinction fit in three.
Current (9 lines): // ClearOldToolResults replaces the content of every tool result EXCEPT the most
// recent keepRecent with ClearedToolResultPlaceholder, leaving message and
// tool-call structure intact. This is the cheap, model-free first tier of
// compaction: tool results (file reads, command output) dominate an agentic
// session's tokens, so clearing stale ones alone often brings a request back
// under a model's window without summarizing. Returns the number of tool
// results cleared. No-ops (returns 0) when there are at most keepRecent.
//
// Pure: no I/O. keepRecent < 0 is treated as 0.Suggestion: // ClearOldToolResults replaces every tool result except the most recent
// keepRecent with ClearedToolResultPlaceholder, leaving structure intact.
// Tool results dominate agentic-session tokens; clearing stale ones is the
// cheap, model-free Tier-1 step that often avoids a full summarization.
// Pure: no I/O. keepRecent < 0 is treated as 0.Was 9 lines restating behavior visible from the signature; the key WHY (token dominance → cheap first step) fits in 5.
Current (11 lines): // RewriteForCompaction keeps system context + a synthesized summary + the most
// recent keepRecentTurns non-system messages, eliding everything older. Unlike
// RewriteForHandover (which keeps only the latest user message), this preserves
// a tail of recent turns so the model retains immediate working context, which
// is what Claude Code's own compaction does. Tool results orphaned by the trim
// (their tool_use fell in the elided region) are stripped so the request stays
// wire-valid. The recent window is aligned to begin on a user message so the
// [summary, ...recent] sequence alternates roles correctly. Returns the number
// of original messages elided.
//
// Pure: no I/O. keepRecentTurns <= 0 is treated as 1 (summary + latest user).Suggestion: // RewriteForCompaction rewrites history to [summary + recent keepRecentTurns
// non-system messages], aligned to begin on a user turn. Orphaned tool
// results (whose tool_use was elided) are stripped to keep the request
// wire-valid. Unlike RewriteForHandover, a tail is kept so the model
// retains immediate working context. Returns the number of messages elided.
// Pure: no I/O. keepRecentTurns <= 0 is treated as 1.Was 11 lines; the invariants (user-alignment, orphan strip, contrast with RewriteForHandover) fit in 6.
Current (8 lines): // maybeCompact runs the proactive compaction cascade when the request is at or
// over compactionTriggerPct of maxWindow (the largest eligible model's window).
// It mutates env in place — the caller MUST recompute feats/estimates when
// res.Applied is true. Mirrors Claude Code's tiered cascade: (1) clear old tool
// results, (2) structured summarization with a window-aware model, (3) trim
// recent turns. Returns ErrContextWindowExceeded when even a maximally-trimmed
// request cannot fit maxWindow. A nil/zero-threshold Service, or a request
// already comfortably under threshold, is a no-op returning res.Applied=false.Suggestion: // maybeCompact runs the compaction cascade when needed ≥ compactionTriggerPct
// of maxWindow: (1) clear old tool results, (2) summarize with a window-aware
// model, (3) progressive trim. Mutates env in place — caller MUST recompute
// estimates when res.Applied is true. Returns ErrContextWindowExceeded if the
// history overflows even after all tiers; no-ops when pct is zero or unset.Was 8 lines; the tiers and two key contracts fit in 5. |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
 T-Rex LogsWhat T-Rex did
Reviews (1): Last reviewed commit: "feat(proxy): proactively compact over-wi..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Cursor Bugbot (2 threads) + claude advisory: - High: maybeCompact ran before turn classification, so it could rewrite Claude Code's own compaction turns (and other hard-pinned types: probe/title-gen/classifier) — corrupting CC's dedicated compaction flow. Extract isHardPinnedTurn (shared with the turn-loop hard-pin branch) and skip the cascade for those turns; classify via DetectFromEnvelope at both call sites. - Medium: WithCompaction reset pct<=0 to the default, so ROUTER_COMPACTION_PCT=0 couldn't disable compaction (contradicting the documented behavior). Allow exactly 0 to pass through (disables); only negative/>1 falls back to default. - Advisory: tightened the ErrContextWindowExceeded / ClearOldToolResults / RewriteForCompaction / maybeCompact godocs per the concise-comments rule. Added tests: hard-pinned turn skip, and WithCompaction(0) disables. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Claude finished @steventohme's task —— View job Comment-length nits (advisory, won't block)The inline comment tool is unavailable in this environment, so flagging here instead.
Current (4 lines): // DefaultCompactionTriggerPct is the fraction of the largest eligible
// model's window at which the cascade engages. Mirrors Claude Code's own
// ~0.85 auto-compact trigger: compacting below the window (not at overflow)
// keeps the pre-summary history small enough for a summarizer to ingest.Suggestion: // DefaultCompactionTriggerPct is the fraction of the largest eligible
// model's window at which compaction engages — below overflow so the
// history still fits a summarizer when the cascade runs.Was 4 lines; the key invariant fits in 3.
Current (4 lines): // largeWindowSummarizerModel is the big-context Anthropic-family model used
// to summarize histories too large for the cheap default summarizer. It is
// Anthropic-family so the Anthropic-only ProviderSummarizer can target it
// with no cross-format translation.Suggestion: // largeWindowSummarizerModel summarizes histories too large for the cheap
// default; Anthropic-family so ProviderSummarizer can target it directly.Was 4 lines; the constraint fits in 2.
Current (3 lines): // compactionSummarizerModels are the Anthropic-family models the cascade may
// summarize with, ordered cheapest-first. The selector picks the first whose
// context window can ingest the (post-Tier-1) history plus summary headroom.Suggestion: // compactionSummarizerModels are tried cheapest-first; the selector picks the
// first whose window fits the (post-Tier-1) history plus summary headroom.Was 3 lines restating the var; fits in 2.
Current (4 lines): // CompactionSummarizer summarizes prior conversation with the structured
// compaction prompt against an explicit model. Implemented by
// *ProviderSummarizer; declared here so the Service depends on the behavior,
// not the concrete type.Suggestion: // CompactionSummarizer summarizes prior conversation with the structured
// compaction prompt. Declared here so the Service depends on the behavior,
// not the concrete type (*ProviderSummarizer).Was 4 lines; fits in 3.
Current (4 lines): // DefaultCompactionMaxTokens caps the structured compaction summary. Larger
// than the switch handover cap because the compaction summary is the ONLY
// record of the elided history the model keeps — it must carry task state, not
// just a gist.Suggestion: // DefaultCompactionMaxTokens caps the compaction summary. Larger than the
// switch-handover cap: this summary is the sole record of elided history.Was 4 lines; the WHY (sole record) fits in 2.
Current (5 lines): // compactionInstruction elicits Claude Code's 9-section structured summary used
// when a long session is compacted to fit a context window. Unlike the terse
// switch-handover instruction, this preserves enough task state (pending work,
// current file, next step) that the model can continue seamlessly, and quotes
// user-stated constraints verbatim so they keep applying after the elision.Suggestion: // compactionInstruction elicits the 9-section structured summary used when a
// long session is compacted. Unlike the terse switch-handover instruction, it
// preserves task state and quotes user constraints verbatim.Was 5 lines; fits in 3.
Current (6 lines): // WithCompaction installs the summarizer and trigger threshold for the
// proactive context-window compaction cascade (maybeCompact). pct == 0
// disables compaction (operators set ROUTER_COMPACTION_PCT=0 to turn the
// cascade off); an out-of-range pct (negative or > 1) falls back to
// DefaultCompactionTriggerPct. A nil summarizer leaves Tier-3 summarization off
// (Tier-1 cleanup + trim rescue still run).Suggestion: // WithCompaction installs the summarizer and trigger threshold for the
// compaction cascade. pct==0 disables; negative or >1 falls back to
// DefaultCompactionTriggerPct. nil summarizer disables Tier-3 only.Was 6 lines; the three key contracts fit in 3.
Current (4 lines): // Snapshot inbound (client-sent) state BEFORE any env rewrite. The
// compaction tracker, spiral scan, and tool-output telemetry must compare
// what the client actually sent, not a router-shortened body — either the
// proactive compaction just below or runTurnLoop's switch-handover rewrite.Suggestion: // Snapshot inbound state before any env rewrite (proactive compaction below
// or switch-handover in runTurnLoop) so telemetry reflects what the client sent.Was 4 lines; fits in 2.
Current (4 lines): // Proactive context-window compaction: shrink an over-long conversation to
// fit the largest eligible model BEFORE routing, so a genuinely huge
// session is compacted (à la Claude Code) instead of dead-ending in the
// scorer with no eligible provider. Mutates env; feats is recomputed after.Suggestion: // Proactive compaction: shrink an over-long session before routing so it
// doesn't dead-end the scorer. Mutates env; feats is recomputed after.Was 4 lines; fits in 2.
Current (5 lines): // isHardPinnedTurn reports whether a turn type bypasses pin lookup/write,
// planner, and scorer entirely via the boot-time hard pin. These turns are
// also skipped by proactive compaction: they are either tiny (probe/title-gen/
// classifier) or carry their own dedicated flow (Claude Code's compaction turn,
// whose request the router must not rewrite).Suggestion: // isHardPinnedTurn reports whether tt bypasses pin/planner/scorer via the
// boot-time hard pin — and is skipped by proactive compaction: these turns
// are tiny (probe/title-gen/classifier) or self-compact (Compaction turn).Was 5 lines; fits in 3.
Current (5 lines): // ClearOldToolResults replaces every tool result except the most recent
// keepRecent with ClearedToolResultPlaceholder, leaving structure intact and
// returning the number cleared. Tool results dominate agentic-session tokens;
// clearing stale ones is the cheap, model-free Tier-1 step that often avoids a
// full summarization. Pure: no I/O. keepRecent < 0 is treated as 0.Suggestion: // ClearOldToolResults replaces every tool result except the most recent
// keepRecent with ClearedToolResultPlaceholder. Tool results dominate
// agentic-session tokens; clearing stale ones is the cheap Tier-1 step.
// Pure: no I/O. keepRecent < 0 is treated as 0.Was 5 lines; fits in 4.
Current (6 lines): // RewriteForCompaction rewrites history to [summary + recent keepRecentTurns
// non-system messages], aligned to begin on a user turn so roles alternate.
// Orphaned tool results (whose tool_use was elided) are stripped to keep the
// request wire-valid. Unlike RewriteForHandover, a tail is kept so the model
// retains immediate working context. Returns the number of messages elided.
// Pure: no I/O. keepRecentTurns <= 0 is treated as 1.Suggestion: // RewriteForCompaction rewrites history to [summary + recent keepRecentTurns
// non-system messages], user-turn-aligned. Orphaned tool results (tool_use
// elided) are stripped. Unlike RewriteForHandover, a tail is kept.
// Returns elided count. Pure: no I/O. keepRecentTurns <= 0 is treated as 1.Was 6 lines; fits in 4.
Current (4 lines): // userAlignedStart returns the index at which to begin a recent-message window
// of size keepRecent drawn from msgs, advanced forward to the first user
// message so the window starts on a user turn. Falls back to the last user
// message's index when the window contains none.Suggestion: // userAlignedStart returns the start index for a keepRecent-message window,
// advanced to the first user turn. Falls back to the last user message index.Was 4 lines restating the code; fits in 2. |
Cursor Bugbot (Medium): maybeCompact's fit/floor check used `ContextOverflowTokenEstimate + outputReserve` (signature-keeping), but the context-overflow pre-filter subtracts SignatureTokenSavings for signature-stripping (non-Anthropic) targets. A signature-heavy Anthropic session could thus hit the 413 floor even though a deployed stripping model (e.g. gpt-5.5) would fit after the translator drops the base64 thought signatures. maxEligibleContextWindow now adds sigSavings to a stripping model's effective window (mirroring excludeContextOverflowModels), so the trigger and 413 floor match the pre-filter. Callers pass env.SignatureTokenSavings() (0 for non-Anthropic ingress, so OpenAI is unchanged). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Claude finished @steventohme's task —— View job Comment-length nits (advisory, won't block)The inline comment tool is unavailable in this environment, so flagging here instead.
Current (4 lines): // DefaultCompactionTriggerPct is the fraction of the largest eligible
// model's window at which the cascade engages. Mirrors Claude Code's own
// ~0.85 auto-compact trigger: compacting below the window (not at overflow)
// keeps the pre-summary history small enough for a summarizer to ingest.Suggestion: // DefaultCompactionTriggerPct is the fraction of the largest eligible
// model's window at which compaction engages — below overflow so the
// history still fits a summarizer when the cascade runs.Was 4 lines; the key invariant fits in 3.
Current (4 lines): // largeWindowSummarizerModel is the big-context Anthropic-family model used
// to summarize histories too large for the cheap default summarizer. It is
// Anthropic-family so the Anthropic-only ProviderSummarizer can target it
// with no cross-format translation.Suggestion: // largeWindowSummarizerModel summarizes histories too large for the cheap
// default; Anthropic-family so ProviderSummarizer can target it directly.Was 4 lines; fits in 2.
Current (3 lines): // compactionSummarizerModels are the Anthropic-family models the cascade may
// summarize with, ordered cheapest-first. The selector picks the first whose
// context window can ingest the (post-Tier-1) history plus summary headroom.Suggestion: // compactionSummarizerModels are tried cheapest-first; the selector picks the
// first whose window fits the (post-Tier-1) history plus summary headroom.Was 3 lines restating the var; fits in 2.
Current (4 lines): // CompactionSummarizer summarizes prior conversation with the structured
// compaction prompt against an explicit model. Implemented by
// *ProviderSummarizer; declared here so the Service depends on the behavior,
// not the concrete type.Suggestion: // CompactionSummarizer summarizes prior conversation with the structured
// compaction prompt. Declared here so the Service depends on the behavior,
// not the concrete type (*ProviderSummarizer).Was 4 lines; fits in 3.
Current (8 lines): // maxEligibleContextWindow returns the largest effective context window among
// available routing models that are not policy-excluded. A signature-stripping
// (non-Anthropic) target gets sigSavings added to its window: the translator
// drops base64 thought-signature blocks before dispatch, so that model can
// serve sigSavings more of this request's estimated tokens — mirroring the
// per-model discount in excludeContextOverflowModels, so a signature-heavy
// session isn't falsely 413'd when a stripping model would still fit. Zero when
// none are known (availableModels unset), which disables compaction.Suggestion: // maxEligibleContextWindow returns the largest effective context window among
// available routing models that are not policy-excluded. Signature-stripping
// (non-Anthropic) targets get sigSavings added to their window, mirroring the
// per-model discount in excludeContextOverflowModels so a signature-heavy
// session isn't falsely 413'd. Zero when availableModels is unset.Was 8 lines; the key invariant (signature discount mirrors pre-filter) fits in 5.
Current (7 lines): // maybeCompact runs the compaction cascade when needed ≥ compactionTriggerPct
// of maxWindow: (1) clear old tool results, (2) summarize with a window-aware
// model, (3) progressive trim. Mutates env in place — caller MUST recompute
// estimates when res.Applied is true. Returns ErrContextWindowExceeded if the
// history overflows even after all tiers; no-ops when pct is zero/unset, below
// threshold, or the turn is hard-pinned (Claude Code's own compaction turn must
// not be rewritten, and probe/title-gen/classifier turns bypass the scorer).Suggestion: // maybeCompact runs the compaction cascade when needed ≥ compactionTriggerPct
// of maxWindow: (1) clear old tool results, (2) summarize with a window-aware
// model, (3) progressive trim. Mutates env in place — caller MUST recompute
// estimates when res.Applied is true. Returns ErrContextWindowExceeded if the
// history overflows even after all tiers; no-ops when pct is zero or the turn
// is hard-pinned.Was 7 lines; fits in 6.
Current (4 lines): // DefaultCompactionMaxTokens caps the structured compaction summary. Larger
// than the switch handover cap because the compaction summary is the ONLY
// record of the elided history the model keeps — it must carry task state, not
// just a gist.Suggestion: // DefaultCompactionMaxTokens caps the compaction summary. Larger than the
// switch-handover cap: this summary is the sole record of elided history.Was 4 lines; the WHY fits in 2.
Current (5 lines): // compactionInstruction elicits Claude Code's 9-section structured summary used
// when a long session is compacted to fit a context window. Unlike the terse
// switch-handover instruction, this preserves enough task state (pending work,
// current file, next step) that the model can continue seamlessly, and quotes
// user-stated constraints verbatim so they keep applying after the elision.Suggestion: // compactionInstruction elicits the 9-section structured summary used when a
// long session is compacted. Unlike the terse switch-handover instruction, it
// preserves task state and quotes user constraints verbatim.Was 5 lines; fits in 3.
Current (5 lines): // SummarizeForCompaction summarizes env with the structured 9-section
// compaction prompt, targeting an explicit model (the window-aware selection
// happens in the caller) and a larger output cap. Used by the context-window
// compaction cascade, not the switch-handover path. Same failure contract as
// Summarize.Suggestion: // SummarizeForCompaction summarizes env with the structured compaction prompt,
// targeting an explicit model and a larger output cap. Same failure contract
// as Summarize.Was 5 lines; fits in 3.
Current (6 lines): // WithCompaction installs the summarizer and trigger threshold for the
// proactive context-window compaction cascade (maybeCompact). pct == 0
// disables compaction (operators set ROUTER_COMPACTION_PCT=0 to turn the
// cascade off); an out-of-range pct (negative or > 1) falls back to
// DefaultCompactionTriggerPct. A nil summarizer leaves Tier-3 summarization off
// (Tier-1 cleanup + trim rescue still run).Suggestion: // WithCompaction installs the summarizer and trigger threshold for the
// compaction cascade. pct==0 disables; negative or >1 falls back to
// DefaultCompactionTriggerPct. nil summarizer disables Tier-3 only.Was 6 lines; the three key contracts fit in 3.
Current (4 lines): // Snapshot inbound (client-sent) state BEFORE any env rewrite. The
// compaction tracker, spiral scan, and tool-output telemetry must compare
// what the client actually sent, not a router-shortened body — either the
// proactive compaction just below or runTurnLoop's switch-handover rewrite.Suggestion: // Snapshot inbound state before any env rewrite (proactive compaction below
// or switch-handover in runTurnLoop) so telemetry reflects what the client sent.Was 4 lines; fits in 2.
Current (4 lines): // Proactive context-window compaction: shrink an over-long conversation to
// fit the largest eligible model BEFORE routing, so a genuinely huge
// session is compacted (à la Claude Code) instead of dead-ending in the
// scorer with no eligible provider. Mutates env; feats is recomputed after.Suggestion: // Proactive compaction: shrink an over-long session before routing so it
// doesn't dead-end the scorer. Mutates env; feats is recomputed after.Was 4 lines; fits in 2.
Current (5 lines): // isHardPinnedTurn reports whether a turn type bypasses pin lookup/write,
// planner, and scorer entirely via the boot-time hard pin. These turns are
// also skipped by proactive compaction: they are either tiny (probe/title-gen/
// classifier) or carry their own dedicated flow (Claude Code's compaction turn,
// whose request the router must not rewrite).Suggestion: // isHardPinnedTurn reports whether tt bypasses pin/planner/scorer via the
// boot-time hard pin — and is skipped by proactive compaction: these turns
// are tiny (probe/title-gen/classifier) or self-compact (Compaction turn).Was 5 lines; fits in 3.
Current (5 lines): // ClearOldToolResults replaces every tool result except the most recent
// keepRecent with ClearedToolResultPlaceholder, leaving structure intact and
// returning the number cleared. Tool results dominate agentic-session tokens;
// clearing stale ones is the cheap, model-free Tier-1 step that often avoids a
// full summarization. Pure: no I/O. keepRecent < 0 is treated as 0.Suggestion: // ClearOldToolResults replaces every tool result except the most recent
// keepRecent with ClearedToolResultPlaceholder. Tool results dominate
// agentic-session tokens; clearing stale ones is the cheap Tier-1 step.
// Pure: no I/O. keepRecent < 0 is treated as 0.Was 5 lines; fits in 4.
Current (6 lines): // RewriteForCompaction rewrites history to [summary + recent keepRecentTurns
// non-system messages], aligned to begin on a user turn so roles alternate.
// Orphaned tool results (whose tool_use was elided) are stripped to keep the
// request wire-valid. Unlike RewriteForHandover, a tail is kept so the model
// retains immediate working context. Returns the number of messages elided.
// Pure: no I/O. keepRecentTurns <= 0 is treated as 1.Suggestion: // RewriteForCompaction rewrites history to [summary + recent keepRecentTurns
// non-system messages], user-turn-aligned. Orphaned tool results (tool_use
// elided) are stripped. Unlike RewriteForHandover, a tail is kept.
// Returns elided count. Pure: no I/O. keepRecentTurns <= 0 is treated as 1.Was 6 lines; fits in 4.
Current (4 lines): // userAlignedStart returns the index at which to begin a recent-message window
// of size keepRecent drawn from msgs, advanced forward to the first user
// message so the window starts on a user turn. Falls back to the last user
// message's index when the window contains none.Suggestion: // userAlignedStart returns the start index for a keepRecent-message window,
// advanced to the first user turn. Falls back to the last user message index.Was 4 lines restating the code; fits in 2. |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit f563a3e. Configure here.
…tion Cursor Bugbot (Medium): proactive maybeCompact shrinks env before runTurnLoop, so the client-trim detector (checkAndRecord) compares post-compaction counts to the prior turn and sets PrefixTrimmed as a false positive. On a non-Anthropic route that triggered a second, redundant runCompactionHandover — an extra billed summarizer call that also replaced the recent-turn tail maybeCompact deliberately kept with [summary, latestUser]. Gate the compaction handover on !compRes.Applied: when the router already compacted this turn, its env is the intended shape, so re-summarizing is both wasteful and destructive. (The prefixBroken free-switch signal inside runTurnLoop is left as-is — the cache is genuinely dead after the rewrite, so skipping the switch handover there is correct.) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Claude finished @steventohme's task —— View job Comment-length nits (advisory, won't block)The inline comment tool is unavailable in this environment, so flagging here instead.
Current (4 lines): // DefaultCompactionTriggerPct is the fraction of the largest eligible
// model's window at which the cascade engages. Mirrors Claude Code's own
// ~0.85 auto-compact trigger: compacting below the window (not at overflow)
// keeps the pre-summary history small enough for a summarizer to ingest.Suggestion: // DefaultCompactionTriggerPct is the fraction of the largest eligible
// model's window at which compaction engages — below overflow so the
// history still fits a summarizer when the cascade runs.Was 4 lines; the key invariant fits in 3.
Current (4 lines): // largeWindowSummarizerModel is the big-context Anthropic-family model used
// to summarize histories too large for the cheap default summarizer. It is
// Anthropic-family so the Anthropic-only ProviderSummarizer can target it
// with no cross-format translation.Suggestion: // largeWindowSummarizerModel summarizes histories too large for the cheap
// default; Anthropic-family so ProviderSummarizer can target it directly.Was 4 lines; fits in 2.
Current (3 lines): // compactionSummarizerModels are the Anthropic-family models the cascade may
// summarize with, ordered cheapest-first. The selector picks the first whose
// context window can ingest the (post-Tier-1) history plus summary headroom.Suggestion: // compactionSummarizerModels are tried cheapest-first; the selector picks the
// first whose window fits the (post-Tier-1) history plus summary headroom.Was 3 lines restating the var; fits in 2.
Current (4 lines): // CompactionSummarizer summarizes prior conversation with the structured
// compaction prompt against an explicit model. Implemented by
// *ProviderSummarizer; declared here so the Service depends on the behavior,
// not the concrete type.Suggestion: // CompactionSummarizer summarizes prior conversation with the structured
// compaction prompt. Declared here so the Service depends on the behavior,
// not the concrete type (*ProviderSummarizer).Was 4 lines; fits in 3.
Current (8 lines): // maxEligibleContextWindow returns the largest effective context window among
// available routing models that are not policy-excluded. A signature-stripping
// (non-Anthropic) target gets sigSavings added to its window: the translator
// drops base64 thought-signature blocks before dispatch, so that model can
// serve sigSavings more of this request's estimated tokens — mirroring the
// per-model discount in excludeContextOverflowModels, so a signature-heavy
// session isn't falsely 413'd when a stripping model would still fit. Zero when
// none are known (availableModels unset), which disables compaction.Suggestion: // maxEligibleContextWindow returns the largest effective context window among
// available routing models that are not policy-excluded. Signature-stripping
// (non-Anthropic) targets get sigSavings added to their window, mirroring the
// per-model discount in excludeContextOverflowModels so a signature-heavy
// session isn't falsely 413'd. Zero when availableModels is unset.Was 8 lines; the key invariant (signature discount mirrors pre-filter) fits in 5.
Current (7 lines): // maybeCompact runs the compaction cascade when needed ≥ compactionTriggerPct
// of maxWindow: (1) clear old tool results, (2) summarize with a window-aware
// model, (3) progressive trim. Mutates env in place — caller MUST recompute
// estimates when res.Applied is true. Returns ErrContextWindowExceeded if the
// history overflows even after all tiers; no-ops when pct is zero/unset, below
// threshold, or the turn is hard-pinned (Claude Code's own compaction turn must
// not be rewritten, and probe/title-gen/classifier turns bypass the scorer).Suggestion: // maybeCompact runs the compaction cascade when needed ≥ compactionTriggerPct
// of maxWindow: (1) clear old tool results, (2) summarize with a window-aware
// model, (3) progressive trim. Mutates env in place — caller MUST recompute
// estimates when res.Applied is true. Returns ErrContextWindowExceeded if the
// history overflows even after all tiers; no-ops when pct is zero or the turn
// is hard-pinned.Was 7 lines; fits in 6.
Current (5 lines): // runCompactionSummary picks a window-aware summarizer model and dispatches the
// structured summary call, honoring the tenant-boundary credential rules used
// by the switch-handover path. Returns ok=false (and logs) when no summarizer
// fits the history, the tenant boundary forbids the call, or the call fails —
// in every such case the caller falls through to trimming.Suggestion: // runCompactionSummary picks a window-aware summarizer model and dispatches the
// structured summary call, honoring tenant-boundary credential rules. Returns
// ok=false (and logs) when no summarizer fits, the boundary forbids the call,
// or the call fails — caller falls through to trimming.Was 5 lines; fits in 4.
Current (4 lines): // DefaultCompactionMaxTokens caps the structured compaction summary. Larger
// than the switch handover cap because the compaction summary is the ONLY
// record of the elided history the model keeps — it must carry task state, not
// just a gist.Suggestion: // DefaultCompactionMaxTokens caps the compaction summary. Larger than the
// switch-handover cap: this summary is the sole record of elided history.Was 4 lines; the WHY fits in 2.
Current (5 lines): // compactionInstruction elicits Claude Code's 9-section structured summary used
// when a long session is compacted to fit a context window. Unlike the terse
// switch-handover instruction, this preserves enough task state (pending work,
// current file, next step) that the model can continue seamlessly, and quotes
// user-stated constraints verbatim so they keep applying after the elision.Suggestion: // compactionInstruction elicits the 9-section structured summary used when a
// long session is compacted. Unlike the terse switch-handover instruction, it
// preserves task state and quotes user constraints verbatim.Was 5 lines; fits in 3.
Current (6 lines): // WithCompaction installs the summarizer and trigger threshold for the
// proactive context-window compaction cascade (maybeCompact). pct == 0
// disables compaction (operators set ROUTER_COMPACTION_PCT=0 to turn the
// cascade off); an out-of-range pct (negative or > 1) falls back to
// DefaultCompactionTriggerPct. A nil summarizer leaves Tier-3 summarization off
// (Tier-1 cleanup + trim rescue still run).Suggestion: // WithCompaction installs the summarizer and trigger threshold for the
// compaction cascade. pct==0 disables; negative or >1 falls back to
// DefaultCompactionTriggerPct. nil summarizer disables Tier-3 only.Was 6 lines; the three key contracts fit in 3.
Current (4 lines): // Snapshot inbound (client-sent) state BEFORE any env rewrite. The
// compaction tracker, spiral scan, and tool-output telemetry must compare
// what the client actually sent, not a router-shortened body — either the
// proactive compaction just below or runTurnLoop's switch-handover rewrite.Suggestion: // Snapshot inbound state before any env rewrite (proactive compaction below
// or switch-handover in runTurnLoop) so telemetry reflects what the client sent.Was 4 lines; fits in 2.
Current (4 lines): // Proactive context-window compaction: shrink an over-long conversation to
// fit the largest eligible model BEFORE routing, so a genuinely huge
// session is compacted (à la Claude Code) instead of dead-ending in the
// scorer with no eligible provider. Mutates env; feats is recomputed after.Suggestion: // Proactive compaction: shrink an over-long session before routing so it
// doesn't dead-end the scorer. Mutates env; feats is recomputed after.Was 4 lines; fits in 2.
Current (5 lines): // isHardPinnedTurn reports whether a turn type bypasses pin lookup/write,
// planner, and scorer entirely via the boot-time hard pin. These turns are
// also skipped by proactive compaction: they are either tiny (probe/title-gen/
// classifier) or carry their own dedicated flow (Claude Code's compaction turn,
// whose request the router must not rewrite).Suggestion: // isHardPinnedTurn reports whether tt bypasses pin/planner/scorer via the
// boot-time hard pin — and is skipped by proactive compaction: these turns
// are tiny (probe/title-gen/classifier) or self-compact (Compaction turn).Was 5 lines; fits in 3.
Current (5 lines): // ClearOldToolResults replaces every tool result except the most recent
// keepRecent with ClearedToolResultPlaceholder, leaving structure intact and
// returning the number cleared. Tool results dominate agentic-session tokens;
// clearing stale ones is the cheap, model-free Tier-1 step that often avoids a
// full summarization. Pure: no I/O. keepRecent < 0 is treated as 0.Suggestion: // ClearOldToolResults replaces every tool result except the most recent
// keepRecent with ClearedToolResultPlaceholder. Tool results dominate
// agentic-session tokens; clearing stale ones is the cheap Tier-1 step.
// Pure: no I/O. keepRecent < 0 is treated as 0.Was 5 lines; fits in 4.
Current (6 lines): // RewriteForCompaction rewrites history to [summary + recent keepRecentTurns
// non-system messages], aligned to begin on a user turn so roles alternate.
// Orphaned tool results (whose tool_use was elided) are stripped to keep the
// request wire-valid. Unlike RewriteForHandover, a tail is kept so the model
// retains immediate working context. Returns the number of messages elided.
// Pure: no I/O. keepRecentTurns <= 0 is treated as 1.Suggestion: // RewriteForCompaction rewrites history to [summary + recent keepRecentTurns
// non-system messages], user-turn-aligned. Orphaned tool results (tool_use
// elided) are stripped. Unlike RewriteForHandover, a tail is kept.
// Returns elided count. Pure: no I/O. keepRecentTurns <= 0 is treated as 1.Was 6 lines; fits in 4.
Current (4 lines): // userAlignedStart returns the index at which to begin a recent-message window
// of size keepRecent drawn from msgs, advanced forward to the first user
// message so the window starts on a user turn. Falls back to the last user
// message's index when the window contains none.Suggestion: // userAlignedStart returns the start index for a keepRecent-message window,
// advanced to the first user turn. Falls back to the last user message index.Was 4 lines restating the code; fits in 2. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Router session
e0f9de9f-eb02-4f3f-86c8-3a88cbd99a53(prod) failed every request withcluster: no eligible provider for request. It's a Claude Code CLI session that grew to a ~1M-token context. Chain:needed = est(~988k) + output_reserve(64k) ≈ 1.05M, exceeding evengpt-5.5's 1.05M window → it excluded all large-window models (including the pinnedgpt-5.5).ErrNoEligibleProvider.Root cause: the router only reacted at overflow (>100% of every window) — too late for any model to ingest the history and summarize it. Claude Code avoids this by auto-compacting at ~85% of the window, below the limit, while the history still fits a model.
Fix
Mirror Claude Code's compaction.
maybeCompactruns before routing (inProxyMessages+ProxyOpenAIChatCompletion), engaging when the estimate reachesROUTER_COMPACTION_PCT(default 0.85) of the largest eligible model's window:ClearOldToolResults: local, clears all but the most recent tool results (tool output dominates agentic-session tokens). Often enough alone.claude-haiku-4-5when the history fits its window,claude-fable-5(1M) for larger), rewriting history to[summary + recent 12 turns]with tool_use/tool_result pairing preserved.If even the trimmed floor overflows the largest eligible window, return a new
ErrContextWindowExceeded→ HTTP 413 with an actionable message, replacing the misleadingErrNoEligibleProvidermapping for the overflow case. The summary call is billed as a_precompaction_summaryledger row.Trigger below the window is load-bearing: a summarizer can only ingest a history that still fits some model.
Config
ROUTER_COMPACTION_PCT(default0.85, range(0,1];0disables → over-window requests 413).Changes
internal/translate/compaction.go— pureClearOldToolResults+RewriteForCompaction(Anthropic/OpenAI/Gemini), reusing existing orphaned-tool-result stripping.internal/proxy/compaction.go— cascade orchestration, window-aware summarizer selection,ErrContextWindowExceeded, summary billing.internal/proxy/handover.go— parametric summary builder +SummarizeForCompaction(9-section prompt, larger cap).internal/proxy/dispatch_error.go— classifyErrContextWindowExceeded→ 413.internal/proxy/service.go— wire cascade into both entry points; snapshots taken pre-rewrite.cmd/router/main.go—WithCompaction,ROUTER_COMPACTION_PCT.Testing
go build,go vet, and fullgo test(-tags no_onnx) pass. New unit tests cover the cascade tiers, window-aware selection, the 413 floor, and the error classification. Tool_use/tool_result pairing is asserted preserved across rewrites.🤖 Generated with Claude Code