Repository navigation
fix(tau3-bench-banking): end episodes whose conversation outgrows any context window - #151
abhinav-pola wants to merge 1 commit into
Conversation
|
I'll fix CI failures and address comments from users with write access that start with 'Devin'.
Original prompt from Abhinav
|
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
There was a problem hiding this comment.
Devin Review found 2 potential issues.
1 flag not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)
| return ( | ||
| message.content.length + (message.reasoning?.length ?? 0) + toolCallChars | ||
| ); |
There was a problem hiding this comment.
🔴 Encrypted reasoning bypasses the conversation cap
When the model returns large encrypted reasoning, messageChars counts none of that payload. messagesToResponses replays responseItems verbatim, so oversized requests can still exhaust the trial's heap.
Learn more
The assistant output from responsesTurnToModelOutput includes the provider's raw responseItems. The provider requests encrypted reasoning and extractReasoning does not put encrypted content in message.reasoning. The next request uses messagesToResponses, which forwards the entire responseItems array instead of reconstructing it from counted fields. The counter therefore stays low even when those raw items make the next request large.
Example: An assistant turn contains responseItems: [{type: "reasoning", encrypted_content: "x".repeat(17_000_000)}] with empty content and no tool calls. The counter adds zero, then the solver sends the large raw item again instead of ending the episode.
Recommended fix: Count the serialized payload actually replayed from responseItems when present, including encrypted reasoning, while avoiding double-counting the synthesized fields for that same request. Add a solver-level regression that returns an oversized raw reasoning item and verifies termination before another model.generate.
Was this helpful? React with 👍 or 👎 to provide feedback.
| if (conversationChars > MAX_CONVERSATION_CHARS) { | ||
| terminationReason = TerminationReason.ConversationTooLarge; | ||
| break; | ||
| } |
There was a problem hiding this comment.
🟡 Oversized final turns evade cap termination
When an oversized user turn includes a stop token, pushMessage leaves UserStop in place, allowing a passing score. At the step or tool-error limit, earlier guards likewise replace ConversationTooLarge with another reason.
Learn more
The solver appends each message through pushMessage, but this check only runs at the start of a subsequent iteration. A user response containing a stop token sets done and never reaches that iteration; a turn reaching the step or error limit reaches the preceding guards first. The evaluator accepts UserStop but fails an oversized-conversation reason.
Example: A user response longer than 16,000,000 characters ends with ###STOP###. The solver records USER_STOP and can score it normally, while an equally oversized non-stop response records CONVERSATION_TOO_LARGE unless it hits another earlier limit.
Recommended fix: Resolve the cap immediately after each append and before accepting a stop or other termination, or prioritize cap termination above other guards and explicitly check the final user turn. Cover both stop-token and limit-boundary cases in a solver-level test.
Was this helpful? React with 👍 or 👎 to provide feedback.
1 participant
TL;DR
tau3 banking now ends an episode with
CONVERSATION_TOO_LARGE(scored as a failure) once the conversation passes 16M characters, instead of the trial pod running out of JavaScript heap.What changed?
solver.ts: tracks the conversation's total characters (content, reasoning, tool call names and arguments) as messages are appended. AboveMAX_CONVERSATION_CHARS(16,000,000) it stops running the remaining tool calls in the turn and ends the episode with the newTerminationReason.ConversationTooLarge.solver.test.ts: coversmessageCharsand checks that a runaway parallel-tool-call turn goes over the cap.Why?
In the mimo-v2.6-flash price-order run, 19 trials crashed with
JavaScript heap out of memory(exit 134, about 507MB heap in a 1Gi pod). In each crashed task, ClickHouse shows a Makora generation with 500–7,221 parallel tool calls in one turn, ending intool_calls. The solver runs every call, and eachKB_searchreturns its top 10 full documents (about 1.5KB each, so roughly 15KB per call). A 4,000-call turn adds about 60MB of tool-result text, and building the next request copies it several more times.16M characters is about 4M tokens, well above any context window in use. The largest prompt that succeeded in these runs was about 909k tokens. A request this large would be rejected upstream, so the cap only replaces a pod crash with a failed episode.
How to test
bun test src/benchmarks/tau3-bench-bankingEnd to end: after the subtree sync, rerun the crashed mimo-v2.6-flash price-order tasks. Any that hit the runaway turn should finish as
CONVERSATION_TOO_LARGEinstead of exiting with code 134.Benchmark impact
Scores are unchanged for every episode under 16M characters. Episodes above the cap used to crash the trial and now score 0, since the evaluator fails any termination other than user or agent stop.
Reviewer focus
Checklist
Link to Devin session: https://openrouter.devinenterprise.com/sessions/72a3f13c847645c5b06306d88cec089d
Open in Devin Desktop: https://openrouter.devinenterprise.com/desktop/session/72a3f13c847645c5b06306d88cec089d?variant=devin
Requested by: @abhinav-pola