Keep sonnet-5 on introductory pricing, and stop a cost cap discarding the run's output - #19
Open
lujstn wants to merge 2 commits into
Open
Keep sonnet-5 on introductory pricing, and stop a cost cap discarding the run's output#19lujstn wants to merge 2 commits into
lujstn wants to merge 2 commits into
Conversation
The cost engine carried a dated ramp that would have repriced sonnet-5 from $2/$10 to $3/$15 per Mtok on 1 September. That increase is not happening, so the ramp would have overstated every recorded cost by 50% and, worse, made max_cost_usd enforcement fire a third sooner on identical work. Sonnet-5 now sits in the ordinary price table. It was the only date-dependent price in the engine, so the `now` argument that existed to serve it goes too.
The budget handler predates the store-complete rescue by four months, so it recorded a status and discarded whatever the run had produced. A cap firing therefore turned a nearly-finished run into a total loss, and the caller paid full price for nothing before escalating. It now resolves the output the same way a normal finish does and records a verdict: the store's answer when there is one, result.json otherwise, and success only when that output validates and the store passes the completeness gate. Salvage never calls the LLM, because the budget that ended the run is the budget schema coercion would spend, so output that does not already validate is kept as it stands and recorded as a failure. Terminal status now matches the success path, so one over-budget turn no longer bricks a keep-alive session for good.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two independent cost fixes that came out of benchmarking the retry ladder.
Sonnet-5 keeps its introductory pricing
app/agent/cost.pycarried a dated ramp repricingclaude-sonnet-5from $2/$10 to $3/$15 per Mtok on 1 September. That increase is not happening. Left in, it would have overstated every recorded cost by 50% from that date, and mademax_cost_usdenforcement fire a third sooner on identical work, so sonnet runs would have started dying against their caps for no reason.It was the only date-dependent price in the engine, so the
nowargument that existed to serve it goes too, fromusage_cost,history_cost,_lookupand bothrunner.pycall sites.test_sonnet_5_introductory_pricingpins the intro rate so the ramp cannot creep back.Every cost already recorded was computed at the intro rate, so nothing needs restating.
A run stopped by its cost cap keeps its output
BudgetExceededErrorwas added in April; the store-complete rescue that can judge a finished answer without adonecall arrived in August. The handler never learned about it, so a cap fired, set a status, and threw away whatever the run had produced. The caller then paid full price for nothing and escalated to a dearer rung, which is the opposite of what a cost cap is for.The handler now calls a new
_budget_salvage(agent, store, clipboard, output_model): the store's output when the store holds anything,result.jsonotherwise, and success only when that output validates against the model and the store passes_gate_empty_fields, which is the same arbiter the normal finish already trusts.outputandis_task_successfulare written either way, so a partial becomes visible instead of vanishing.Two deliberate behaviours, both intended:
_coerce_to_schemato repair output, which costs tokens. Doing that after a budget kill would spend the budget that just ran out, so output that does not already validate is kept as it stands and recorded as a failure. It is commented as an invariant.idlerather thanstopped. Before this, one over-budget turn bricked such a session permanently, becausePOST /v3/sessionsrefuses to reuse anything that is notidleorcreated. It is also the choice the budget contract requires:maxCostUsdis a ceiling on the session's whole spend, and fix: keep-alive follow-ups continue the session instead of restarting it #15 tops that pot back up on each dispatch, so a session that hit its cap has to stay reusable for the top-up to reach it. A session that went tostoppedcould never be dispatched to again, and the refill would be unreachable.One consequence worth knowing: when the store is empty but
result.jsonholds a valid, complete answer, the run is recorded as a failure even though that output is kept. The completeness gate is the only thing that can vouch for an answer on a run that never reacheddone, and it has nothing to judge against an empty store. This is the same rule the normal path already applies — its store-complete rescue also requiresfrom_store— so a budget kill is not treated more harshly than any other run that ended early. The output is still written and returned; only the success flag is false, which is exactly the "best partial" shape a caller's ladder already understands.The completion message now names the outcome, with the item count when there is a partial:
Stopped: Cost $1.5012 exceeded budget $1.50. A partial output of 9 items is kept.Tests
512 pass, up from 507. Five new tests cover the salvage: a complete store blessed, an incomplete store kept but recorded as a failure, an empty store falling back to
result.json, nothing to keep, and no schema meaning no success claim.Not covered: a live run actually tripping a cap and confirming the partial lands in the database. That needs a run on real hardware and has not been done.
Merging alongside #15
These two touch the same nine lines and nothing else in common: #15 leaves the
except BudgetExceededErrorblock exactly asmainhas it, and this PR rewrites it. Take this PR's version, with two adjustments once #15 has landed:keep_alive, which it re-reads each turn, rather thansession.get("keep_alive")from the row read at worker start;_finalise_task. Routing the budget path through the general finaliser would look tidier, but that function is allowed to call_coerce_to_schema, and spending on schema repair is exactly what a blown budget must not do.The cost columns need no attention: #15's step hook writes them immediately before raising.