Skip to content

Keep sonnet-5 on introductory pricing, and stop a cost cap discarding the run's output - #19

Open
lujstn wants to merge 2 commits into
mainfrom
worktree-budget-cap-output
Open

Keep sonnet-5 on introductory pricing, and stop a cost cap discarding the run's output#19
lujstn wants to merge 2 commits into
mainfrom
worktree-budget-cap-output

Conversation

@lujstn

@lujstn lujstn commented Aug 18, 2026

Copy link
Copy Markdown
Owner

Two independent cost fixes that came out of benchmarking the retry ladder.

Sonnet-5 keeps its introductory pricing

app/agent/cost.py carried a dated ramp repricing claude-sonnet-5 from $2/$10 to $3/$15 per Mtok on 1 September. That increase is not happening. Left in, it would have overstated every recorded cost by 50% from that date, and made max_cost_usd enforcement fire a third sooner on identical work, so sonnet runs would have started dying against their caps for no reason.

It was the only date-dependent price in the engine, so the now argument that existed to serve it goes too, from usage_cost, history_cost, _lookup and both runner.py call sites. test_sonnet_5_introductory_pricing pins the intro rate so the ramp cannot creep back.

Every cost already recorded was computed at the intro rate, so nothing needs restating.

A run stopped by its cost cap keeps its output

BudgetExceededError was added in April; the store-complete rescue that can judge a finished answer without a done call arrived in August. The handler never learned about it, so a cap fired, set a status, and threw away whatever the run had produced. The caller then paid full price for nothing and escalated to a dearer rung, which is the opposite of what a cost cap is for.

The handler now calls a new _budget_salvage(agent, store, clipboard, output_model): the store's output when the store holds anything, result.json otherwise, and success only when that output validates against the model and the store passes _gate_empty_fields, which is the same arbiter the normal finish already trusts. output and is_task_successful are written either way, so a partial becomes visible instead of vanishing.

Two deliberate behaviours, both intended:

  • Salvage never calls the LLM. The success path may call _coerce_to_schema to repair output, which costs tokens. Doing that after a budget kill would spend the budget that just ran out, so output that does not already validate is kept as it stands and recorded as a failure. It is commented as an invariant.
  • Terminal status now matches the success path, so a keep-alive session goes to idle rather than stopped. Before this, one over-budget turn bricked such a session permanently, because POST /v3/sessions refuses to reuse anything that is not idle or created. It is also the choice the budget contract requires: maxCostUsd is a ceiling on the session's whole spend, and fix: keep-alive follow-ups continue the session instead of restarting it #15 tops that pot back up on each dispatch, so a session that hit its cap has to stay reusable for the top-up to reach it. A session that went to stopped could never be dispatched to again, and the refill would be unreachable.

One consequence worth knowing: when the store is empty but result.json holds a valid, complete answer, the run is recorded as a failure even though that output is kept. The completeness gate is the only thing that can vouch for an answer on a run that never reached done, and it has nothing to judge against an empty store. This is the same rule the normal path already applies — its store-complete rescue also requires from_store — so a budget kill is not treated more harshly than any other run that ended early. The output is still written and returned; only the success flag is false, which is exactly the "best partial" shape a caller's ladder already understands.

The completion message now names the outcome, with the item count when there is a partial: Stopped: Cost $1.5012 exceeded budget $1.50. A partial output of 9 items is kept.

Tests

512 pass, up from 507. Five new tests cover the salvage: a complete store blessed, an incomplete store kept but recorded as a failure, an empty store falling back to result.json, nothing to keep, and no schema meaning no success claim.

Not covered: a live run actually tripping a cap and confirming the partial lands in the database. That needs a run on real hardware and has not been done.

Merging alongside #15

These two touch the same nine lines and nothing else in common: #15 leaves the except BudgetExceededError block exactly as main has it, and this PR rewrites it. Take this PR's version, with two adjustments once #15 has landed:

  • use fix: keep-alive follow-ups continue the session instead of restarting it #15's local keep_alive, which it re-reads each turn, rather than session.get("keep_alive") from the row read at worker start;
  • keep the salvage out of _finalise_task. Routing the budget path through the general finaliser would look tidier, but that function is allowed to call _coerce_to_schema, and spending on schema repair is exactly what a blown budget must not do.

The cost columns need no attention: #15's step hook writes them immediately before raising.

lujstn added 2 commits August 18, 2026 22:49
The cost engine carried a dated ramp that would have repriced sonnet-5 from
$2/$10 to $3/$15 per Mtok on 1 September. That increase is not happening, so
the ramp would have overstated every recorded cost by 50% and, worse, made
max_cost_usd enforcement fire a third sooner on identical work.

Sonnet-5 now sits in the ordinary price table. It was the only date-dependent
price in the engine, so the `now` argument that existed to serve it goes too.
The budget handler predates the store-complete rescue by four months, so it
recorded a status and discarded whatever the run had produced. A cap firing
therefore turned a nearly-finished run into a total loss, and the caller paid
full price for nothing before escalating.

It now resolves the output the same way a normal finish does and records a
verdict: the store's answer when there is one, result.json otherwise, and
success only when that output validates and the store passes the completeness
gate. Salvage never calls the LLM, because the budget that ended the run is the
budget schema coercion would spend, so output that does not already validate is
kept as it stands and recorded as a failure.

Terminal status now matches the success path, so one over-budget turn no longer
bricks a keep-alive session for good.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant