feat(cost): cache-aware token and cost accounting; meter DeepSeek and OpenRouter

[Ruqii]

Summary

For a model × harness experiment, report.json now gives, per case and per model: uncached input, cache-read, cache-write and output tokens, total cost_usd, and cache_cost_usd (the part of the total spent on cache). This works across vendors: Anthropic, OpenAI (Chat Completions and Responses), DeepSeek, OpenRouter, Moonshot and Mistral.

• Two new providers.
  • deepseek (DEEPSEEK_BASE_URL). Its deepseek-search entry also meters the DeepSeek harness's web-search side call.
  • openrouter, which redirects both OPENROUTER_BASE_URL and litellm's OPENROUTER_API_BASE.
• Usage parsing. Cache fields are parsed for every vendor, streaming and non-streaming, and normalised to four disjoint counts.
• Cache pricing. Cache rates come from the site's price table, like every other price: three optional columns. Until the site serves them, a clearly marked interim module fills a missing rate from the vendor's documented multiple. The CLI itself only parses and computes. An unknown cost stays null.
• Where the counts show up. ModelCost and CaseCost in report.json, the live run-context patch, and the terminal table.

Everything below was checked against the vendors' current docs or source on 2026-09-11. The URLs are inline in pricing.py and tests/test_usage_shapes.py.

What report.json carries now

Each field is on every cost.by_model[] entry, and CaseCost sums each one the same way it sums prompt_tokens:

Field	Meaning
prompt_tokens	uncached input
cache_read_tokens	input read from the prompt cache
cache_write_tokens	input written to the cache (every TTL)
completion_tokens	output (reasoning included)
cost_usd	total, cache included
cache_cost_usd	the part of cost_usd spent on cache reads + writes (inside the total, not on top)

Decision: prompt_tokens stays "uncached input"

The four counts are disjoint, so the input a model was sent is prompt_tokens + cache_read_tokens + cache_write_tokens. I chose this over "all input" for three reasons:

1. It's Anthropic's native split. Anthropic's input_tokens already excludes both cache counts.
2. It's the site's split. The site's run context (lib/runs/context.ts usageEntry) is input / cache_read / cache_creation, with input excluding cache. The hook-fed v2/runs/{id}/usage route normalises to the same thing.
3. It fails safe on the site. Server-side repricing multiplies the input count by the full input rate. That covers priceTokens/priceTokensDetailed in lib/model-pricing.ts (and scripts/recalc-null-costs.ts on main; it is gone on uat). With "all input", those would charge cache reads at 10× their price. With "uncached input", they only leave out a component the site already declares as "cache tokens recorded, not priced".

tp submit is unaffected. /api/submit stores by_model verbatim into cases.usage (jsonb) and only sums cost_usd. The new fields ride along, and older CLIs ignore them.

⚠️ Behaviour changes (numbers move versus the current CLI)

• prompt_tokens drops by the cached count for OpenAI, DeepSeek, OpenRouter, Moonshot and Mistral. Those vendors report cached tokens as part of the prompt; they now move to cache_read_tokens / cache_write_tokens. Anthropic is unchanged.
• cost_usd now prices the cache. Before, Anthropic's cache tokens weren't priced at all, so the cost was understated. OpenAI-style cached tokens were priced at the full input rate, so the cost was overstated.
• OpenAI Responses-API streams used to meter as zero; now they meter. Their usage is nested under response.completed, which the old parser never read. This is probably the highest-value fix in the branch.
• Anthropic streams with server tools now take the cumulative input from message_delta. Before, only message_start's smaller count was used (e.g. 2679 → 10682 in Anthropic's own web-search sample).
• cost_usd can now be null where it used to show a number. This happens when a call used the cache on a model with no cache rate anywhere: not on its price row, and no documented multiplier. That includes every routed OpenRouter id (anthropic/…, openai/…, deepseek/… …) until the site serves cache rates for it, since routed ids get no interim fallback.
  • This is deliberate: pricing those tokens at 0× or 1× would be a wrong number, which is worse than a missing one.
  • It goes away once the site serves cache columns (see below).

Providers

Registry entry	Key	Base URL redirected → upstream	Why
deepseek	DEEPSEEK_API_KEY	DEEPSEEK_BASE_URL → https://api.deepseek.com	The DeepSeek harness (packages/llm/llm-deepseek) reads it and POSTs ${base}/chat/completions with no /v1, always streaming with include_usage.
deepseek-search (filed as deepseek)	DEEPSEEK_API_KEY	DEEPSEEK_SEARCH_BASE_URL → https://api.deepseek.com/anthropic/v1	The harness's web_search is a separate billed call: non-streaming, Anthropic format, to ${base}/messages. DEEPSEEK_BASE_URL doesn't reach it.
openrouter	OPENROUTER_API_KEY	OPENROUTER_BASE_URL and OPENROUTER_API_BASE → https://openrouter.ai/api/v1	See below.

Why openrouter redirects both variables: litellm (and so Aider) reads only OPENROUTER_API_BASE, the same pattern as the MOONSHOT_API_BASE precedent. OpenRouter's TypeScript SDK reads only OPENROUTER_BASE_URL. Both now point at one port. When a user has set both, OPENROUTER_BASE_URL wins as the upstream.

_ProviderConfig.base_env: str became base_envs: tuple[str, ...], plus an optional label for the report's provider name.

Claude Code on a third-party Anthropic-compatible endpoint was already metered, because a user-set ANTHROPIC_BASE_URL is the upstream. test_claude_code_on_a_third_party_anthropic_endpoint_is_metered proves it for DeepSeek's https://api.deepseek.com/anthropic:

• the path prefix is kept (/anthropic/v1/messages);
• the call is priced at the model's vendor rates (DeepSeek's hit price), not the port's.

Pricing

The site's table is the source. PriceRow gains three optional columns. Older clients ignore them (pydantic ignores extra fields), and the on-disk cache is written with exclude_none.

• cache_read_per_mtok
• cache_write_per_mtok: a standard write, which is Anthropic's 5-minute TTL
• cache_write_1h_per_mtok: Anthropic's 1-hour write

pricing.py parses and computes; it holds no prices of its own.

Interim fallback, to be deleted. src/trap/cost/interim_cache_rates.py holds each vendor's documented cache multiple of the input rate, with its source. Its docstring and its test file (tests/test_interim_cache_rates.py) say to delete both once /api/pricing serves the three columns. It is deliberately narrow:

• Per rate: a rate the row serves always wins; the fallback only fills a missing one. So a Claude row that serves read/write but not yet the 1-hour rate still prices a 1-hour write.
• Native ids only: a routed id (vendor/model, e.g. OpenRouter) is priced from served rates or not at all. There is no routing-exception logic.
• Families covered:
  • Claude: read 0.1× (0.025× on Fable/Mythos 5.1), write 1.25×, 1-hour write 2×.
  • OpenAI: read 0.1× on gpt-5.x/6, 0.25× on gpt-4.1/o3/o4-mini, 0.5× on gpt-4o/o1/o3-mini, 1× on -pro; write 1.25× on GPT-5.6+ and gpt-6.
  • DeepSeek: 0.02× on deepseek-flash/-v4-flash, 1/30× on -v4-pro.
  • Kimi: 0.1×–0.2×, depending on the model.

Unknown stays null. An unknown model, or cached tokens at an unknown rate, gives cost_usd = null. cache_cost_usd follows cost_usd, and both are contagious through the folds, exactly like cost_usd was.

What trapstreet-web needs (not in this PR; target its uat branch)

Field names below were checked against the live sources on 2026-09-11.

1. Serve the cache columns on /api/pricing. This is the main path, and it lets the CLI's interim table be deleted.

   • db/schema.ts model_pricing: add three nullable doublePrecision columns, cache_read_per_mtok, cache_write_per_mtok and cache_write_1h_per_mtok. Null means unknown; 0 is a real price (e.g. DeepSeek's free writes).

   • lib/pricing-sync.ts: read the rates from the sources it already fetches:

     Source	Cache read	Cache write	1-hour write
     models.dev	cost.cache_read	cost.cache_write	none
     LiteLLM	cache_read_input_token_cost	cache_creation_input_token_cost	cache_creation_input_token_cost_above_1hr
     OpenRouter	input_cache_read	input_cache_write	input_cache_write_1h

     A rate the supplier lacks, such as models.dev's 1-hour rate, can come from the referee.

   • planSync's changed check must compare the cache rates too. Today it compares only input/output. The table is append-only, so without this an existing row never gains cache prices. The first sync after deploy appends one new row per cached model.

   • lib/model-pricing.ts: ModelPrice / latestPerPrefix carry the three fields, and /api/pricing then serves them as-is. This is additive, so older CLIs ignore the fields.

   That fixes every null case above, including all routed OpenRouter ids, with no CLI release. Afterwards interim_cache_rates.py is deleted in one commit and default_prices.json is re-captured.

2. Price the cache on the server too. priceTokensDetailed (lib/model-pricing.ts, used by the v2/runs/{id}/usage route and the run-context mirror) prices only input/output and reports cache as cache_tokens_unpriced. With served rates it can price cache_read / cache_creation, leaving only rate-less tokens unpriced. (scripts/recalc-null-costs.ts is already gone on uat.)

3. Fix DeepSeek's price rows (a data fix, and the one the DeepSeek experiment needs first).

   • Production's /api/pricing has no deepseek-flash row. That is DeepSeek's current flagship id and the one its /anthropic endpoint maps Sonnet/Haiku to, and deepseek-v4-flash does not prefix-match it. So a call reporting deepseek-flash prices as null today.
   • models.dev already lists deepseek-flash, and production still serves deepseek-v4-flash at its pre-2026-09-10 rate. That suggests the daily sync has not landed since; sync_state and the logs are worth a look.
   • The sources also disagree on deepseek-v4-pro: models.dev has 0.435 input / 0.003625 cached, LiteLLM has 1.32 / 0.044, which matches DeepSeek's peak list. DeepSeek is prefer-primary (models.dev), so the sync would take the former. The site needs to decide which DeepSeek rate is "the" list price, since DeepSeek charges peak/off-peak.

4. Nothing needed for ingestion. /api/submit stores by_model verbatim. The run context's usageEntry already has cache_read / cache_creation; tp now fills them, and a new contract test checks every key tp sends is one the spec keeps.

Known gaps (verified, out of scope)

• Codex CLI isn't metered out of the box. It dropped OPENAI_BASE_URL (Remove OPENAI_BASE_URL config fallback openai/codex#16720, 2026-04); the base URL is now the openai_base_url config key. It also tries a WebSocket transport first, which the proxy doesn't read.
• OpenRouter reports one cache_write_tokens total with no TTL. It's priced as a 5-minute write. OpenRouter's own usage.cost is the exact charge and could serve as a cross-check later. I kept one pricing source so runs stay comparable.
• Some OpenRouter response model ids don't match the catalogue slug, e.g. anthropic/claude-4.7-opus-20260416 vs anthropic/claude-opus-4.7. That's a pre-existing prefix-match miss (null cost), not something this PR introduces.
• DeepSeek:
  • Off-peak pricing halves both rates, so the ratio holds; the row's base rate decides.
  • deepseek-v4-pro reroutes to Flash billing on 2026-09-14.
  • The usage fields of the /anthropic endpoint are undocumented, so that test uses Anthropic's own shape and says so.
• The bundled default_prices.json snapshot is not refreshed here, to keep the diff reviewable. It has no native deepseek-* rows at all, so offline or first-run DeepSeek costs are null until the live table is fetched. The live table has a few, but not deepseek-flash (see site item 3).
• Not modelled: Anthropic's compaction beta (usage.iterations sits outside the top-level counts), and long-context / batch / flex / priority tiers.

Testing

• tests/test_usage_shapes.py (new): each provider × streaming/non-streaming, from verbatim documented or recorded samples with their source:
  • Anthropic caching and streaming samples;
  • openai-python's recorded chat stream;
  • OpenAI openapi.yaml Responses samples;
  • DeepSeek's API reference;
  • an OpenRouter recording from opencode's fixtures;
  • Kimi's streaming guide;
  • Mistral's caching page.
• tests/test_pricing.py: cache arithmetic from served rates, including the 1-hour column; per-rate precedence (served, then interim); routed ids priced only from served rates; unknown → null.
• tests/test_interim_cache_rates.py (new, to be deleted with its module): every interim multiple family, and routed ids getting no fallback.
• tests/test_cost.py: CostProxy end to end through a fake upstream for the DeepSeek harness stream, its web search, OpenRouter via OPENROUTER_API_BASE, Claude Code on DeepSeek's /anthropic, and a Responses-API stream. Also covers base-URL precedence and the bucket fold.
• tests/test_models.py: the report.json round trip of the cache breakdown per case and per model, legacy reports still loading, and unknown being contagious.
• tests/test_context.py / tests/test_contract_web.py: cache counts in the site's own split, checked against the web's usageEntry spec.

Local results: uv run pytest --cov gives 760 passed at 100% coverage; ruff check, ruff format --check, pyright src/ and the docs lint are clean. With TRAPSTREET_WEB_SRC pointed at trapstreet-web's uat tree, the contract tests pass except test_every_reason_the_cli_can_quote_is_one_the_bulk_route_gives. That one already fails on main: it's grading-reason drift and unrelated to cost.

Installing this branch locally, before a release

The version comes from git tags, so fetch them. Without tags the build falls back to 0.0.0.dev0, which the site's build gate may refuse on a graded run.

git -C ~/Documents/Projects/trap fetch origin --tags
git -C ~/Documents/Projects/trap switch feat/cache-cost-accounting
uv tool install --force ~/Documents/Projects/trap
tp --version   # 0.0.15.devN+g<sha>

uv tool install --force replaces the installed tp for every run. To go back to the release: uv tool install --force trap-cli.

To try it without touching the installed tp, run it ephemerally from the checkout:

uvx --from ~/Documents/Projects/trap tp run …

No release is cut.

🤖 Generated with Claude Code