Skip to content

ENG-648: Reduce token usage — prompt caching, prompts→skills, cheap router tier#239

Draft
torrmal wants to merge 11 commits into
stagingfrom
jorge/eng-648-reduce-anton-cowork-server-token-usage-promptskills
Draft

ENG-648: Reduce token usage — prompt caching, prompts→skills, cheap router tier#239
torrmal wants to merge 11 commits into
stagingfrom
jorge/eng-648-reduce-anton-cowork-server-token-usage-promptskills

Conversation

@torrmal

@torrmal torrmal commented Jul 7, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Linear: ENG-648

Full analysis + plan in plans/eng-648-token-efficiency.md. Phases 2 and 3 are implemented in this PR; Phase 1 (prompt caching) is pending a design decision with the mindshub_inference gateway.

Problem

Every turn — even a trivial "answer from context" one — paid for a ~11.4K-token system prompt, ~4.2K tokens of tool schemas, the full conversation history, and the frontier planning model, multiplied by up to 25 tool rounds per turn, with no prompt caching at all.

What this PR does

Phase 3 — Cheap front-model router (respond-or-delegate) · anton/core/llm/router.py

With ANTON_ROUTER_ENABLED=true, a cost-effective router model (new router role on LLMClient, defaults to the coding role, i.e. Haiku) runs first on every text turn with a ~420-token prompt and no tool schemas, over a condensed text-only view of history, and either:

  • answers directly when the reply is fully derivable from the conversation + stable general knowledge (planning model never invoked — the turn costs a single small Haiku call), or
  • delegates via a forced delegate tool call, optionally naming skills that get preloaded into history as a synthetic recall_skill exchange — payload-identical to what the planning model would have fetched itself, minus the full-context planning round.

Conservative by construction: image turns, truncated router answers, empty responses, unknown skill labels, and router exceptions all fall through to the planning model. Off by default until eval'd.

Phase 2 — Prompt tutorials → built-in skills · anton/core/llm/builtin_skills.py

~57% of the system prompt was two always-on tutorials + a URL catalog. They now ship as built-in skills — html-dashboards (~2.7K tok), backend-apps (~3.9K tok), public-data-sources (~0.7K tok) — listed as one-line entries in the procedural-memory section and loaded on demand via the existing recall_skill tool or the router's preload path. Built-ins resolve before the on-disk SkillStore and shadow same-labelled user skills. Prompt content is unchanged; it just moved.

Measured: base system prompt 45,511 → 18,847 chars (~11,377 → ~4,711 tokens, −59%).

Test coverage

tests/test_router.py (19 tests: condensation, decision parsing, session integration, fail-open) and tests/test_builtin_skills.py (12 tests: registry, prompt slimming with a <25K-char regression guard, recall resolution, router preload). Full suite passes (1 pre-existing unrelated failure in test_chat_scratchpad.py, fails on clean main too).

Still to come (this ticket)

  • Phase 1 — prompt caching: cache_control breakpoints in AnthropicProvider + passthrough/usage-surfacing decisions in mindshub_inference (which already has a gateway-side cache_align heuristic; client-driven breakpoints need schema + usage plumbing there).
  • Phase 0 — per-turn token metrics through the harness event stream.
  • Phase 4 — cowork-server diet: separate PR in mindsdb/cowork-server.

🤖 Generated with Claude Code

This was referenced Jul 8, 2026
Open
Merged
torrmal and others added 7 commits July 7, 2026 23:31
@torrmal @claude
ENG-648: token-efficiency analysis and plan (prompt caching, prompts→…
1d6b9b7 
…skills, cheap router tier)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@torrmal @claude
ENG-648 Phase 3: cheap front-model router (respond-or-delegate)
dca724a 
Every turn used to go straight to the planning model with the full
~11.4K-token system prompt and ~4.2K tokens of tool schemas. With
ANTON_ROUTER_ENABLED=true, a cost-effective router model now runs first
with a ~400-token prompt and no tool schemas, and either:

- answers trivial/from-context turns directly (planning model never
  invoked), or
- delegates via a forced tool call, optionally naming procedural skills
  that get preloaded into history as a synthetic recall_skill exchange
  so the planning model doesn't spend a full-context round fetching them.

Conservative by construction: image turns, truncated answers, empty
responses, and router errors all fall through to the planning model.
New LLMClient router role defaults to the coding provider/model, so
hosts constructing LLMClient directly need no changes. Off by default.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@torrmal @claude
ENG-648 Phase 2: move prompt tutorials into built-in skills
75667bd 
The always-on system prompt carried two full tutorials — backend/
fullstack app generation (~3.8K tokens) and the HTML-dashboard build
discipline (~2.7K tokens) — plus a public-data-source catalog (~0.7K),
sent on every call of every tool round regardless of the task.

They now ship as built-in skills (html-dashboards, backend-apps,
public-data-sources) listed as one-line entries in the procedural
memory section and loaded on demand: the model calls recall_skill, or
the router preloads them at delegation time. Built-ins resolve before
the on-disk SkillStore and shadow same-labelled user skills.

Measured base system prompt: 45,511 -> 18,847 chars (~11,377 -> ~4,711
tokens, -59%) in dashboards mode. Prompt content itself is unchanged —
it just moved; brace escapes collapse at recall exactly as they did at
build time.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@torrmal @claude
ENG-648: rename the front-model gate to 'thalamus'
e8a8b36 
anton names its subsystems after brain regions (Cortex, Cerebellum,
AnteriorCingulate, hippocampus). The respond-or-delegate front-model is
the thalamus: the brain's central relay, which every signal hits first
and which gates what reaches the cortex versus what a fast subcortical
path handles. That IS respond-vs-delegate.

The biology also maps onto behavior we already had, and names it:
- default-inhibit (thalamic reticular nucleus 'searchlight') = the
  conservative 'when in doubt, delegate' bias;
- tonic vs burst firing = respond-directly vs alert-the-cortex;
- fail-open (a lesioned thalamus must not silently drop signals) = any
  gating error relays the turn up untouched;
- corticothalamic feedback (cortex biases the gate) = documented seam at
  gate_turn for letting recent outcomes tune future gating (not built).

Mechanical rename, no behavior change: router.py -> thalamus.py,
route_turn -> gate_turn, RouterDecision -> ThalamicDecision, LLMClient
.route -> .gate + thalamus_provider/thalamus_model, ANTON_ROUTER_* ->
ANTON_THALAMUS_*. Full suite passes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@torrmal @claude
ENG-660: route history summarization through the thalamus role
22db3a8 
The cheap front-model that gates turns now also owns conversation
summarization, so hosts can expose a single 'routing and summarization'
model choice distinct from the coding model (scratchpad code gen).

New LLMClient.summarize() runs on the thalamus provider/model;
_summarize_history calls it instead of code(). thalamus_* still default
to the coding role, so this is behavior-preserving unless a distinct
routing/summarization model is configured.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@torrmal @claude
ENG-660: name the config surface router_* (mechanism stays thalamus)
edd5087 
Keeps one vocabulary end-to-end: the internal mechanism is the thalamus
(class, module, gate(), ThalamicDecision), but everything a user or host
configures is 'router' — router_enabled / router_max_tokens (CoreSettings),
router_provider / router_model (AntonSettings), ANTON_ROUTER_* env. This
matches cowork-server's router_provider/router_model settings 1:1 so
there's no incongruence across repos. LLMClient's constructor kwargs stay
thalamus_provider/thalamus_model (mechanism); from_settings maps the
router_* settings onto them.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@torrmal @claude
ENG-660: normalize router_provider minds-cloud->openai-compatible
7b5b78d 
Extends #240's AntonSettings validator to cover router_provider. Without
it, LLMClient.from_settings validates the router role the same way as
planning/coding and a shared minds-cloud config re-crashes with 'Unknown
router provider: minds-cloud' — the ENG-655 bug for the routing role.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@torrmal
torrmal force-pushed the jorge/eng-648-reduce-anton-cowork-server-token-usage-promptskills branch from be306b7 to 7b5b78d Compare July 8, 2026 06:34
@torrmal
torrmal changed the base branch from main to staging July 8, 2026 06:34
@torrmal
torrmal requested review from SailingSF and ea-rus July 10, 2026 00:21
ea-rus and others added 2 commits July 10, 2026 19:05
@ea-rus
merge from staging
0cf77c2
@ea-rus @claude
fix(session): stop reusing turn_count for thalamus preload tool_use ids
93ec5f3 
turn() (unlike turn_stream()) never increments _turn_count, so
_inject_recalled_skills emitted the same tool_use id on every call made
through the non-streaming turn() API when the thalamus delegated with
skill preloads — duplicate ids in history can break provider validation
on subsequent turns. Use a dedicated monotonic counter instead.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ea-rus ea-rus ea-rus approved these changes
@SailingSF SailingSF Awaiting requested review from SailingSF

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@torrmal @ea-rus