fix(cache): improve Anthropic prompt cache hit rate with system split and tool stability

[bhagirathsinh-vaghela]

Issue for this PR

Closes #5416, #5224
Related: #14065, #5422, #14203

Type of change

• Bug fix

What does this PR do?

Fixes cross-repo and cross-session Anthropic prompt cache misses. Same-session caching already works (AI SDK places markers correctly). This PR fixes the cases where the prefix changes between repos, sessions, or process restarts — causing full cache writes on every first prompt.

Anthropic hashes tools → system → messages in prefix order. Any change to an earlier block invalidates everything after it. OpenCode has several sources of unnecessary prefix changes.

Terminology (1-indexed): S1/S2 = system block 1/2. M1/M2 = cache marker on S1/S2.

Always-active fixes:

1. System prompt is a single block — dynamic content (env, project AGENTS.md) invalidates the stable provider prompt. Split into 2 blocks: stable (provider prompt + global AGENTS.md) first, dynamic (env + project) second.

2. Bash tool schema includes Instance.directory — changes per-repo, invalidating tool hash. Removed; model gets cwd from the environment block.

3. Skill tool ordering is nondeterministic — Object.values() on glob results. Sorted by name.

Opt-in fixes (behind env var flags):

4. Date and instructions change between turns — OPENCODE_EXPERIMENTAL_CACHE_STABILIZATION=1 freezes date and caches instruction file reads for the process lifetime.

5. Extended cache TTL — OPENCODE_EXPERIMENTAL_CACHE_1H_TTL=1 sets 1h TTL on M1 (2x write cost vs 1.25x for default 5-min). Useful for sessions with idle gaps.

Commits:

#	What	Behind flag?
1	Cache token audit logging	OPENCODE_CACHE_AUDIT
2	Stabilize system prefix (freeze date + instructions)	OPENCODE_EXPERIMENTAL_CACHE_STABILIZATION
3	Split system prompt into 2 blocks	Always active
4	Remove cwd from bash tool schema	Always active
5	Sort skill tool ordering	Always active
6	Optional 1h TTL on M1	OPENCODE_EXPERIMENTAL_CACHE_1H_TTL

What this doesn't fix:

• Per-project skills or MCP tools that differ across repos — the skill tool description changes per project, breaking M1 even on the same machine. This is expected; per-project tools are inherently dynamic.
• Cross-machine cache sharing (different skill tool descriptions per machine)
• Plan/build mode switches (TaskTool description changes per mode) — deferred
• Compaction cache alignment (/compact doesn't utilize prompt caching #10342 — planned follow-up)

Impact beyond Anthropic: The prefix stability fixes also benefit providers with automatic prefix caching (OpenAI, DeepSeek, Gemini, xAI, Groq) — no markers needed, just a stable prefix.

How did you verify your code works?

OPENCODE_CACHE_AUDIT=1 logs [CACHE] hit/miss per LLM call. Tested with Claude Sonnet 4.6 on Anthropic direct API, bun dev, Feb 23 2026.

Cross-repo (different folder, within 5-min TTL — the key improvement):

BEFORE (no fixes):

Prompt 1: hit=0.0%   read=0      write=17,786  new=3   (full miss, no reuse)
Prompt 2: hit=99.9%  read=17,786 write=10      new=3
Prompt 3: hit=99.9%  read=17,796 write=14      new=3

AFTER (system split + tool stability):

Prompt 1: hit=97.6%  read=17,345 write=428     new=3   (block 1 reused, only env block misses)
Prompt 2: hit=99.9%  read=17,773 write=10      new=3
Prompt 3: hit=99.9%  read=17,783 write=14      new=3

The first prompt in a new repo goes from 0% → 97.6% cache hit. S1 (tools + provider prompt + global AGENTS.md) is reused across repos. These numbers are based on my setup — S1 is ~17,345 tokens, mostly tool definitions (~12k tokens), with provider prompt (~2k) and global AGENTS.md (~2.8k) making up the rest. Your numbers will differ based on your tool set (MCP servers, skills) and global AGENTS.md size, but the cross-repo miss is eliminated regardless.

Only block 2 (env with different cwd = 428 tokens) is a cache write on the first prompt in a new repo.

To reproduce:

OPENCODE_CACHE_AUDIT=1 bun dev /tmp/folder-a
# send a prompt, exit
OPENCODE_CACHE_AUDIT=1 bun dev /tmp/folder-b
# send a prompt within 5 minutes
grep '\[CACHE\]' ~/.local/share/opencode/log/dev.log

Screenshots / recordings

N/A — no UI changes.

Updates (post-rebase, Mar 21 2026)

• Rebased onto current dev
• Dropped skill ordering commit (already merged via fix: stabilize agent and skill ordering in prompt descriptions #18261)
• fix: stabilize agent and skill ordering in prompt descriptions #18261 also added SystemPrompt.skills() which puts skill descriptions in the system prompt. Global skills (from ~/.config/opencode/skills/) are now placed in S1 (stable) and project skills in S2 (dynamic), so global skills don't cause cache writes on cross-repo switch. On my setup, cross-repo cache hit improved from 87% → 97.7%.
• Added commit 6: splitSystemPrompt provider config option. Providers that reject multiple system messages (e.g. llama.cpp with Qwen templates) can set provider.<id>.options.splitSystemPrompt: false to get single-block behavior.

Updated commit table:

#	What	Behind flag?
1	Cache token audit logging	OPENCODE_CACHE_AUDIT
2	Stabilize system prefix (freeze date + instructions)	OPENCODE_EXPERIMENTAL_CACHE_STABILIZATION
3	Split system prompt and skills into stable/dynamic blocks	Always active
4	Remove cwd from bash tool schema	Always active
5	Optional 1h TTL on M1	OPENCODE_EXPERIMENTAL_CACHE_1H_TTL
6	splitSystemPrompt provider option to opt out of split	Config: provider.<id>.options.splitSystemPrompt: false

Checklist

• I have tested my changes locally
• I have not included unrelated changes in this PR

[github-actions]

The following comment was made by an LLM, it may be inaccurate:

Potential related PRs found:

1. fix: split system prompt into 2 messages for proper Anthropic prompt caching #14203 - "fix: split system prompt into 2 messages for proper Anthropic prompt caching"

   • fix: split system prompt into 2 messages for proper Anthropic prompt caching #14203
   • Directly related: This appears to address the same system prompt splitting approach that is a core fix in PR fix(cache): improve Anthropic prompt cache hit rate with system split and tool stability #14743.

2. feat(provider): add provider-specific cache configuration system (significant token usage reduction) #5422 - "feat(provider): add provider-specific cache configuration system (significant token usage reduction)"

   • feat(provider): add provider-specific cache configuration system (significant token usage reduction) #5422
   • Related: This PR deals with provider-specific cache configuration and token usage optimization, which aligns with the cache improvements in fix(cache): improve Anthropic prompt cache hit rate with system split and tool stability #14743.

3. feat: add experimental compaction prompt and preserve prefix support #11492 - "feat: add experimental compaction prompt and preserve prefix support"

   • feat: add experimental compaction prompt and preserve prefix support #11492
   • Related: Addresses prefix preservation during compaction, which relates to the cache stability goals.

Note: PR #14203 appears to be the most directly related, as it's specifically about the system prompt splitting strategy that is a key component of PR #14743's improvements.

[github-actions]

Thanks for updating your PR! It now meets our contributing guidelines. 👍

[bhagirathsinh-vaghela]

Reviewer's guide — supplementary context not covered in the PR description. Uses same terminology (S1/S2, M1/M2) defined there.

---

AI SDK cache marker mechanics

Ref: Anthropic prompt caching docs | Anthropic engineers' caching best practices (Feb 19 2026): Thariq Shihipar, R. Lance Martin

Max 4 cache_control markers per request. The AI SDK already places markers on the first 2 system blocks and the last 2 conversation turns. That part works — the problem is OpenCode mutating blocks before these markers, cascading hash changes downstream.

Key subtlety: before this PR, OpenCode had a single system block. M1 covered it, but M2 was unused — it fell through to conversation. The system split (commit 3) is what activates both markers, letting S1 (stable) cache independently from S2 (dynamic).

Since M1 covers the tool block too (tools hash before system in Anthropic's ordering), any tool instability (commits 4–5) completely invalidates M1 — the entire cached prefix up to that marker is lost.

Related open PRs

Several open PRs address parts of this (#5422, #14203, #10380, #11492). This PR addresses the root causes directly.

Update (post-rebase, Mar 21 2026)

• Skill ordering commit dropped (covered by fix: stabilize agent and skill ordering in prompt descriptions #18261)
• Global skills now placed in S1 (stable block) alongside provider prompt and global AGENTS.md. Project skills remain in S2. This was needed because fix: stabilize agent and skill ordering in prompt descriptions #18261 added SystemPrompt.skills() to the system prompt during the rebase.
• Added splitSystemPrompt provider option (default: on). Providers that reject multiple system messages can set splitSystemPrompt: false in their provider config to get single-block behavior.

[bhagirathsinh-vaghela]

CI failure seems pre-existing — same NotFoundError affecting all PRs since the Windows path fixes landed in dev. Unrelated to this PR. All other checks pass.

[ShanePresley]

I pulled this into my fork and it's working beautifully. Unfortunately I only found this after getting a huge bill from Anthropic. Thanks OpenCode!

[TomLucidor]

@bhagirathsinh-vaghela could you check this with SLMs like Qwen3 or Nemotron or Kimi-Linear or GPT-OSS? Or providers using the OpenAI-compatible APIs (e.g. OpenRouter)?
Also why are some of the E2E tests failing in OpenCode PR?

Bonus ask: would Speculative Decoding work with this fork? I am looking at this from the lens of vLLM-MLX and MLX-OpenAI-Server (for non-MLX there is vLLM).

[bhagirathsinh-vaghela]

@bhagirathsinh-vaghela could you check this with SLMs like Qwen3 or Nemotron or Kimi-Linear or GPT-OSS? Or providers using the OpenAI-compatible APIs (e.g. OpenRouter)? Also why are some of the E2E tests failing in OpenCode PR?

Bonus ask: would Speculative Decoding work with this fork? I am looking at this from the lens of vLLM-MLX and MLX-OpenAI-Server (for non-MLX there is vLLM).

@TomLucidor

The fixes are provider/model-agnostic — they stabilize the request prefix so it is byte-for-byte identical across calls. Any provider with server-side prefix caching benefits automatically. See my reviewer's guide comment above for the full breakdown of each fix.

The specific model behind the provider does not matter — the changes are purely at the request layer. You can verify with any provider using OPENCODE_CACHE_AUDIT=1 to see hit/miss per call.

E2E failures — pre-existing upstream issue, since fixed. CI is green now.

Speculative decoding — orthogonal. This PR only changes what is sent in the request, not how the server processes it.

[kamelkace]

Would it make sense to change the wording here, to hint to the LLM that this isn't a live updating value? Otherwise it might make some weird choices elsewhere for long lived conversations. E.g.

Suggested change

	` Today's date: ${date.toDateString()}`,
	` Session started at: ${date.toDateString()}`,

[bhagirathsinh-vaghela]

Good point — this is better to show when the date is frozen. I'm keeping Today's date in this PR for now since it's what all OpenCode users expect(at least by experience even if they are not aware), but I'm not against the change if maintainers agree.

Separately, I've been experimenting locally with a progressive disclosure approach — making the env block fully static, instructing the model to fetch cwd, date, platform, etc. via tool calls when needed. Eliminates the block 2 cache write entirely at the cost of an occasional extra round-trip.

Interesting finding in this approach: completely removing the env block tended to result in models not bothering to fetch the info at all and assume things which is non deterministic. A static block with explicit "figure out when needed" instructions worked much better, at least with Anthropic models.

[kamelkace]

Separately, I've been experimenting locally with a progressive disclosure approach — making the env block fully static, instructing the model to fetch cwd, date, platform, etc. via tool calls when needed. [...] A static block with explicit "figure out when needed" instructions worked much better, at least with Anthropic models.

Hmm! I'll have to give that a shot when I patch from this PR later; I'm running locally against one of the Qwen3.5 models, so it'll be interesting data to see how they respond.

[fkroener]

Looking forward to seeing less prompt re-processing with opencode. Unfortunately it seems currently this patchset breaks llama.cpp support:

[60919] srv operator(): got exception: {"error":{"code":400,"message":"Unable to generate parser for this template. Automatic parser generation failed: \n------------\nWhile executing CallExpression at line 85, column 32 in source:\n...first %}↵ {{- raise_exception('System message must be at the beginnin...\n ^\nError: Jinja Exception: System message must be at the beginning.","type":"invalid_request_error"}}

Tested with and without the new autoparser. Maybe I'm using it wrong?

[fkroener]

So, after partially reverting fix(cache): split system prompt into 2 blocks for independent caching, or rather naively ensuring llama.cpp gets just one system prompt (revert.patch) opencode now flies with this patchset using a llama.cpp endpoint (openai api though).

No more "erased invalidated context checkpoint" for all checkpoints and reprocessing of the entire context seemingly whenever I send a new query.

Checkpoint reuse happens usually at around 99 %, sometimes drops to 93 % - lowest was in the 70 % with > 60k tokens.

Much appreciated!

Wonder whether the split system message is something @pwilkin would be willing to support or whether it should be guarded to only be sent to Antrophic endpoints.

[pwilkin]

Any chance the system message could be moved to the top of the messages list? We could possibly do this for the Anthropic API, but technically the system prompt should be the first message.

[fkroener]

Thanks @pwilkin. Given this is actually coming from the model template (Qwen 3.5) and not the parser:

    {%- if message.role == "system" %}
        {%- if not loop.first %}
            {{- raise_exception('System message must be at the beginning.') }}
        {%- endif %}
    {%- elif message.role == "user" %}

this should probably best be handled on OpenCode's end.

[spacepluk]

Is there any reason not to apply the 1h TTL to all the cache breakpoints? Right now it's only for the system portion if I'm not mistaken.
@spacepluk
Src: https://platform.claude.com/docs/en/build-with-claude/prompt-caching#pricing

I personally am not willing to pay for `2x` cache write price for the messages portion of my sessions. In my setup, I have modded my OpenCode to have this status line with the cache expiry timestamp from Anthropic's server-side response and a daemon scanning all my tmux panes to find all OpenCode sessions with this timestamp and keeping my cache warm by sending `ping` messages. I pay `0.1x` price for entire context window for cache reads per ping but since leaving open sessions unattended for more than 5m is rare in my workflow, the pings rarely fire and when they do it is still cheap. Honestly, if you just have normal sessions where you are pretty much actively getting work done, paying `1.25x` for 5m works out way better than paying `2x` for the 1h marker. Yes, 1h TTL means fewer cache misses on the read side over a session's lifetime, but without a daemon keeping the cache warm you can have a session cache go stale for 1h even and then that miss costs you the full `2x` cache write anyway.

That said, making it configurable for the messages markers between 5m and 1h is trivial if you need.

Anthropic has made Claude Code use the 1h markers for all of their markers including the messages portion (at least the main chat) which is overkill IMO. Everyone is paying the 2x price unnecessarily without realizing it. I wouldn't want it for my active interactive sessions at least since I see no value in it.

That makes sense but I think it's heavily dependent on your usage patterns. I haven't tried automating pings like that but not all of my sessions are running in tmux so I would need a different solution for that.

I usually have 5-10 long-running sessions that will last the whole day (or more) and context-switching/testing consistently takes longer than 5m. In that case the 1h cache is completely worth it in my opinion. I'll still use the 5m cache for short/active sessions though. I've been tracking my cost for a few weeks and I've greatly reduced costs thanks to your patches + 1h cache for everything with an additional patch in my fork.

I would suggest making OPENCODE_EXPERIMENTAL_CACHE_1H_TTL apply to all messages (like claude-code's ENABLE_PROMPT_CACHING_1H) and maybe have a separate SYSONLY option.

[martinffx]

I've tried to break it up into 4 seperate PR's to make it easier to review and merge

1. feat(cache): add cache token audit logging behind OPENCODE_EXPERIMENTAL_CACHE_AUDIT
   
2. feat(cache): split system prompt into stable/dynamic blocks for independent caching
3. fix(cache): stabilize system prefix behind OPENCODE_EXPERIMENTAL_CACHE_STABILIZATION
4. TODO: feat(cache): add optional 1h TTL on first system cache marker

[spacepluk]

In case it helps, here are my patches to use 1h for all the cache breakpoints and another one that implements my SYSONLY suggestion above

[spacepluk]

FYI I had to disable the system split for Gemini because it was ignoring the system prompt completely.

{
  "provider": {
    "google": {
      "options": {
        "splitSystemPrompt": false
      }
    }
  }
}

[SamirSaidani]

@martinffx all your PRs have been closed by github bots. Aren't we both losing money and wasting energy by not taking proper care of prompt caching?

[Arrowyi]

@rekram1-node @thdxr — production evidence that the cache problem is worse than this PR assumes

I filed #40790 with production data from 5 Claude Opus 5 Max sessions: $24.45 out of $38.35 (63.8%) was cache writes. The worst session (ses_03563) had 9 cache busts across 43 messages, including 5 consecutive busts (messages #34–38) each costing $1.63.

Why this matters for #14743:

This PR's premise is "Same-session caching already works." My data contradicts that. All 9 busts happened within the same session, triggered by the known tool-call prompt loop bug (#24841): the loop reloads messages from DB, and tool part states change from pending → completed, producing different serialized bytes → Anthropic cache bust.

What #14743 would fix and what it wouldn't:

• ✅ Would fix: cross-repo system/tool prefix instability → saves ~12K tokens of tool definitions per bust
• ❌ Would NOT fix: the within-session conversation busts I'm seeing (prompt-loop DB reload changes message bytes)

In my case, even with this PR merged, I'd still see ~$0.50 busts (conversation portion) instead of $1.63 (full bust). Better, but not solved.

Two complementary fixes are needed:

1. This PR (fix(cache): improve Anthropic prompt cache hit rate with system split and tool stability #14743): stabilizes the system/tool prefix across repos and sessions
2. A follow-up: cache messages across the prompt loop to prevent DB-reload-induced byte drift (Production evidence: Anthropic cache bust bug causes $14.82 waste across 3 sessions (22.5% bust rate) #40790 root cause)

I've verified this with database queries anyone can run:

-- Verify busts happen exclusively after tool-calls:
WITH msg_list AS (
  SELECT json_extract(data, '$.finish') as finish,
         json_extract(data, '$.tokens.cache.read') as cache_r,
         json_extract(data, '$.tokens.cache.write') as cache_w,
         ROW_NUMBER() OVER (ORDER BY json_extract(data, '$.time.created')) as rn
  FROM message WHERE session_id = '<your-session-id>'
    AND json_extract(data, '$.role') = 'assistant'
)
SELECT m1.rn, m1.cache_r, m1.cache_w, m2.finish as prev_finish
FROM msg_list m1 LEFT JOIN msg_list m2 ON m1.rn = m2.rn + 1
WHERE m1.cache_r = 0 AND m1.cache_w > 0;

Bottom line: This PR has been sitting for 3 months with 14+ participants and multiple rebases. marcusquinn's rebased branch (pr-14743-prompt-caching-update) and martinffx's split PRs (#27007, #27377, #27378) have all been auto-closed without review. The cache problem is costing real money — merging this would be a meaningful first step even if it doesn't solve everything.

Happy to provide more data or help move this forward however I can.

[TomLucidor]

@Arrowyi could you replicate this with other model providers? Wondering if Kimi/GLM/DeepSeek/MiMo would have similar problems, or if it is server-side

[Arrowyi]

Sorry for the slow turnaround, @TomLucidor — here's a cross-provider check from my local OpenCode database (production usage on the same harness; not a controlled experiment):

DeepSeek (v4-pro) — rewrites exist, but not in the #40790 shape. In my largest session (ses_f09663f2bffe1OYV2uKbMmz3hH, 204 turns), the 18 full-context rewrites are almost all TTL-shaped: 16 followed idle gaps ≥6.4 min (turns after 1–5 min gaps hit cache 22/22 times in this session), 1 followed an aborted generation + resend, and 1 followed a 21 s gap I can't attribute. Unlike Anthropic, there is no separate cache-write price — those re-reads bill at standard input rates.

Kimi (k3 / k2.7-code) — 34 sessions, 1,901 assistant turns: 0 full-context rewrites (271M cache reads, 0 cache writes).

GLM (5.2 / 5.3) — 45 sessions, 725 turns: 0 full-context rewrites (66M cache reads; only small incremental writes, 179K total).

MiMo — no usage on my side, so no data.

Anthropic (with caveats) — my worst session from #40790 is no longer in the live DB, but the original investigation's query outputs are still stored and I re-verified them (session totals, the per-message table, and the run of 5 consecutive rewrites all match the thread). The other two archived Opus sessions show only post-idle rewrites (6–15 min idle), i.e. TTL behavior rather than in-loop rewrites. I stopped running Anthropic workloads because of the cost, so I can't confirm whether the in-loop pattern still reproduces on current builds (recorded on 1.18.9; I'm now on 1.18.32, with several cache fixes merged in late September).

Net: in my data the expensive in-loop shape shows up on the Anthropic route; DeepSeek only shows TTL-boundary rewrites (without a write premium), and Kimi/GLM show none at all. So it doesn't look like something the harness does to every provider — beyond that I can't localize it from usage data alone. Happy to re-run the same workload on Anthropic (before/after) whenever a fix or diagnostic lands.

_{Criterion for "full-context rewrite": assistant turn with cache_read = 0 and cache_write > 1k, excluding session-start writes.}

[thdxr]

This branch has been upgraded to V2. We're no longer taking PRs for V1. If you think this is still relevant, please port it over and open it for V2.

— from 𝕺𝖕𝖊𝖓𝕮𝖔𝖉𝖊

[bhagirathsinh-vaghela]

Thanks to everyone who tested this, and to marcusquinn and martinffx for rebasing and splitting it. It was closed in the V1 freeze, and I'm not planning to port it to V2 myself.

I'd said more was coming. It's in my fork, Faber, built on OpenCode v1.1.50. Over three months of my own use (161k Anthropic requests):

• 99% of input tokens came from cache, every month.
• Compaction reads the cache. Since September it costs about a ninth of a cold summary.
• Subagents start on the parent's cache (99% of their first requests hit it), can be limited to a read-only tool preset, and their results arrive exactly once, even across a server restart.
• Shell commands survive a server restart. Anything past 5 seconds becomes a background job that reports back.
• Voice dictation and read-aloud, from a desktop or a phone.

I've only tested this with Anthropic models, and some of the mechanisms are Anthropic-specific (cache markers, the 1h lifetime, keep-warm timing). The principles apply to any provider with prefix caching: keep the prefix byte-stable, append instead of editing, and compact on the cached prefix. This page shows how they carry over, provider by provider.

It's missing a lot of newer upstream work (the TUI, Windows, the newer provider integrations). It also has no switch to turn off the system split yet, so llama.cpp/Qwen and Gemini setups will break on it. Use it as is, or port whatever ideas help to V2. There's a getting started guide.

Thanks to the OpenCode team for the base I built it on.