fix(cache): improve Anthropic prompt cache hit rate with system split and tool stability

[bhagirathsinh-vaghela]

Issue for this PR

Closes#5416, #5224
Related: #14065, #5422, #14203

Type of change

• Bug fix

What does this PR do?

Fixes cross-repo and cross-session Anthropic prompt cache misses. Same-session caching already works (AI SDK places markers correctly). This PR fixes the cases where the prefix changes between repos, sessions, or process restarts — causing full cache writes on every first prompt.

Anthropic hashes tools → system → messages in prefix order. Any change to an earlier block invalidates everything after it. OpenCode has several sources of unnecessary prefix changes.

Terminology (1-indexed): S1/S2 = system block 1/2. M1/M2 = cache marker on S1/S2.

Always-active fixes:

1. System prompt is a single block — dynamic content (env, project AGENTS.md) invalidates the stable provider prompt. Split into 2 blocks: stable (provider prompt + global AGENTS.md) first, dynamic (env + project) second.

2. Bash tool schema includes Instance.directory — changes per-repo, invalidating tool hash. Removed; model gets cwd from the environment block.

3. Skill tool ordering is nondeterministic — Object.values() on glob results. Sorted by name.

Opt-in fixes (behind env var flags):

4. Date and instructions change between turns — OPENCODE_EXPERIMENTAL_CACHE_STABILIZATION=1 freezes date and caches instruction file reads for the process lifetime.

5. Extended cache TTL — OPENCODE_EXPERIMENTAL_CACHE_1H_TTL=1 sets 1h TTL on M1 (2x write cost vs 1.25x for default 5-min). Useful for sessions with idle gaps.

Commits:

#	What	Behind flag?
1	Cache token audit logging	OPENCODE_CACHE_AUDIT
2	Stabilize system prefix (freeze date + instructions)	OPENCODE_EXPERIMENTAL_CACHE_STABILIZATION
3	Split system prompt into 2 blocks	Always active
4	Remove cwd from bash tool schema	Always active
5	Sort skill tool ordering	Always active
6	Optional 1h TTL on M1	OPENCODE_EXPERIMENTAL_CACHE_1H_TTL

What this doesn't fix:

• Per-project skills or MCP tools that differ across repos — the skill tool description changes per project, breaking M1 even on the same machine. This is expected; per-project tools are inherently dynamic.
• Cross-machine cache sharing (different skill tool descriptions per machine)
• Plan/build mode switches (TaskTool description changes per mode) — deferred
• Compaction cache alignment (/compact doesn't utilize prompt caching #10342 — planned follow-up)

Impact beyond Anthropic: The prefix stability fixes also benefit providers with automatic prefix caching (OpenAI, DeepSeek, Gemini, xAI, Groq) — no markers needed, just a stable prefix.

How did you verify your code works?

OPENCODE_CACHE_AUDIT=1 logs [CACHE] hit/miss per LLM call. Tested with Claude Sonnet 4.6 on Anthropic direct API, bun dev, Feb 23 2026.

Cross-repo (different folder, within 5-min TTL — the key improvement):

BEFORE (no fixes):

Prompt 1: hit=0.0% read=0 write=17,786 new=3 (full miss, no reuse)
Prompt 2: hit=99.9% read=17,786 write=10 new=3
Prompt 3: hit=99.9% read=17,796 write=14 new=3

AFTER (system split + tool stability):

Prompt 1: hit=97.6% read=17,345 write=428 new=3 (block 1 reused, only env block misses)
Prompt 2: hit=99.9% read=17,773 write=10 new=3
Prompt 3: hit=99.9% read=17,783 write=14 new=3

The first prompt in a new repo goes from 0% → 97.6% cache hit. S1 (tools + provider prompt + global AGENTS.md) is reused across repos. These numbers are based on my setup — S1 is ~17,345 tokens, mostly tool definitions (~12k tokens), with provider prompt (~2k) and global AGENTS.md (~2.8k) making up the rest. Your numbers will differ based on your tool set (MCP servers, skills) and global AGENTS.md size, but the cross-repo miss is eliminated regardless.

Only block 2 (env with different cwd = 428 tokens) is a cache write on the first prompt in a new repo.

To reproduce:

OPENCODE_CACHE_AUDIT=1 bun dev /tmp/folder-a
# send a prompt, exit
OPENCODE_CACHE_AUDIT=1 bun dev /tmp/folder-b
# send a prompt within 5 minutes
grep '\[CACHE\]' ~/.local/share/opencode/log/dev.log

Screenshots / recordings

N/A — no UI changes.

Updates (post-rebase, Mar 21 2026)

• Rebased onto current dev
• Dropped skill ordering commit (already merged via fix: stabilize agent and skill ordering in prompt descriptions #18261)
• fix: stabilize agent and skill ordering in prompt descriptions #18261 also added SystemPrompt.skills() which puts skill descriptions in the system prompt. Global skills (from ~/.config/opencode/skills/) are now placed in S1 (stable) and project skills in S2 (dynamic), so global skills don't cause cache writes on cross-repo switch. On my setup, cross-repo cache hit improved from 87% → 97.7%.
• Added commit 6: splitSystemPrompt provider config option. Providers that reject multiple system messages (e.g. llama.cpp with Qwen templates) can set provider.<id>.options.splitSystemPrompt: false to get single-block behavior.

Updated commit table:

#	What	Behind flag?
1	Cache token audit logging	OPENCODE_CACHE_AUDIT
2	Stabilize system prefix (freeze date + instructions)	OPENCODE_EXPERIMENTAL_CACHE_STABILIZATION
3	Split system prompt and skills into stable/dynamic blocks	Always active
4	Remove cwd from bash tool schema	Always active
5	Optional 1h TTL on M1	OPENCODE_EXPERIMENTAL_CACHE_1H_TTL
6	splitSystemPrompt provider option to opt out of split	Config: provider.<id>.options.splitSystemPrompt: false

Checklist

• I have tested my changes locally
• I have not included unrelated changes in this PR

[github-actions]

The following comment was made by an LLM, it may be inaccurate:

Potential related PRs found:

1. fix: split system prompt into 2 messages for proper Anthropic prompt caching #14203 - "fix: split system prompt into 2 messages for proper Anthropic prompt caching"

   • fix: split system prompt into 2 messages for proper Anthropic prompt caching #14203
   • Directly related: This appears to address the same system prompt splitting approach that is a core fix in PR fix(cache): improve Anthropic prompt cache hit rate with system split and tool stability #14743.

2. feat(provider): add provider-specific cache configuration system (significant token usage reduction) #5422 - "feat(provider): add provider-specific cache configuration system (significant token usage reduction)"

   • feat(provider): add provider-specific cache configuration system (significant token usage reduction) #5422
   • Related: This PR deals with provider-specific cache configuration and token usage optimization, which aligns with the cache improvements in fix(cache): improve Anthropic prompt cache hit rate with system split and tool stability #14743.

3. feat: add experimental compaction prompt and preserve prefix support #11492 - "feat: add experimental compaction prompt and preserve prefix support"

   • feat: add experimental compaction prompt and preserve prefix support #11492
   • Related: Addresses prefix preservation during compaction, which relates to the cache stability goals.

Note: PR #14203 appears to be the most directly related, as it's specifically about the system prompt splitting strategy that is a key component of PR #14743's improvements.

[github-actions]

Thanks for updating your PR! It now meets our contributing guidelines. 👍

[bhagirathsinh-vaghela]

Reviewer's guide — supplementary context not covered in the PR description. Uses same terminology (S1/S2, M1/M2) defined there.

---

AI SDK cache marker mechanics

Ref: Anthropic prompt caching docs | Anthropic engineers' caching best practices (Feb 19 2026): Thariq Shihipar, R. Lance Martin

Max 4 cache_control markers per request. The AI SDK already places markers on the first 2 system blocks and the last 2 conversation turns. That part works — the problem is OpenCode mutating blocks before these markers, cascading hash changes downstream.

Key subtlety: before this PR, OpenCode had a single system block. M1 covered it, but M2 was unused — it fell through to conversation. The system split (commit 3) is what activates both markers, letting S1 (stable) cache independently from S2 (dynamic).

Since M1 covers the tool block too (tools hash before system in Anthropic's ordering), any tool instability (commits 4–5) completely invalidates M1 — the entire cached prefix up to that marker is lost.

Related open PRs

Several open PRs address parts of this (#5422, #14203, #10380, #11492). This PR addresses the root causes directly.

Update (post-rebase, Mar 21 2026)

• Skill ordering commit dropped (covered by fix: stabilize agent and skill ordering in prompt descriptions #18261)
• Global skills now placed in S1 (stable block) alongside provider prompt and global AGENTS.md. Project skills remain in S2. This was needed because fix: stabilize agent and skill ordering in prompt descriptions #18261 added SystemPrompt.skills() to the system prompt during the rebase.
• Added splitSystemPrompt provider option (default: on). Providers that reject multiple system messages can set splitSystemPrompt: false in their provider config to get single-block behavior.

[bhagirathsinh-vaghela]

CI failure seems pre-existing — same NotFoundError affecting all PRs since the Windows path fixes landed in dev. Unrelated to this PR. All other checks pass.

[ShanePresley]

I pulled this into my fork and it's working beautifully. Unfortunately I only found this after getting a huge bill from Anthropic. Thanks OpenCode!

[TomLucidor]

@bhagirathsinh-vaghela could you check this with SLMs like Qwen3 or Nemotron or Kimi-Linear or GPT-OSS? Or providers using the OpenAI-compatible APIs (e.g. OpenRouter)?
Also why are some of the E2E tests failing in OpenCode PR?

Bonus ask: would Speculative Decoding work with this fork? I am looking at this from the lens of vLLM-MLX and MLX-OpenAI-Server (for non-MLX there is vLLM).

[bhagirathsinh-vaghela]

@bhagirathsinh-vaghela could you check this with SLMs like Qwen3 or Nemotron or Kimi-Linear or GPT-OSS? Or providers using the OpenAI-compatible APIs (e.g. OpenRouter)? Also why are some of the E2E tests failing in OpenCode PR?

Bonus ask: would Speculative Decoding work with this fork? I am looking at this from the lens of vLLM-MLX and MLX-OpenAI-Server (for non-MLX there is vLLM).

@TomLucidor

The fixes are provider/model-agnostic — they stabilize the request prefix so it is byte-for-byte identical across calls. Any provider with server-side prefix caching benefits automatically. See my reviewer's guide comment above for the full breakdown of each fix.

The specific model behind the provider does not matter — the changes are purely at the request layer. You can verify with any provider using OPENCODE_CACHE_AUDIT=1 to see hit/miss per call.

E2E failures — pre-existing upstream issue, since fixed. CI is green now.

Speculative decoding — orthogonal. This PR only changes what is sent in the request, not how the server processes it.

[kamelkace]

Would it make sense to change the wording here, to hint to the LLM that this isn't a live updating value? Otherwise it might make some weird choices elsewhere for long lived conversations. E.g.

Suggested change

	` Today's date: ${date.toDateString()}`,
	` Session started at: ${date.toDateString()}`,

[bhagirathsinh-vaghela]

Good point — this is better to show when the date is frozen. I'm keeping Today's date in this PR for now since it's what all OpenCode users expect(at least by experience even if they are not aware), but I'm not against the change if maintainers agree.

Separately, I've been experimenting locally with a progressive disclosure approach — making the env block fully static, instructing the model to fetch cwd, date, platform, etc. via tool calls when needed. Eliminates the block 2 cache write entirely at the cost of an occasional extra round-trip.

Interesting finding in this approach: completely removing the env block tended to result in models not bothering to fetch the info at all and assume things which is non deterministic. A static block with explicit "figure out when needed" instructions worked much better, at least with Anthropic models.

[kamelkace]

Separately, I've been experimenting locally with a progressive disclosure approach — making the env block fully static, instructing the model to fetch cwd, date, platform, etc. via tool calls when needed. [...] A static block with explicit "figure out when needed" instructions worked much better, at least with Anthropic models.

Hmm! I'll have to give that a shot when I patch from this PR later; I'm running locally against one of the Qwen3.5 models, so it'll be interesting data to see how they respond.

[fkroener]

Looking forward to seeing less prompt re-processing with opencode. Unfortunately it seems currently this patchset breaks llama.cpp support:

[60919] srv operator(): got exception: {"error":{"code":400,"message":"Unable to generate parser for this template. Automatic parser generation failed: \n------------\nWhile executing CallExpression at line 85, column 32 in source:\n...first %}↵ {{- raise_exception('System message must be at the beginnin...\n ^\nError: Jinja Exception: System message must be at the beginning.","type":"invalid_request_error"}}

Tested with and without the new autoparser. Maybe I'm using it wrong?

[fkroener]

So, after partially reverting fix(cache): split system prompt into 2 blocks for independent caching, or rather naively ensuring llama.cpp gets just one system prompt (revert.patch) opencode now flies with this patchset using a llama.cpp endpoint (openai api though).

No more "erased invalidated context checkpoint" for all checkpoints and reprocessing of the entire context seemingly whenever I send a new query.

Checkpoint reuse happens usually at around 99 %, sometimes drops to 93 % - lowest was in the 70 % with > 60k tokens.

Much appreciated!

Wonder whether the split system message is something @pwilkin would be willing to support or whether it should be guarded to only be sent to Antrophic endpoints.

[pwilkin]

Any chance the system message could be moved to the top of the messages list? We could possibly do this for the Anthropic API, but technically the system prompt should be the first message.

[fkroener]

Thanks @pwilkin. Given this is actually coming from the model template (Qwen 3.5) and not the parser:

 {%- if message.role == "system" %}
{%- if not loop.first %}
{{- raise_exception('System message must be at the beginning.') }}
{%- endif %}
{%- elif message.role == "user" %}

this should probably best be handled on OpenCode's end.

[bhagirathsinh-vaghela]

@rekram1-node@thdxr
Just checking in on this. It's been nearly 3 months since this PR was opened, and almost 7 weeks since a maintainer said they'd merge it that day. No follow-up, no explanation.

Meanwhile, cache TTL mapping, auto-placement of cache markers, and breakpoint management just landed in packages/llm via separate PRs — covering the same core problem this PR identified and implemented months ago. None of those PRs reference this one, despite 14+ participants here and months of public discussion on exactly these topics.

This PR has merge conflicts that I'm not resolving until I know whether external contributions are actually being considered here. Is this going to be merged, or should I stop spending time on it?

[martinffx]

Meanwhile, cache TTL mapping, auto-placement of cache markers, and breakpoint management just landed in packages/llm via separate PRs

packages/llm seems to be a new EffectTS based package intended to replace AI SDK, not currently in the code path. Looks like part of a larger refactor moving to EffectTS, which is nice, but I wish they communicated a little be more about what they are working on.

[marcusquinn]

I rebased this patchset onto current dev (7235c9c9b) and pushed the conflict-resolved branch here:

marcusquinn:pr-14743-prompt-caching-update

What changed during the rebase:

• Adapted the cache audit sidebar to the current TUI sidebar plugin structure.
• Moved the new cache flags into packages/core/src/flag/flag.ts after the flag package refactor.
• Reapplied the stable/dynamic system prompt split to the current Effect-based session services.
• Preserved the splitSystemPrompt: false provider escape hatch for llama.cpp/Qwen-style providers that require one leading system message.
• Adapted cache marker/1h TTL handling to the current provider transform code.
• Dropped obsolete bash.ts/bash.test.ts conflicts in favour of the current ShellTool path.

Verification:

• bun test --timeout 30000 test/provider/transform.test.ts test/session/instruction.test.ts test/session/system.test.ts — 233 pass, 1 todo, 0 fail.
• bun run typecheck — passed.
• bun run lint — 0 errors; existing repository warnings only.
• bun run build was attempted but stopped before compilation because the local environment has Bun 1.3.11 while this repo requires bun@^1.3.13.

I also checked the latest discussion before posting. The recent comments are mostly about whether this PR is still relevant while packages/llm evolves; this rebase keeps the current packages/opencode code path mergeable/tested so maintainers can either merge it directly or cherry-pick the cleaned-up pieces.

---

aidevops.sh v3.15.29 plugin for OpenCode v1.14.48 with gpt-5.5 spent 9h 15m and 656,830 tokens on this with the user in an interactive session.

[marcusquinn]

Created a linked PR for the rebased/conflict-resolved branch:

#26875

It references this PR as For #14743 in the title/body so it should appear in the linked timeline and be easier for maintainers to review or cherry-pick.

---

aidevops.sh v3.15.29 plugin for OpenCode v1.14.48 with gpt-5.5 spent 9h 23m and 678,875 tokens on this with the user in an interactive session.

[spacepluk]

Is there any reason not to apply the 1h TTL to all the cache breakpoints? Right now it's only for the system portion if I'm not mistaken.
@spacepluk
Src: https://platform.claude.com/docs/en/build-with-claude/prompt-caching#pricing

I personally am not willing to pay for `2x` cache write price for the messages portion of my sessions. In my setup, I have modded my OpenCode to have this status line with the cache expiry timestamp from Anthropic's server-side response and a daemon scanning all my tmux panes to find all OpenCode sessions with this timestamp and keeping my cache warm by sending `ping` messages. I pay `0.1x` price for entire context window for cache reads per ping but since leaving open sessions unattended for more than 5m is rare in my workflow, the pings rarely fire and when they do it is still cheap. Honestly, if you just have normal sessions where you are pretty much actively getting work done, paying `1.25x` for 5m works out way better than paying `2x` for the 1h marker. Yes, 1h TTL means fewer cache misses on the read side over a session's lifetime, but without a daemon keeping the cache warm you can have a session cache go stale for 1h even and then that miss costs you the full `2x` cache write anyway.

That said, making it configurable for the messages markers between 5m and 1h is trivial if you need.

Anthropic has made Claude Code use the 1h markers for all of their markers including the messages portion (at least the main chat) which is overkill IMO. Everyone is paying the 2x price unnecessarily without realizing it. I wouldn't want it for my active interactive sessions at least since I see no value in it.

That makes sense but I think it's heavily dependent on your usage patterns. I haven't tried automating pings like that but not all of my sessions are running in tmux so I would need a different solution for that.

I usually have 5-10 long-running sessions that will last the whole day (or more) and context-switching/testing consistently takes longer than 5m. In that case the 1h cache is completely worth it in my opinion. I'll still use the 5m cache for short/active sessions though. I've been tracking my cost for a few weeks and I've greatly reduced costs thanks to your patches + 1h cache for everything with an additional patch in my fork.

I would suggest making OPENCODE_EXPERIMENTAL_CACHE_1H_TTL apply to all messages (like claude-code's ENABLE_PROMPT_CACHING_1H) and maybe have a separate SYSONLY option.

[martinffx]

I've tried to break it up into 4 seperate PR's to make it easier to review and merge

1. feat(cache): add cache token audit logging behind OPENCODE_EXPERIMENTAL_CACHE_AUDIT
   
2. feat(cache): split system prompt into stable/dynamic blocks for independent caching
3. fix(cache): stabilize system prefix behind OPENCODE_EXPERIMENTAL_CACHE_STABILIZATION
4. TODO: feat(cache): add optional 1h TTL on first system cache marker

[spacepluk]

In case it helps, here are my patches to use 1h for all the cache breakpoints and another one that implements my SYSONLY suggestion above

[spacepluk]

FYI I had to disable the system split for Gemini because it was ignoring the system prompt completely.

{
"provider": {
"google": {
"options": {
"splitSystemPrompt": false
}
}
}
}

[SamirSaidani]

@martinffx all your PRs have been closed by github bots. Aren't we both losing money and wasting energy by not taking proper care of prompt caching?

[Arrowyi]

@rekram1-node@thdxr — production evidence that the cache problem is worse than this PR assumes

I filed #40790 with production data from 5 Claude Opus 5 Max sessions: $24.45 out of $38.35 (63.8%) was cache writes. The worst session (ses_03563) had 9 cache busts across 43 messages, including 5 consecutive busts (messages #34–38) each costing $1.63.

Why this matters for #14743:

This PR's premise is "Same-session caching already works." My data contradicts that. All 9 busts happened within the same session, triggered by the known tool-call prompt loop bug (#24841): the loop reloads messages from DB, and tool part states change from pending → completed, producing different serialized bytes → Anthropic cache bust.

What #14743 would fix and what it wouldn't:

• ✅ Would fix: cross-repo system/tool prefix instability → saves ~12K tokens of tool definitions per bust
• ❌ Would NOT fix: the within-session conversation busts I'm seeing (prompt-loop DB reload changes message bytes)

In my case, even with this PR merged, I'd still see ~$0.50 busts (conversation portion) instead of $1.63 (full bust). Better, but not solved.

Two complementary fixes are needed:

1. This PR (fix(cache): improve Anthropic prompt cache hit rate with system split and tool stability #14743): stabilizes the system/tool prefix across repos and sessions
2. A follow-up: cache messages across the prompt loop to prevent DB-reload-induced byte drift (Production evidence: Anthropic cache bust bug causes $14.82 waste across 3 sessions (22.5% bust rate) #40790 root cause)

I've verified this with database queries anyone can run:

-- Verify busts happen exclusively after tool-calls:
WITH msg_list AS (
SELECT json_extract(data, '$.finish') as finish,
json_extract(data, '$.tokens.cache.read') as cache_r,
json_extract(data, '$.tokens.cache.write') as cache_w,
ROW_NUMBER() OVER (ORDER BY json_extract(data, '$.time.created')) as rn
FROM message WHERE session_id ='<your-session-id>'AND json_extract(data, '$.role') ='assistant'
)
SELECTm1.rn, m1.cache_r, m1.cache_w, m2.finishas prev_finish
FROM msg_list m1 LEFT JOIN msg_list m2 ONm1.rn=m2.rn+1WHEREm1.cache_r=0ANDm1.cache_w>0;

Bottom line: This PR has been sitting for 3 months with 14+ participants and multiple rebases. marcusquinn's rebased branch (pr-14743-prompt-caching-update) and martinffx's split PRs (#27007, #27377, #27378) have all been auto-closed without review. The cache problem is costing real money — merging this would be a meaningful first step even if it doesn't solve everything.

Happy to provide more data or help move this forward however I can.

[TomLucidor]

@Arrowyi could you replicate this with other model providers? Wondering if Kimi/GLM/DeepSeek/MiMo would have similar problems, or if it is server-side