Skip to content
This repository was archived by the owner on May 15, 2026. It is now read-only.

refactor: extract shared prompt cache breakpoint layer from 4 providers - #11415

Closed
hannesrudolph wants to merge 1 commit into
mainfrom
refactor/shared-prompt-cache-layer
Closed

refactor: extract shared prompt cache breakpoint layer from 4 providers#11415
hannesrudolph wants to merge 1 commit into
mainfrom
refactor/shared-prompt-cache-layer

Conversation

@hannesrudolph

Copy link
Copy Markdown
Contributor

Summary

Moves cache breakpoint placement from individual providers to a shared utility called from Task.ts before createMessage(). Messages arrive at providers pre-annotated with providerOptions, and the AI SDK routes the correct options to the active provider automatically.

Architecture

Task.ts message preparation:
cleanConversationHistory
→ resolveCacheProviderOptions(provider, modelInfo, settings)
→ applyCacheBreakpoints(messages, options, config)
→ api.createMessage(systemPrompt, annotatedMessages, metadata)

The AI SDK's providerOptions namespacing ensures each provider only picks up its own cache hints — anthropic.cacheControl for Anthropic family, bedrock.cachePoint for Bedrock.

New Files

FilePurpose
src/api/transform/prompt-cache.tsresolveCacheProviderOptions() — maps provider name → cache config; applyCacheBreakpoints() — annotates last N non-assistant batches
src/api/transform/__tests__/prompt-cache.spec.ts14 test cases

Provider Changes

ProviderRemovedPreserved
anthropic.tstargeting block + applyCacheControlToAiSdkMessages()systemProviderOptions in streamText()
anthropic-vertex.tstargeting block + applyCacheControlToAiSdkMessages()systemProviderOptions in streamText()
minimax.tstargeting block + applyCacheControlToAiSdkMessages()systemProviderOptions in streamText()
bedrock.tstargeting block + applyCachePointsToAiSdkMessages()systemProviderOptions, usePromptCache, supportsAwsPromptCache()

Key Improvements

  1. Fixes tool message caching gap — Targets non-assistant batches (role: user + role: tool) instead of only role: user. After PR feat: implement ModelMessage storage layer with AI SDK response messages #11409, tool results are separate role: tool messages that now correctly receive cache breakpoints.

  2. Single source of truth — Cache strategy defined once in prompt-cache.ts with provider adapter mapping.

  3. Provider-specific config preserved — Bedrock: 3 breakpoints + anchor at ~1/3 for 20-block lookback coverage. Anthropic family: 2 breakpoints. Bedrock opt-in via awsUsePromptCache.

Not Changed

  • systemProviderOptions in all 4 providers
  • OpenAI Native promptCacheRetention (provider-level option, not per-message)
  • No settings types or UI changes (separate scope)

Stats

+426 / -216 lines • 14 new tests • 5,491 total tests pass • 0 regressions

Move cache breakpoint logic from individual providers to a shared utility
called from Task.ts before createMessage(). Messages arrive at providers
pre-annotated with providerOptions, and the AI SDK routes the correct
options to the active provider automatically.
New files:
- src/api/transform/prompt-cache.ts: resolveCacheProviderOptions() +
applyCacheBreakpoints() with provider adapter mapping
- src/api/transform/__tests__/prompt-cache.spec.ts: 14 test cases
Changes per provider:
- anthropic.ts: removed targeting block + applyCacheControlToAiSdkMessages()
- anthropic-vertex.ts: same
- minimax.ts: same
- bedrock.ts: removed targeting block + applyCachePointsToAiSdkMessages()
Key improvements:
- Targets non-assistant batches (user + tool) instead of only role=user.
After PR #11409, tool results are separate role=tool messages that now
correctly receive cache breakpoints.
- Single source of truth: cache strategy defined once in prompt-cache.ts
- Provider-specific config preserved: Bedrock gets 3 breakpoints + anchor,
Anthropic family gets 2 breakpoints
Preserved (untouched):
- systemProviderOptions in all providers' streamText() calls
- OpenAI Native promptCacheRetention (provider-level, not per-message)
- Bedrock usePromptCache opt-in + supportsAwsPromptCache()
5,491 tests pass, 0 regressions.
@dosubotdosubotBot added the size:XL This PR changes 500-999 lines, ignoring generated files. label Feb 11, 2026
@ghost

ghost commented Feb 11, 2026

Copy link
Copy Markdown

RooviewerClockSee task

Clean refactor overall. One issue flagged around Bedrock's cache checkpoint budget.

  • Bedrock maxMessageBreakpoints = 3 + anchor can produce 5 total checkpoints (including system), exceeding the documented 4-checkpoint limit

Mention @roomote in a comment to request specific changes to this pull request or fix all unresolved issues.

Comment threadsrc/core/task/Task.ts
applyCacheBreakpoints(
cleanConversationHistory,
cacheOptions,
isBedrock ? 3 : 2,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The old bedrock code documented a hard limit of 4 cache checkpoints and budgeted them as 1 system + 2 trailing + 1 anchor = 4. Here, maxMessageBreakpoints = 3 for Bedrock plus useAnchor = true yields up to 3 trailing + 1 anchor = 4 message-level breakpoints, plus the system prompt checkpoint applied in bedrock.ts via systemProviderOptions. That totals 5, which exceeds the documented 4-checkpoint limit and could cause Bedrock API errors once a conversation is long enough to trigger the anchor. Changing the trailing count to 2 (matching the old behavior) would keep the total at 4.

Suggested change
isBedrock ? 3 : 2,
isBedrock ? 2 : 2,

Fix it with Roo Code or mention @roomote and request a fix.

Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

size:XLThis PR changes 500-999 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@hannesrudolph