feat(runtime): reactive context-overflow compact-and-retry recovery by Astro-Han · Pull Request #1017 · apache/maka

Astro-Han · 2026-07-15T04:20:38Z

Summary

Refs #882 (PR 2 of 3, per the split in #882 (comment)).

When a provider rejects a request for exceeding the context window, the AI SDK surfaces it as a fullStream{type:'error'} chunk while the finishReason promise rejects — and our .catch(() => 'stop') swallowed that rejection into a fabricated end_turn with success telemetry. This PR makes the send pump treat a stream error as a first-class outcome and adds the reactive second line of defense behind PR 1's proactive compaction: classify the error, and if it is a genuine input overflow, fold the durable turn ledger once and retry the request.

Design points:

Recovery reuses the proactive machinery (computeMidTurnCompactionReplacement): same plan, same replay-admissibility gate, same anchor decoration, same checkpoint-before-projection persist order — diagnostics carry reason: 'overflow'. One retry per send (overflowRetryUsed latch); an unrecoverable error terminates with the real provider error, never a synthesized capacity outcome. No ledger seam (child sessions) → no recovery.
A retry breaks the "one send = one streamText" assumption, so one translation point in send() owns the attempt→send conversion: it rebases the SDK's attempt-local prepareStep.stepNumber onto the send-global step clock and presents a send-global steps view (completed steps archived from dead attempts). Every per-step consumer — the capacity hook's durability wait, same-turn load_tools activations, the active tool-result prune's eligible IDs — stays untouched and is send-correct by construction. Cross-attempt usage is owned by the per-step accumulator (an unusable sample in any attempt fails the whole record closed, per fix(headless): harden real-provider smoke reliability #972), the step limit is a send-level cap (a retry gets only the remaining budget), and the shrink baseline is the verdict owner's measure of the request the provider actually rejected.
The overflow classifier ranks evidence by strength over the real input domain. A normalizer accepts what the call sites actually produce — APICallError (with data/responseBody), plain stream-error objects (OpenAI Chat/Responses, Anthropic), and bare strings (openai-compatible) — then: explicit numeric statuses; structured provider codes (context_length_exceeded et al., the only unconditional signal); bare 413; veto-first free text (throttling/quota and complete output-cap relations veto — history compaction cannot fix an output cap); generic 5xx; weak word heuristics last. Pattern table ported from pi's battle-tested set and validated against the locked SDK sources.

Verification

npm --workspace @maka/runtime test: 1921 tests, 0 fail (7 pre-existing skips). The reactive suite (14 tests) locks the fabricated-end_turn repro (red before the fix), successful compact-and-retry (proactive-shaped and both real in-stream error shapes — Chat and Responses, per locked @ai-sdk/openai transform behavior), the single-retry latch, cross-attempt usage/step-budget/baseline ownership, post-retry durability (slow-consumer race), load_tools activation across retries, prune placeholders staying pruned in the retry request, and the unrecoverable path ending with the real provider error. Classifier tests cover every provider signature plus the structured-code, 413, output-cap, throttling, and word-boundary counterexamples — each was a demonstrated misclassification first.
CLI (359) and desktop (2538) suites, repo-wide npm run typecheck and npm run build: clean (the classifier's evidence reordering touches shared error-class mapping).
External review: 10 codex review rounds against the full diff. Rounds 1–3 converged the retry loop's send-level owners (three findings shared one root: attempt-local state treated as send state — fixed at the single translation point, and the round-2 bespoke availability set was deleted when the translation point subsumed it). Rounds 4–9 adversarially drove the classifier from word-order patches to the evidence-strength design, including the real APICallError.data/stream-error input domain. Round 10: PASS, no open findings.
Not run: real-provider overflow E2E (needs a live small-window model; the in-stream shapes are locked against the SDK's own transform sources instead).

Review focus

The invariant that carries the retry loop: all send-level state derives from the send-global clock and steps view produced by the one translation point in send() — never from the SDK's attempt-scoped stepNumber/steps. If a future hook consumes steps, it composes inside the pipeline and is correct by construction; do not add per-hook accumulators.

The invariant that carries the classifier: recovery triggers only on positive evidence of an input overflow — structured codes are the only unconditional signal; all free text is vetoable; output-cap wording never triggers a persisted compaction.

Add an exclusion-first ContextLength bucket to classifyError, matching the raw provider error message against a ported overflow-pattern table (the providers Maka ships) while excluding throttling/quota wording that merely mentions tokens. This is the reactive-recovery trigger for issue #882 PR 2; the status-based classes still win, so an explicit 429/5xx never lands here.

Second line of defense for issue #882 (PR 2). A request-level provider failure surfaces as a fullStream error chunk — both when the transport throws (finishReason then rejects with NoOutputGeneratedError) and when it streams an error part. The old pump caught the rejected finishReason as `stop` and emitted a fabricated end_turn completion with success telemetry. The pump now captures that error chunk and, at most once per send, folds the durable turn ledger and resends on a context-length overflow — a pi-style single compact-and-retry latch reusing the PR 1 mid-turn compaction machinery (planMidTurnCapacityCompaction, the checkpoint protocol, the anchor-tail decoration, and the replay-admissibility gate), tagged with reason 'overflow'. When recovery is impossible or spent — a non-context-length error, no mid-turn seam, no safe completed span, or a second overflow — the real provider error becomes the terminal outcome, never a fabricated success and never a synthesized context_budget_exhausted (the provider, not the runtime, rejected). The compaction core is extracted into computeMidTurnCompactionReplacement, shared by the proactive prepareStep hook and the reactive path so there is one fold implementation, one acceptance standard, and one persist order. errorReasonFromClass maps ContextLength to a context_overflow error reason.

…fallback patterns Review findings P2-1/P2-2: match the overflow signatures against the composite original-error text (name + code + status + message) so a structured code like OpenAI's context_length_exceeded classifies even with a generic HTTP message, and constrain the two over-broad fallbacks — the Copilot form now requires a token-count subject and the generic 'too many tokens' an input/prompt/context subject — so a file-size limit or a max_tokens parameter error never triggers a persisted compaction retry.

… baseline, usage, and step budget Review round-1 P1 findings, all owner-boundary errors in the retry loop: - P1-1: the shrink baseline for a reactive fold is now the verdict owner's per-request payload measure (state.lastRequestPayloadChars) — the request the provider actually rejected — instead of the attempt-initial messages, which undercount by every same-turn tool step and refused folds that genuinely shrank the real request (replacement_not_smaller, 0 retry). - P1-2: send-level usage is owned by the cross-attempt per-step accumulator. After a retry the terminal record carries BOTH attempts' completed steps; the last attempt's totalUsage is authoritative only for a single-attempt send, and an unusable sample in any attempt fails the whole record closed (#972) — a later attempt's valid totalUsage cannot wash it back. - P1-3: the step limit is a send-level cap counted by runtimeSteps across attempts. startStream takes a per-call maxSteps override and a retry gets only the remaining budget; with the budget spent the overflow is terminal. The extraction contract locks the adapter-owned default + per-call override.

…allback Review round-2 P2-C: the bare /token limit exceeded/ fallback also matched OUTPUT caps ('Output token limit exceeded', 'Maximum output token limit exceeded'), which history compaction cannot fix, so they triggered a pointless persisted compaction retry. The fallback now carries the same input/prompt/context subject constraint as the round-1 'too many tokens' fix; the input-side form keeps classifying.

…rflow retries Review round-2 P1-A + P1-B: a reactive retry re-invokes streamText, which resets two attempt-scoped views that send-level state was derived from. - P1-A: the SDK numbers prepareStep steps per streamText call, while flushedSteps / replacedStepNumber / lastShapeFailure / the semantic-compact yield all keep send-level state. An attempt-local durability bound already satisfied by a PREVIOUS attempt's flushed boundary let a post-retry capacity compaction read the ledger before the retry step's streamed assistant text was durable — and the replacement projection then dropped it from both the covered span and the tail. One translation point in send() now rebases each attempt's local step numbers onto the send-global clock (completed steps at attempt start) before any hook sees them. - P1-B: active tools were re-derived per streamText call from the ledger seed plus that call's own steps, so a retry (fresh call, empty steps) silently revoked a group loaded before the overflow — the gated tool vanished from the provider request and the execute boundary rejected it. The availability owner now holds a send-scoped monotonic activation set: groups accumulate from every attempt's steps and never unload within a send. Cross-turn behavior is unchanged (rebuilt per send from the ledger seed). Both repros ride the new fixture levers: a slow appendMessage that parks the pump inside flushStep while the consumer has drained the queue (P1-A), and a gated tool group loaded before the overflow (P1-B).

…ross overflow retries A reactive overflow retry starts a fresh streamText call, so the SDK's per-call `steps` restarted empty mid-send. The active tool-result prune derives its eligible tool-call IDs from `steps`; an empty view revoked the prune on the retry request and the ledger-rebuilt recovery projection resurrected archived raw tool results (review round-3 P1) — the third instance of attempt-local SDK state consumed as send-level state, after the step clock (round-2 P1-A) and tool activations (round-2 P1-B). Converge all of them into the existing single translation point: sendScopedPrepareStep archives each dead attempt's observed steps and hands every hook `[...completedAttemptSteps, ...options.steps]` alongside the already-rebased send-global stepNumber. Consumers stay untouched and any future steps consumer is send-correct by construction; steps folded into a checkpoint remain in the view because ID-based consumers only act on messages present in the projection. This also lets tool-availability drop its round-2 bespoke monotonic activation set and return to deriving activations statelessly from the (now send-global) steps; its round-2 regression test stays green. Two pairing consequences of the union view, each at its single owner: the capacity hook only anchors its next-request estimate on the last step's usage when the verdict owner has a payload baseline for the same request, and a successful overflow recovery resets that baseline so the retry starts from the whole-payload cold-start estimate instead of a stale pairing against the rejected request.

…erflow pattern Review round-4 P1: /token count of N exceeds the limit of M/ also matches output and completion caps ('output token count of 8192 exceeds the limit of 4096'), which history compaction cannot fix — the misclassification triggered a persisted compaction and a doomed retry. The pattern now requires the same input subject as the sibling generic fallbacks; the real Copilot form ('prompt token count of X exceeds the limit of Y') still classifies, with output/completion negatives locked.

…'s exclusion owner Review round-5 P1: the input-subject constraints can be bypassed by a generic prefix — 'Invalid request: output token count of 8192 exceeds the limit of 4096' classified as ContextLength because 'request' satisfied the subject alternation without modifying the token count. The invariant is categorical, not positional: history compaction can only fix INPUT overflow, so explicit output/completion/max_tokens cap wording is now excluded at the exclusion-first owner regardless of surrounding words. Exclusions stay adjacency-tight; OpenAI's classic input-overflow message (mentioning 'the completion' and 'max_tokens' amounts) keeps classifying, locked by new positives alongside the prefixed negatives.

… vetoes, then fuzzy subjects Review round-6 P1s: blocklisting output-cap word orders can never converge ('completion has too many tokens', 'max_tokens token limit exceeded' bypassed the previous exclusions), and the unconditional noun-phrase exclusion vetoed a genuine input overflow whose message breaks usage down into prompt AND completion token counts alongside a context_length_exceeded code. The classifier is now tiered by evidence strength instead of patched by wording: (1) definitive provider signals — including structured codes — win unconditionally; (2) throttling/quota wording and complete output-cap RELATIONS (subject and predicate, not noun phrases) veto; (3) ambiguous token-limit wording counts only with an input-like subject, and 'request' is dropped from that subject list because a generic 'Invalid request:' prefix carries no input semantics. Both round-6 bypasses and the over-exclusion are locked as tests.

…ern patches Three rounds of word-table patches to the overflow classifier kept being pierced (review round 7, four P1s) because the design let weak evidence outrank strong evidence: - A real AI SDK APICallError carries the provider's structured error JSON in `data` (or raw in `responseBody`); there is no top-level `.code`, so `data.error.code = 'context_length_exceeded'` with a generic 'Bad Request' message never classified. - `text.includes('rate')` ran before overflow detection at equal strength with an explicit 429, so 'Failed to generate response: context_length_exceeded' became RateLimit ('generate' contains 'rate'). - The output-cap veto only knew the subject-before-predicate voice, so the passive 'too many tokens were requested for the completion' slipped through as ContextLength. - The free-text 'definitive' tier was unconditional, so a bare capacity statement ('maximum context length is N tokens') quoted inside a ThrottlingException overrode the throttle/quota veto. Redesign classifyError at its owner by descending evidence strength: abort, then explicit numeric statuses/codes from fields, then the structured provider code (new extraction walking data.error.code/.type and the same paths in responseBody JSON — the exact shapes createJsonErrorResponseHandler produces), then free-text overflow relations, and only last the rate/auth/timeout/network substring heuristics. Free text collapses from three tiers to two: vetoes first (throttle/quota + output-cap relations in both voices plus the count-of form), then all positive overflow relations, none unconditional — only a structured provider code is. Structured-code positives in tests now use the real APICallError shape instead of the invented top-level code.

…king evidence The classifier's callers hand it whatever the AI SDK surfaced, and that input domain is not just Error instances (review round 8, 4 P1s): - In-stream error parts carry the provider's PARSED error value: OpenAI Chat emits the inner {message, type?, code?} object, OpenAI Responses the whole {type:'error', error:{type, code, message}} chunk, Anthropic the inner {type, message} object, and openai-compatible a bare message string. The instanceof Error gate classified all of them Other, so a genuine in-stream overflow could never reach reactive recovery. - Generic 5xx ranked above specific overflow evidence, so LiteLLM-style 503 wrappers around a provider overflow became ProviderUnavailable. - A bare 413 with no body (Cerebras) carried no text signal at all, yet HTTP 413 is itself input-side evidence. - The output-cap veto missed the embedded-role permutation ('Too many completion tokens were requested…'), letting a trailing capacity statement classify as overflow. Introduce a single evidence normalizer at the classifier owner: it maps Error | string | plain object into {composite text, statusCode, code, structuredCodes} using the exact shapes the providers produce (data / responseBody for APICallError, code/type on the value or its error wrapper for stream parts). classifyError then ranks normalized evidence strictly by strength: abort > 402 > 429 > 401/403 > structured overflow code > bare 413 > vetoable free-text overflow > generic 5xx > weak word heuristics. The weak rate heuristic becomes word-shaped so 'generate' / 'separate' are no longer rate limits (P2), the output-cap veto gains the embedded-role and role-tokens-exceed permutations (P1-4), and the responseBody test fixture now uses a body the OpenAI errorSchema genuinely rejects, proving the fallback (P3). An end-to-end reactive test drives recovery from a plain-object in-stream error part with the production finish-after-error stream shape.

…t the in-stream fixture shapes Two review round-9 findings: - P2: when the provider error JSON fails the schema, the real createJsonErrorResponseHandler degrades message to the statusText and keeps the provider wording ONLY in responseBody. The normalizer read the body just for structured codes, so an OpenAI-compatible {error: string} overflow ('Your input exceeds the context window…') classified as AI_APICallError. The raw body now joins the composite text so positives AND vetoes run over the full evidence; the test constructs the error through the real handler and also locks an output-cap body against misclassification. - P3: the round-8 e2e fixture mixed provider families (Responses-shaped error value with a Chat-shaped 'error' finish trailer), a stream that no locked provider produces. The Chat shape (inner error object + finishReason 'error' trailer) is now the main test and a Responses variant (whole error chunk + finishReason 'other' trailer, which the isErrorChunk branch never reassigns) locks recovery against per-family trailer drift.

…-compact-retry

Astro-Han · 2026-07-15T04:20:38Z

Summary

Refs #882 (PR 2 of 3, per the split in #882 (comment)).

When a provider rejects a request for exceeding the context window, the AI SDK surfaces it as a fullStream{type:'error'} chunk while the finishReason promise rejects — and our .catch(() => 'stop') swallowed that rejection into a fabricated end_turn with success telemetry. This PR makes the send pump treat a stream error as a first-class outcome and adds the reactive second line of defense behind PR 1's proactive compaction: classify the error, and if it is a genuine input overflow, fold the durable turn ledger once and retry the request.

Design points:

Recovery reuses the proactive machinery (computeMidTurnCompactionReplacement): same plan, same replay-admissibility gate, same anchor decoration, same checkpoint-before-projection persist order — diagnostics carry reason: 'overflow'. One retry per send (overflowRetryUsed latch); an unrecoverable error terminates with the real provider error, never a synthesized capacity outcome. No ledger seam (child sessions) → no recovery.
A retry breaks the "one send = one streamText" assumption, so one translation point in send() owns the attempt→send conversion: it rebases the SDK's attempt-local prepareStep.stepNumber onto the send-global step clock and presents a send-global steps view (completed steps archived from dead attempts). Every per-step consumer — the capacity hook's durability wait, same-turn load_tools activations, the active tool-result prune's eligible IDs — stays untouched and is send-correct by construction. Cross-attempt usage is owned by the per-step accumulator (an unusable sample in any attempt fails the whole record closed, per fix(headless): harden real-provider smoke reliability #972), the step limit is a send-level cap (a retry gets only the remaining budget), and the shrink baseline is the verdict owner's measure of the request the provider actually rejected.
The overflow classifier ranks evidence by strength over the real input domain. A normalizer accepts what the call sites actually produce — APICallError (with data/responseBody), plain stream-error objects (OpenAI Chat/Responses, Anthropic), and bare strings (openai-compatible) — then: explicit numeric statuses; structured provider codes (context_length_exceeded et al., the only unconditional signal); bare 413; veto-first free text (throttling/quota and complete output-cap relations veto — history compaction cannot fix an output cap); generic 5xx; weak word heuristics last. Pattern table ported from pi's battle-tested set and validated against the locked SDK sources.

Verification

npm --workspace @maka/runtime test: 1921 tests, 0 fail (7 pre-existing skips). The reactive suite (14 tests) locks the fabricated-end_turn repro (red before the fix), successful compact-and-retry (proactive-shaped and both real in-stream error shapes — Chat and Responses, per locked @ai-sdk/openai transform behavior), the single-retry latch, cross-attempt usage/step-budget/baseline ownership, post-retry durability (slow-consumer race), load_tools activation across retries, prune placeholders staying pruned in the retry request, and the unrecoverable path ending with the real provider error. Classifier tests cover every provider signature plus the structured-code, 413, output-cap, throttling, and word-boundary counterexamples — each was a demonstrated misclassification first.
CLI (359) and desktop (2538) suites, repo-wide npm run typecheck and npm run build: clean (the classifier's evidence reordering touches shared error-class mapping).
External review: 10 codex review rounds against the full diff. Rounds 1–3 converged the retry loop's send-level owners (three findings shared one root: attempt-local state treated as send state — fixed at the single translation point, and the round-2 bespoke availability set was deleted when the translation point subsumed it). Rounds 4–9 adversarially drove the classifier from word-order patches to the evidence-strength design, including the real APICallError.data/stream-error input domain. Round 10: PASS, no open findings.
Not run: real-provider overflow E2E (needs a live small-window model; the in-stream shapes are locked against the SDK's own transform sources instead).