You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Assistant runs can silently finalize as 'completed' without a terminal RuntimeEvent, permanently breaking the session (stuck 'streaming' UI + RuntimeReadModelError) #397
When an assistant turn's underlying stream (AiSdkFlow / RuntimeRunner) exhausts without ever producing a complete/abortSessionEvent — no exception thrown, the async iterable just ends — AgentRun.finalize() falls back to marking the run 'completed' (a terminal AgentRunStore status) even though no terminal RuntimeEvent was ever written to the ledger.
This produces two directly-observable symptoms and one silent, permanent data-integrity problem:
UI stuck forever: the composer keeps showing "Maka 正在回答…" with the Stop button and a blinking cursor, even though the assistant's full response text is already rendered (all text_deltas arrived fine — only the final terminal signal is missing). There is no timeout/fallback in the renderer to recover from this.
Every subsequent read of the session throws: sessions:readMessages → SessionManager.getMessages → RuntimeReadModel.getSessionView throws RuntimeReadModelError: RuntimeEvent ledger has no terminal fact for a terminal run (packages/runtime/src/runtime-read-model.ts), because the run's status is terminal but the ledger has no matching terminal fact. This is intentional fail-fast behavior in the read model (not itself a bug), but nothing upstream prevents the inconsistent state from being created in the first place.
Not self-healing: the ledger-backfill path (runtime-read-model.ts) only triggers when the ledger is completely empty — here it's non-empty (content events wrote fine, just the terminal one is missing), so backfill never runs. The app-restart recovery path (classifyAgentRunRecovery in packages/runtime/src/agent-run-recovery.ts) explicitly skips runs whose AgentRunHeader.status is already terminal (isTerminalRunStatus early-return) — but finalize()'s fallback is exactly what put the run into that terminal status. So restarting the app does not fix it either; the session is permanently stuck reproducing the same error on every read.
Root cause chain
packages/runtime/src/runtime-runner.ts: when flow.run()'s AsyncIterable completes without ever yielding a terminal RuntimeEvent, RuntimeRunner.run() correctly classifies this as failure.class: 'missing_terminal_event' (there's already a unit test for exactly this: packages/runtime/src/__tests__/runtime-runner.test.ts, "a flow that exhausts without a terminal event maps to a failed result") — but it resolves the promise with status: 'failed' on the returned result object rather than rejecting. In packages/runtime/src/runtime-kernel.ts, that result only reaches runtimeInvocationObserver (a fire-and-forget observability callback); the .then()'s rejection branch (which calls sessionEvents.fail(error)) never fires because the promise didn't reject.
AiSdkFlow's onFinally (packages/runtime/src/ai-sdk-flow.ts) unconditionally calls run.finalize() and then sessionEvents.close() — a clean close, not a fail(), so the renderer's event stream ends with no error at all.
AgentRun.finalize() / finishRun() (packages/runtime/src/agent-run.ts): when this.finalStatus is undefined (never set, because no complete/abortSessionEvent was ever recorded), finishRun falls through to status: 'completed' — silently promoting an incomplete run to a terminal success state.
Separately, AgentRun.recordSessionEvent() (packages/runtime/src/agent-run.ts) calls hooks.updateStatus / hooks.appendTurnState with no try/catch, unlike the store-write paths a few lines below (enqueueRunStore/enqueueRuntimeEventStore), which deliberately catch and flip an Available flag instead of throwing. SessionStore.updateHeader (packages/storage/src/session-store.ts) does a full read-modify-write of the session's .jsonl file on every call, so a transient I/O failure here on the finalcomplete/abort event could be a second way into the same broken state (this variant does at least propagate an errorSessionEvent via main.ts's streamEvents catch-all, so it's less silent than the first path, but still leaves the ledger without a terminal fact).
Frontend: apps/desktop/src/renderer/app-shell.tsx's streamingBySession state machine only clears/settles on receiving a complete/abort/errorSessionEvent (handleEvent's case 'complete' etc., and settleAssistantStreaming). There's no independent timeout or reconciliation against the session's actual backend status, so if none of those three events ever arrive, the UI has no way to recover on its own.
Suggested fix directions (not attempting a PR myself — filing for the maintainers to prioritize/scope)
AgentRun.finalize()/finishRun() should not silently default to 'completed' when finalStatus was never observed. At minimum it should write a terminal RuntimeEvent (e.g. status: 'failed', reason missing_terminal_event) so the ledger and the run header stay consistent, and should push a corresponding terminal SessionEvent to the renderer instead of a clean sessionEvents.close().
Wire RuntimeRunner's already-computed missing_terminal_event result back into runtime-kernel.ts's main path instead of only the observability callback.
Add the same failure-isolation (.catch() + availability flag) around AgentRun.recordSessionEvent()'s calls to hooks.updateStatus/hooks.appendTurnState that enqueueRunStore/enqueueRuntimeEventStore already have.
RuntimeReadModel's backfill logic (runtime-read-model.ts) currently only engages when the ledger is completely empty; consider also handling "non-empty ledger, missing only the terminal fact."
classifyAgentRunRecovery (agent-run-recovery.ts) skips any run whose header status is already terminal. Consider detecting "status is terminal but the ledger has no matching terminal RuntimeEvent" as its own recoverable case in the startup recovery path (recoverAgentRunsFromLedger in session-manager.ts), so a restart can actually repair this instead of reproducing the same error forever.
Renderer: give streamingBySession (app-shell.tsx) an independent timeout-based fallback that clears the streaming indicator and surfaces a "connection interrupted, please retry" state if no terminal SessionEvent arrives within some bound, rather than hanging indefinitely.
Repro (approximate — I hit this organically, haven't isolated a minimal repro)
I don't have a reliable minimal repro yet; I hit this on an ordinary chat turn where the response streamed in fully and the composer never left the "streaming" state afterward. Every subsequent app launch reproduced the same RuntimeReadModelError on that session. Happy to share the affected session's JSONL files privately if useful for debugging — didn't want to paste session content into a public issue.
Environment
Branch: main @ 9d0e0a69 (at the time of investigation)
Summary
When an assistant turn's underlying stream (
AiSdkFlow/RuntimeRunner) exhausts without ever producing acomplete/abortSessionEvent— no exception thrown, the async iterable just ends —AgentRun.finalize()falls back to marking the run'completed'(a terminalAgentRunStorestatus) even though no terminalRuntimeEventwas ever written to the ledger.This produces two directly-observable symptoms and one silent, permanent data-integrity problem:
text_deltas arrived fine — only the final terminal signal is missing). There is no timeout/fallback in the renderer to recover from this.sessions:readMessages→SessionManager.getMessages→RuntimeReadModel.getSessionViewthrowsRuntimeReadModelError: RuntimeEvent ledger has no terminal fact for a terminal run(packages/runtime/src/runtime-read-model.ts), because the run'sstatusis terminal but the ledger has no matching terminal fact. This is intentional fail-fast behavior in the read model (not itself a bug), but nothing upstream prevents the inconsistent state from being created in the first place.runtime-read-model.ts) only triggers when the ledger is completely empty — here it's non-empty (content events wrote fine, just the terminal one is missing), so backfill never runs. The app-restart recovery path (classifyAgentRunRecoveryinpackages/runtime/src/agent-run-recovery.ts) explicitly skips runs whoseAgentRunHeader.statusis already terminal (isTerminalRunStatusearly-return) — butfinalize()'s fallback is exactly what put the run into that terminal status. So restarting the app does not fix it either; the session is permanently stuck reproducing the same error on every read.Root cause chain
packages/runtime/src/runtime-runner.ts: whenflow.run()'sAsyncIterablecompletes without ever yielding a terminalRuntimeEvent,RuntimeRunner.run()correctly classifies this asfailure.class: 'missing_terminal_event'(there's already a unit test for exactly this:packages/runtime/src/__tests__/runtime-runner.test.ts, "a flow that exhausts without a terminal event maps to a failed result") — but it resolves the promise withstatus: 'failed'on the returned result object rather than rejecting. Inpackages/runtime/src/runtime-kernel.ts, that result only reachesruntimeInvocationObserver(a fire-and-forget observability callback); the.then()'s rejection branch (which callssessionEvents.fail(error)) never fires because the promise didn't reject.AiSdkFlow'sonFinally(packages/runtime/src/ai-sdk-flow.ts) unconditionally callsrun.finalize()and thensessionEvents.close()— a clean close, not afail(), so the renderer's event stream ends with no error at all.AgentRun.finalize()/finishRun()(packages/runtime/src/agent-run.ts): whenthis.finalStatusisundefined(never set, because nocomplete/abortSessionEventwas ever recorded),finishRunfalls through tostatus: 'completed'— silently promoting an incomplete run to a terminal success state.AgentRun.recordSessionEvent()(packages/runtime/src/agent-run.ts) callshooks.updateStatus/hooks.appendTurnStatewith notry/catch, unlike the store-write paths a few lines below (enqueueRunStore/enqueueRuntimeEventStore), which deliberately catch and flip anAvailableflag instead of throwing.SessionStore.updateHeader(packages/storage/src/session-store.ts) does a full read-modify-write of the session's.jsonlfile on every call, so a transient I/O failure here on the finalcomplete/abortevent could be a second way into the same broken state (this variant does at least propagate anerrorSessionEventviamain.ts'sstreamEventscatch-all, so it's less silent than the first path, but still leaves the ledger without a terminal fact).apps/desktop/src/renderer/app-shell.tsx'sstreamingBySessionstate machine only clears/settles on receiving acomplete/abort/errorSessionEvent(handleEvent'scase 'complete'etc., andsettleAssistantStreaming). There's no independent timeout or reconciliation against the session's actual backend status, so if none of those three events ever arrive, the UI has no way to recover on its own.Suggested fix directions (not attempting a PR myself — filing for the maintainers to prioritize/scope)
AgentRun.finalize()/finishRun()should not silently default to'completed'whenfinalStatuswas never observed. At minimum it should write a terminalRuntimeEvent(e.g.status: 'failed', reasonmissing_terminal_event) so the ledger and the run header stay consistent, and should push a corresponding terminalSessionEventto the renderer instead of a cleansessionEvents.close().RuntimeRunner's already-computedmissing_terminal_eventresult back intoruntime-kernel.ts's main path instead of only the observability callback..catch()+ availability flag) aroundAgentRun.recordSessionEvent()'s calls tohooks.updateStatus/hooks.appendTurnStatethatenqueueRunStore/enqueueRuntimeEventStorealready have.RuntimeReadModel's backfill logic (runtime-read-model.ts) currently only engages when the ledger is completely empty; consider also handling "non-empty ledger, missing only the terminal fact."classifyAgentRunRecovery(agent-run-recovery.ts) skips any run whose header status is already terminal. Consider detecting "status is terminal but the ledger has no matching terminal RuntimeEvent" as its own recoverable case in the startup recovery path (recoverAgentRunsFromLedgerinsession-manager.ts), so a restart can actually repair this instead of reproducing the same error forever.streamingBySession(app-shell.tsx) an independent timeout-based fallback that clears the streaming indicator and surfaces a "connection interrupted, please retry" state if no terminalSessionEventarrives within some bound, rather than hanging indefinitely.Repro (approximate — I hit this organically, haven't isolated a minimal repro)
I don't have a reliable minimal repro yet; I hit this on an ordinary chat turn where the response streamed in fully and the composer never left the "streaming" state afterward. Every subsequent app launch reproduced the same
RuntimeReadModelErroron that session. Happy to share the affected session's JSONL files privately if useful for debugging — didn't want to paste session content into a public issue.Environment
main@9d0e0a69(at the time of investigation)npm run dev)