Uh oh!
There was an error while loading. Please reload this page.
fix(server): settle orphaned provider sessions at startup - #7315
fix(server): settle orphaned provider sessions at startup#7315yassiEmp wants to merge 2 commits into
Conversation
A restart during an active turn left the session status running forever. No provider process existed, no session.exited event could arrive, and nothing reconciled the read model at boot. The thread showed Working permanently and queued user messages never processed (pingdotgg#4584). Add SessionStartupReconciler: after the reactors start and before the command gate opens, settle every session the read model considers live (running, starting, or holding an activeTurnId) that has no live provider process, as an error with an actionable lastError. Runs once, never fails startup, mirrors the ProviderSessionReaper layer shape.
Replace new Date() with DateTime per the Effect lint, and clean the test stubs: dispatch returns a sequence receipt, failures use defects, and command assertions narrow the union before reading threadId.
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Reviewed the new SessionStartupReconciler service against the Effect service conventions. Three convention violations in the newly added service module pair; the rest of the change (dependency acquisition via yield*, namespace imports, Layer.succeed only in test seams) looks consistent.
Posted via Macroscope — Effect Service Conventions
| }); | ||
| }); | ||
| const reconcile: SessionStartupReconcilerShape["reconcile"] = () => |
There was a problem hiding this comment.
Prefer referring to the inferred service interface rather than a standalone shape type (same for the satisfies SessionStartupReconcilerShape on line 111, and the type SessionStartupReconcilerShape import).
| constreconcile: SessionStartupReconcilerShape["reconcile"]=()=> | |
| constreconcile: SessionStartupReconciler["Service"]["reconcile"]=()=> |
Posted via Macroscope — Effect Service Conventions
| export const SessionStartupReconcilerLive = Layer.effect( | ||
| SessionStartupReconciler, | ||
| makeSessionStartupReconciler, | ||
| ); |
There was a problem hiding this comment.
The canonical layer should be exported as layer, with a real exported make beside it (as in provider/Layers/ProviderEventLoggers.ts, workspace/WorkspacePaths.ts), instead of a …Live alias over an unexported constructor. Consumers then use the module namespace (SessionStartupReconciler.layer).
| exportconstSessionStartupReconcilerLive=Layer.effect( | |
| SessionStartupReconciler, | |
| makeSessionStartupReconciler, | |
| ); | |
| exportconstlayer=Layer.effect(SessionStartupReconciler,make); |
(also rename makeSessionStartupReconciler to export const make at line 26)
Posted via Macroscope — Effect Service Conventions
| export interface SessionStartupReconcilerShape { | ||
| /** | ||
| * Settle sessions the read model considers live but whose provider process | ||
| * did not survive the last shutdown. Runs once during startup, after the | ||
| * projection pipeline is bootstrapped and before commands are accepted. | ||
| */ | ||
| readonly reconcile: () => Effect.Effect<void>; | ||
| } | ||
| export class SessionStartupReconciler extends Context.Service< | ||
| SessionStartupReconciler, | ||
| SessionStartupReconcilerShape | ||
| >()("t3/provider/Services/SessionStartupReconciler") {} |
There was a problem hiding this comment.
New service keeps a standalone SessionStartupReconcilerShape and splits the tag from its construction/layer across Services/ and Layers/. Consider defining the interface inline in the Context.Service declaration, and colocating the tag, make, and layer in one canonical module (apps/server/src/provider/SessionStartupReconciler.ts), deleting both split files and updating server.ts / serverRuntimeStartup.ts accordingly.
| exportinterfaceSessionStartupReconcilerShape{ | |
| /** | |
| *Settlesessionsthereadmodelconsiderslivebutwhoseproviderprocess | |
| *didnotsurvivethelastshutdown.Runsonceduringstartup,afterthe | |
| *projectionpipeline is bootstrappedandbeforecommandsareaccepted. | |
| */ | |
| readonly reconcile: ()=>Effect.Effect<void>; | |
| } | |
| exportclassSessionStartupReconcilerextendsContext.Service< | |
| SessionStartupReconciler, | |
| SessionStartupReconcilerShape | |
| >()("t3/provider/Services/SessionStartupReconciler"){} | |
| exportclassSessionStartupReconcilerextendsContext.Service< | |
| SessionStartupReconciler, | |
| { | |
| /** | |
| *Settlesessionsthereadmodelconsiderslivebutwhoseproviderprocess | |
| *didnotsurvivethelastshutdown.Runsonceduringstartup,afterthe | |
| *projectionpipeline is bootstrappedandbeforecommandsareaccepted. | |
| */ | |
| readonly reconcile: ()=>Effect.Effect<void>; | |
| } | |
| >()("t3/provider/SessionStartupReconciler"){} |
Posted via Macroscope — Effect Service Conventions
ApprovabilityVerdict: Needs human review This PR introduces a new SessionStartupReconciler service that modifies session state at startup by transitioning orphaned sessions to error status. This is a new capability with runtime behavior changes at startup, not a simple bug fix, and warrants human review. You can customize Macroscope's approvability policy. Learn more. |
closing as already addressed. The startup session reconciler landed in #7719. |
Problem
A server restart during an active turn leaves the thread stuck on Working forever. The session row keeps
status: runningand itsactiveTurnId, but no provider process exists after the restart, so nosession.exitedevent can ever arrive and nothing reconciles the read model at boot. Every message the user sends afterwards queues behind the zombie turn and is never processed. This is #4584, and it compounds the flood conditions described in #5681 (evidence from my Windows setup is on both issues).Fix
Add a
SessionStartupReconcilerthat runs once during startup, afterreactors.startand before the command gate opens. It reads the shell snapshot (archived threads included), finds sessions the read model considers live (running,starting, or holding anactiveTurnId) that have no live provider process inProviderService.listSessions(), and settles each one by dispatching the samethread.session.setshape the ingestion path uses forsession.exited-status: "error",activeTurnId: null, and an actionablelastError("Provider session did not survive a server restart. Send a new message to continue.").Design points:
ProviderSessionReaperservice/layer shape; wired intoProviderRuntimeLayerLiveand thereactors.startstartup phase.commandGate.signalCommandReady, so queued user commands never race a zombie turn.thread.session.setwitherrordoes not unsettle a settled thread (perdecider.settled.test.ts).Verification
SessionStartupReconciler.test.ts(orphan settled as error, starting + archived covered, settled/live/no-session threads skipped, dispatch failure does not break the sweep).tsgo --noEmitclean on apps/server.Fixes#4584.
Model: Claude Fable 5 · Harness: Claude Code
Note
Medium Risk
Touches orchestration session state at a critical startup boundary; incorrect orphan detection could mark valid sessions as errors, though live sessions are explicitly excluded via listSessions.
Overview
Adds
SessionStartupReconciler, which runs once at boot (after reactors start, before the command gate opens) so threads no longer stay stuck “Working” when a restart leftrunning/startingsessions and zombieactiveTurnIdvalues with no provider process.It compares shell + archived snapshots against
ProviderService.listSessions(), then dispatchesthread.session.setfor each orphan (status: "error", cleared turn, restartlastError). Failures are logged only so startup still completes. Wired viaProviderRuntimeLayerLiveandreactors.start; four unit tests cover settle, skip, and resilient dispatch.Reviewed by Cursor Bugbot for commit 5dd74be. Bugbot is set up for automated code reviews on this repo. Configure here.
Note
Settle orphaned provider sessions at server startup
SessionStartupReconciler, a new service that runs once during startup to detect and settle provider sessions that did not survive the last shutdown.thread.session.setcommand to transition orphaned sessions (statusrunning/startingor with a non-nullactiveTurnId) toerrorwith the message: "Provider session did not survive a server restart. Send a new message to continue."serverRuntimeStartup.tsafter the orchestration reactor andProviderSessionReaperstart, before the command gate opens.Macroscope summarized 5dd74be.