You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
An overnight self-paced /loop run died at 06:34:49Z and never came back. Auto-resume was enabled for the thread and its poller was alive the whole time, but the durable store shows the thread never armed a pending resume and never fired one, despite receiving three account.rate-limits.updated events with status:"rejected" — the exact signal AutoResumeReactor's detection fiber documents as its trigger.
The distinguishing feature of this episode versus #6 and #39: the terminal block was not a plain five_hour exhaustion. It was five_hour rejected plus overage rejected with overageDisabledReason:"org_level_disabled_until", surfacing to the model as You've hit your individual spend limit · run /usage-credits to ask your admin for a higher limit.
Net cost: ~6 hours of wall clock, and an overnight build that stopped at item 5 of 6 in its first wave. The user found out the next morning.
Environment
T3 Code desktop, Windows 11 Pro 10.0.26200. Build installed 2026-08-14 (resources/app.asar, resources/server.asar mtime 20:31).
A /loop in dynamic mode (self-paced via ScheduleWakeup) driving a multi-wave prototype build, with work fanned out through background Workflow subagents. Explicitly an unattended overnight run.
session.started — the human came back and resumed manually
line 1377
The core defect
Detection is documented as:
subscribes once to providerService.streamEvents and, on a Claude account.rate-limits.updated event with status:"rejected", schedules a resume at the structured resetsAt (+ margin)
That precondition was met three times. account.rate-limits.updated appears 25 times in this thread's log; the three carrying status:"rejected" are confirmed to be of that canonical type.
Walking planSchedule with the 06:23:55 event: verdict.rejected true, pendingResumeAtMs null, windowOpensInFuture true (10:40:00Z > 06:23:55Z), firedInCapWindow 0. That returns {kind:"schedule"}. Yet the store reads:
enabled:true, so the per-thread switch was on. firedAtMs is empty and FIRED_HISTORY_RETENTION_MS is 25h, so a fire anywhere in this episode would still be visible — it never fired. And pending is null, so either it never armed or it armed and was cancelled.
The wake fiber was demonstrably alive: GET /api/coil/auto-resume?threadId=57e2846c… polls run continuously from 06:07:06Z to 12:52:04Z with no gap over 5 minutes, straight across the 10:40:00Z reset. So this is not "the machine was asleep".
I could not narrow it further from the outside, because there is no telemetry for the auto-resume decision at all (see item 3).
Never armed, because the parser did not read this payload shape as rejected. The rate_limit_info here carries the newer overage fields (overageStatus, overageResetsAt, overageDisabledReason, isUsingOverage). If the verdict parser reads a composite/effective status rather than rate_limit_info.status, an overageStatus:"rejected" variant may not map to verdict.rejected.
Armed correctly, but at a useless time.resetsAt (10:40:00Z) describes the five_hour bucket. The actual blocker was the individual spend limit — org_level_disabled_until, with overageResetsAt on 2026-09-01. A resume at 10:40 would have fired into a still-closed door and burned a ladder attempt. Auto-resume currently has no representation for "blocked until a human raises a spend cap", which is not a time-based condition at all.
Hypothesis 3 holds regardless of which of 1 or 2 is true, and is the one worth designing for.
Three separable defects
1. A limit stop is recorded as a successful turn.result carries is_error:true and subtype:"success" simultaneously, and the item is status:"completed". Any consumer keyed on subtype sees success. The reactor docstring already acknowledges the upstream shape ("a usage limit does not produce a failed turn (the SDK has no rate-limit result subtype)") and deliberately reads the structured signal instead — but nothing else in the fork does, so the "did my overnight run actually stop?" question has no reliable answer outside the rate-limit stream.
2. A scheduled wake tick burns a turn into a closed limit, then the session exits silently. At 06:35:00 the loop's own ScheduleWakeup started a turn; init reported status:"failed", the model never ran, the command was cancelled, and 4m35s later the session exited gracefully. From the model's side the turn produced no output, so no follow-up ScheduleWakeup was armed and the loop was permanently dead. Two asks: (a) don't dispatch a scheduled wake while a rejection is in force and its window has not opened — defer it to the same resetsAt machinery; (b) a graceful exit while limited with a live loop should raise something the user can see the next morning. It produced no notification of any kind. This is closely related to #38 and #42.
3. Auto-resume decisions are invisible. Grepping every rotated server.trace.ndjson* for already-pending, not-rejected, capped, autoResume, auto-resume scheduled, and rate_limit_event returns zero hits — the only auto-resume evidence in the traces is the HTTP poll route. There is no record of a planSchedule verdict, a cancelReason, or a fire attempt. #6 and #39 were both "auto-resume silently didn't fire" bugs; a third one is now unresolvable from a full set of logs. Emitting the arm/skip/cancel decision (with reason and resumeAtMs) as a domain event or timeline activity would have made this issue a one-line diagnosis.
Suggested direction
Log every planSchedule and cancelReason outcome with its reason. Highest value per line of code here.
Assert which of hypotheses 1/2 applies, using the timeline above as the fixture.
Treat "overage rejected / spend limit / org_level_disabled_until" as a distinct terminal class from a windowed rate limit: do not schedule a time-based resume, mark the thread blocked-on-human, and notify. overageResetsAt two weeks out is not a retry target.
Consider whether a graceful session.exited should preserve rather than invalidate a pending resume.
Repro sketch
Account where five-hour usage is exhausted and org-level overage is disabled (spend cap reached).
Start a dynamic /loop that fans work out through background Workflow subagents, so turns are long and rate-limit rejections land mid-turn.
Let it run into the limit unattended.
Observe: synthetic spend-limit assistant message, resultis_error:true/subtype:"success", session exits gracefully, t3x-auto-resume.json shows pending:null and firedAtMs:[] for the thread, and no resume at the resetsAt boundary.
Both #6 and #39 were "the guard cancelled a resume that should have fired". This is the third instance of that shape, plus a limit class the feature has no model for.
Summary
An overnight self-paced
/looprun died at 06:34:49Z and never came back. Auto-resume was enabled for the thread and its poller was alive the whole time, but the durable store shows the thread never armed a pending resume and never fired one, despite receiving threeaccount.rate-limits.updatedevents withstatus:"rejected"— the exact signalAutoResumeReactor's detection fiber documents as its trigger.The distinguishing feature of this episode versus #6 and #39: the terminal block was not a plain
five_hourexhaustion. It wasfive_hourrejected plus overage rejected withoverageDisabledReason:"org_level_disabled_until", surfacing to the model asYou've hit your individual spend limit · run /usage-credits to ask your admin for a higher limit.Net cost: ~6 hours of wall clock, and an overnight build that stopped at item 5 of 6 in its first wave. The user found out the next morning.
Environment
resources/app.asar,resources/server.asarmtime 20:31).57e2846c-9cb1-40d4-8776-e5f7db3fe7b9, provider session4def7c10-494d-40ef-a685-36ff485c36d6.claude-opus-5[1m], efforthigh, runtime modefull-access.~/.t3/userdata/t3x-auto-resume.json.~/.t3/userdata/logs/provider/events.57e2846c-9cb1-40d4-8776-e5f7db3fe7b9.log.What the user was doing
A
/loopin dynamic mode (self-paced viaScheduleWakeup) driving a multi-wave prototype build, with work fanned out through backgroundWorkflowsubagents. Explicitly an unattended overnight run.Confirmed timeline
All times UTC. Local was UTC-4.
account.rate-limits.updated,status:"rejected",rateLimitType:"five_hour",resetsAt:1786963200(10:40:00Z),overageStatus:"allowed",isUsingOverage:trueoverageStatus:"allowed_warning",isUsingOverage:trueoverageStatus:"rejected",overageDisabledReason:"org_level_disabled_until",overageResetsAt:1788220800(2026-09-01),isUsingOverage:falseUX-06dies:"error":"You've hit your individual spend limit · run /usage-credits…""model":"<synthetic>", content = the spend-limit string,stop_reason:"stop_sequence"item.completeditemType:"assistant_message"status:"completed"result—is_error:truetogether withsubtype:"success",num_turns:1turn.completedScheduleWakeupfires:command_lifecyclestate:"started",command_uuid 9663690csystemsubtype:"init"status:"failed"resultagainis_error:true+subtype:"success"command_uuid→state:"cancelled"session.exited,reason:"Session stopped",exitKind:"graceful"five_hourwindow reset. Nothing happened.session.started— the human came back and resumed manuallyThe core defect
Detection is documented as:
That precondition was met three times.
account.rate-limits.updatedappears 25 times in this thread's log; the three carryingstatus:"rejected"are confirmed to be of that canonical type.Walking
planSchedulewith the 06:23:55 event:verdict.rejectedtrue,pendingResumeAtMsnull,windowOpensInFuturetrue (10:40:00Z > 06:23:55Z),firedInCapWindow0. That returns{kind:"schedule"}. Yet the store reads:enabled:true, so the per-thread switch was on.firedAtMsis empty andFIRED_HISTORY_RETENTION_MSis 25h, so a fire anywhere in this episode would still be visible — it never fired. Andpendingis null, so either it never armed or it armed and was cancelled.The wake fiber was demonstrably alive:
GET /api/coil/auto-resume?threadId=57e2846c…polls run continuously from 06:07:06Z to 12:52:04Z with no gap over 5 minutes, straight across the 10:40:00Z reset. So this is not "the machine was asleep".I could not narrow it further from the outside, because there is no telemetry for the auto-resume decision at all (see item 3).
Narrowed hypotheses
cancelReasonruns against a fresh snapshot.thread-advancedshould be excluded (the fix from Auto-resume never fires in practice: 'thread-advanced' false cancellation when the limited turn settles during the wait #6:latestTurnis null once the turn settles, and null is not treated as advancement) andprogressingshould be false aftersession.exited. That leavesthread-gone— issettledOverrideset to"settled", orarchivedAtpopulated, by a gracefulsession.exited? If so, every unattended run that exits cleanly while limited loses its pending resume. That would be the same class of bug as Auto-resume never fires in practice: 'thread-advanced' false cancellation when the limited turn settles during the wait #6/[Bug]: auto-resume is cancelled as "user-took-over" by any user message — including one that was itself rejected by a limit #39: a guard meant to detect human takeover firing on an ordinary lifecycle transition.rate_limit_infohere carries the newer overage fields (overageStatus,overageResetsAt,overageDisabledReason,isUsingOverage). If the verdict parser reads a composite/effective status rather thanrate_limit_info.status, anoverageStatus:"rejected"variant may not map toverdict.rejected.resetsAt(10:40:00Z) describes the five_hour bucket. The actual blocker was the individual spend limit —org_level_disabled_until, withoverageResetsAton 2026-09-01. A resume at 10:40 would have fired into a still-closed door and burned a ladder attempt. Auto-resume currently has no representation for "blocked until a human raises a spend cap", which is not a time-based condition at all.Hypothesis 3 holds regardless of which of 1 or 2 is true, and is the one worth designing for.
Three separable defects
1. A limit stop is recorded as a successful turn.
resultcarriesis_error:trueandsubtype:"success"simultaneously, and the item isstatus:"completed". Any consumer keyed onsubtypesees success. The reactor docstring already acknowledges the upstream shape ("a usage limit does not produce a failed turn (the SDK has no rate-limit result subtype)") and deliberately reads the structured signal instead — but nothing else in the fork does, so the "did my overnight run actually stop?" question has no reliable answer outside the rate-limit stream.2. A scheduled wake tick burns a turn into a closed limit, then the session exits silently. At 06:35:00 the loop's own
ScheduleWakeupstarted a turn;initreportedstatus:"failed", the model never ran, the command wascancelled, and 4m35s later the session exited gracefully. From the model's side the turn produced no output, so no follow-upScheduleWakeupwas armed and the loop was permanently dead. Two asks: (a) don't dispatch a scheduled wake while a rejection is in force and its window has not opened — defer it to the sameresetsAtmachinery; (b) a graceful exit while limited with a live loop should raise something the user can see the next morning. It produced no notification of any kind. This is closely related to #38 and #42.3. Auto-resume decisions are invisible. Grepping every rotated
server.trace.ndjson*foralready-pending,not-rejected,capped,autoResume,auto-resume scheduled, andrate_limit_eventreturns zero hits — the only auto-resume evidence in the traces is the HTTP poll route. There is no record of aplanScheduleverdict, acancelReason, or a fire attempt. #6 and #39 were both "auto-resume silently didn't fire" bugs; a third one is now unresolvable from a full set of logs. Emitting the arm/skip/cancel decision (with reason andresumeAtMs) as a domain event or timeline activity would have made this issue a one-line diagnosis.Suggested direction
planScheduleandcancelReasonoutcome with its reason. Highest value per line of code here.org_level_disabled_until" as a distinct terminal class from a windowed rate limit: do not schedule a time-based resume, mark the thread blocked-on-human, and notify.overageResetsAttwo weeks out is not a retry target.session.exitedshould preserve rather than invalidate a pending resume.Repro sketch
/loopthat fans work out through backgroundWorkflowsubagents, so turns are long and rate-limit rejections land mid-turn.resultis_error:true/subtype:"success", session exits gracefully,t3x-auto-resume.jsonshowspending:nullandfiredAtMs:[]for the thread, and no resume at theresetsAtboundary.Related
thread-advancedfalse cancellation when the limited turn settles during the waituser-took-overby any user message, including one itself rejected by a limitwake_meinstead #42 — self-paced loops: surface and bound the wake mechanismBoth #6 and #39 were "the guard cancelled a resume that should have fired". This is the third instance of that shape, plus a limit class the feature has no model for.