fix(acp): park session-limit failures until reset without burning retries - #5975
olympusbuildz wants to merge 2 commits into
Conversation
…ries Provider session-limit errors (e.g. Claude "You've hit your session limit · resets 1:50pm …") self-heal after hours, but the ordinary EventQueue retry budget exhausts in ~25 minutes and dead-letters the batch. Classify the error, park via requeue_until with a parsed/fallback delay, and leave retry_counts untouched so the message is still answered when quota returns. Fixes block#5918 Signed-off-by: Olympusbuildz <Olympus.roots@outlook.com> Co-authored-by: Olympusbuildz <Olympus.roots@outlook.com> Signed-off-by: Olympusbuildz <Olympus.roots@outlook.com>
themiguelamador
left a comment
There was a problem hiding this comment.
I found three issues in the submitted implementation:
- The exact observed Claude message includes an explicit timezone (
America/Buenos_Aires), but the parser ignored it and interpreted the wall clock in the host timezone. That can retry before the real reset and then roll the next attempt to tomorrow. - An oversized numeric delay from untrusted provider error text can overflow to infinity during unit conversion and make
Duration::from_secs_f64panic. requeue_untilremoves from the back on overflow (the newest arrivals), while its warning said it dropped the oldest event.
I fixed all three in Complear/buzz commit f8aaed727 (review/pr-5975-fix): explicit named timezones now take the bounded one-hour fallback, duration conversion is clamped before constructing Duration, and the queue documentation/log accurately describe the eviction policy.
Verification: cargo test -p buzz-acp --lib (787 passed), focused parser/requeue/classifier tests, cargo clippy -p buzz-acp --lib -- -D warnings, and cargo fmt --all --check.
Address review on block#5975: - Named timezones in wall-clock reset text (e.g. America/Buenos_Aires) fall back to the 1h park instead of host-local guessing. - Oversized relative delays clamp before Duration construction (no inf panic). - requeue_until overflow log/docs match pop_back newest-at-back eviction. Signed-off-by: Olympusbuildz <Olympus.roots@outlook.com> Co-authored-by: Olympusbuildz <Olympus.roots@outlook.com> Signed-off-by: Olympusbuildz <Olympus.roots@outlook.com>
|
Addressed review (new commit
Mini focused: parse_session_limit (6) + requeue_until + is_session_limit green @ Thanks @themiguelamador. |
|
Field data from a different provider, because this fix has a gap that my logs make obvious. I run a fleet of managed agents on Windows against several providers. I hit #5918 independently this morning, worked it back through the source, and landed on your branch. So the useful part first. This wants merging. Changes were requested on 2026-08-16, you addressed them the same week, and nothing has moved since 2026-08-19. That is a long time to sit on a bug that permanently destroys user messages. Now the gap.
Two words. No reset instant, no
It returns false. An actual rate limit falls straight through to the generic ladder and the message is destroyed 23 minutes later. 24 hours on one machine, 9 agents:
Be careful with that 283, because I nearly overstated it myself. The other 137 are What I ran and what I did not. I did not compile your branch. The fix is small, because your delay side already covers the hard part. My opinion, and it is the reason I bothered writing this. Stop matching English. matches!(error, acp::AcpError::AgentError { code: -32003, .. })Keep the phrase list, because Claude folds quota into Happy to send this as a commit against your branch with tests instead of a comment. Your call. |
Problem
When a turn fails with a provider session limit error,
buzz-acpretries on the ordinary exponential backoff budget (MAX_RETRIES = 10, ~25 minutes) and then dead-letters the batch. Subscription limits reset on a multi-hour wall clock, so the message is discarded before the agent could ever answer it — and it is never reprocessed when quota returns.Observed Claude subscription text:
You've hit your session limit · resets 1:50pm (America/Buenos_Aires)Root cause
handle_prompt_resultalready special-cases non-retryable auth errors (immediate dead-letter). Session-limit errors fall through to genericqueue.requeue(), which burns the retry counter. The error string carries the reset instant; the harness ignored it.Fix
is_session_limit_error).resets in 2h,retry in 45m, wall-clockresets 1:50pmbest-effort in host local TZ). Fallback 1h, cap 6h.EventQueue::requeue_untilparks the batch atretry_afterwithout incrementingretry_counts.handle_prompt_result.Why it matters
Owner DMs and channel mentions are permanently lost during multi-hour subscription windows even though the agent process is healthy. Parking preserves the work until quota returns.
Test plan
All of the above passed @ this head.
Full
cargo test -p buzz-acp --lib: 783 passed; 2 failures are pre-existing onorigin/main(config::tests::lazy_pool_defaults_off,idle_pool_sleep_defaults_disabled_and_accepts_cli_value) — same fail on clean main at78cbffewhen env defaults differ; not introduced by this diff.cargo fmt -p buzz-acpclean;cargo clippy -p buzz-acp --lib -- -D warningsclean.Risk / blast radius
PromptOutcome::Errormessages matching high-precision limit phrases take the park path.Closest work
MAX_RETRIESconfigurable (orthogonal; does not classify session-limit or park until reset)Fixes #5918
Peer-review harden
Addressed @themiguelamador review @
37b5a0a35:Duration::from_secs_f64(no inf panic)requeue_untiloverflow log/docs matchpop_backnewest-at-back eviction