fix verbatim-repeat false positive on auth-retry; add subagent thrash detail - #295
Open
EdbertChan wants to merge 1 commit into
Open
fix verbatim-repeat false positive on auth-retry; add subagent thrash detail#295EdbertChan wants to merge 1 commit into
EdbertChan wants to merge 1 commit into
Conversation
… thrash detail ## Context - Problem: Two false positives in token_audit.py's detection pipeline. - Trigger: reflect-on-thrash hook fired on a session where the user's prompt hit an OAuth 401 (4 retries, no assistant response), the user interrupted, ran /login, and re-sent the same prompt. The verbatim-repeat detector flagged this as frustrated restatement instead of a legitimate retry after auth failure. ## Considerations - R1 (verbatim-repeat): The detector compared consecutive user messages without checking whether the agent ever responded between them. A re-send after an API error (no successful assistant turn) is a retry, not frustration. Fix: pass assistant JSONL line indices into frustration_signals(); suppress verbatim-repeat when no assistant turn occurred between the two sends. Non-Claude modes (Codex, OMP, Cursor) do not pass this parameter, so they retain the existing behavior. - B2 (subagent thrash detail): The --out JSON reported counts and flag names but not which files were redundantly read or what the recurring failure error messages were. Without specifics, the reflect lenses cannot act on the finding. Fix: expose redundant_read_files and recurring_failure_details in audit_claude()'s return dict, aggregate them per-agent in audit_subagents(). - Alternative for R1: could have checked timestamps of API error responses between user messages, but that requires parsing error shapes per-harness. Checking for the presence of a successful assistant turn is simpler and covers all error types (OAuth, rate limit, network). ## Blast Radius - Files touched: token_audit.py, test_token_audit.py - Risks: Changed the user_msgs ordinal from enumerate-counter to JSONL line index in Claude mode. This only affects the "index" field in flagged frustration output (used for reporting, not for logic elsewhere). Other modes are unchanged. ## Verification - 72 tests pass (69 original + 3 new), 6 subtests pass. - New tests cover: suppression when no assistant between sends, no suppression when assistant did respond, backward compat for non-Claude modes, subagent detail fields populated. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_a4af20cc-5047-4b9b-bd3e-f9bdfcfd731a) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The verbatim-repeat detector in token_audit.py flagged a user re-sending the same prompt after an OAuth 401 failure as frustrated restatement. The user interrupted, ran /login, and re-sent -- a legitimate retry, not frustration. The detector now checks whether a successful assistant response occurred between two identical sends. When no assistant turn happened between them, the re-send is suppressed as a retry.
Separately, the subagent thrash section in --out JSON reported counts and flag names but not which files were redundantly read or what the recurring failure signatures were. The reflect lenses could not act on the finding without that detail.
Review Claim
These two mechanical detection fixes reduce false positives in the frustration pipeline and make subagent thrash findings actionable.
Review Lane
behavior
Safety Invariant
Non-Claude modes (Codex, OMP, Cursor) do not pass assistant_turn_indices, so they retain the existing verbatim-repeat behavior. The user_msgs ordinal changed from enumerate-counter to JSONL line index in Claude mode, but this only affects the "index" field in flagged output (reporting, not logic).
Slice Rationale
Both fixes target the same file (token_audit.py) and the same audit pipeline. Splitting them would create two PRs that touch the same function signatures.
Non-goals
No changes to the frustration lens itself, to cat-mode SKILL.md, or to any other reflect script.
Test Plan
cd engine/skills/reflect/scripts && python3 -m pytest tests/test_token_audit.py -v-- 72 passed, 6 subtests passedRevert Plan
git revert <sha>Note
Low Risk
Changes are confined to reflect
token_auditheuristics and report shape; non-Claude paths are unchanged and behavior is covered by unit tests.Overview
Fixes false-positive verbatim-repeat frustration signals and enriches subagent thrash JSON for reflect audits.
Claude
token_auditnow passes JSONL line indices for human utterances and tracks where successful assistant turns occur.frustration_signalsaccepts optionalassistant_turn_indicesand uses_has_assistant_betweenso identical user prompts re-sent within 10 minutes without an assistant response in between (e.g. OAuth 401 → login → retry) no longer count asverbatim-repeat; repeats after an assistant reply still do. Codex/OMP/Cursor keep prior behavior because they do not pass that list.Audit output adds
redundant_read_filesandrecurring_failure_detailson Claude sessions;audit_subagentsrolls them intosubagents.thrashasagent:pathfile lists and per-agent recurring failure rows so--outreports are actionable beyond counts.Tests cover suppression vs real repeats, non-Claude backward compatibility, and subagent thrash detail fields.
Reviewed by Cursor Bugbot for commit 962090f. Bugbot is set up for automated code reviews on this repo. Configure here.