Skip to content

[auto] reflect: corpus_scan skips slow or oversized transcripts instead of crashing - #288

Open
EdbertChan wants to merge 1 commit into
mainfrom
stack/EdbertChan/reflect/pstack-premise-20260908/auto-reflect-corpus-scan-skips-slow-oversized--6b763dad
Open

[auto] reflect: corpus_scan skips slow or oversized transcripts instead of crashing#288
EdbertChan wants to merge 1 commit into
mainfrom
stack/EdbertChan/reflect/pstack-premise-20260908/auto-reflect-corpus-scan-skips-slow-oversized--6b763dad

Conversation

@EdbertChan

@EdbertChanEdbertChan commented Sep 8, 2026•
edited
Loading

Copy link
Copy Markdown
Owner

Summary

The reflect corpus scan no longer dies when one transcript is slow to grep. The slow file is skipped with a stderr line naming it, and the scan continues.

Files above a size cap (default 64 MB, flag --max-file-bytes) are skipped before grep with the same stderr line. The find-failure path now logs instead of swallowing.

Review Claim

_grep_match_files survives one slow or oversized transcript, names every skip on stderr, and still returns every other match.

Review Lane

behavior

Review Unit

engine-runtime

Safety Invariant

No scan result that used to succeed changes: files under the cap that grep in time still match exactly as before. Only the crash case and the new above-cap case turn into named stderr skips. Callers of discover_local pass no new argument and get the default cap.

Assumptions: headless mode, Safety Invariant is unconfirmed by a human.

Slice Rationale

This is the script fix from one reflect pass; its prose sibling (narrow-the-scope) is a separate review unit and sits in the next PR of this stack so each PR carries one Review Unit.

Non-goals

  • No change to remote (SSH) scanning, remote_scan_command, or _grep_count.
  • No change to which transcripts count as sidechain or how they are bucketed.
  • No new hook and no install.sh change.

Test Plan

Test Plan
  • python3 -m unittest engine.skills.reflect.scripts.tests.test_corpus_scan -v BEFORE the fix (test added, script unchanged):
test_audit_one_accepts_cursor_kind_without_tokens (engine.skills.reflect.scripts.tests.test_corpus_scan.TestAuditOneCursor.test_audit_one_accepts_cursor_kind_without_tokens) ... ok
test_detects_concurrent_multi_host_burst (engine.skills.reflect.scripts.tests.test_corpus_scan.TestBucketSummary.test_detects_concurrent_multi_host_burst) ... ok
test_ignores_entries_without_timestamp (engine.skills.reflect.scripts.tests.test_corpus_scan.TestBucketSummary.test_ignores_entries_without_timestamp) ... ok
test_discover_local_docstring_names_cursor (engine.skills.reflect.scripts.tests.test_corpus_scan.TestDiscoverLocalKinds.test_discover_local_docstring_names_cursor) ... ok
test_default_cap_is_64_mb (engine.skills.reflect.scripts.tests.test_corpus_scan.TestGrepMatchFilesSkips.test_default_cap_is_64_mb) ... ERROR
test_file_above_max_bytes_is_skipped_with_stderr_line (engine.skills.reflect.scripts.tests.test_corpus_scan.TestGrepMatchFilesSkips.test_file_above_max_bytes_is_skipped_with_stderr_line) ... ERROR
test_find_failure_logs_to_stderr_and_returns_empty (engine.skills.reflect.scripts.tests.test_corpus_scan.TestGrepMatchFilesSkips.test_find_failure_logs_to_stderr_and_returns_empty) ... FAIL
test_timeout_on_one_file_skips_it_and_keeps_the_rest (engine.skills.reflect.scripts.tests.test_corpus_scan.TestGrepMatchFilesSkips.test_timeout_on_one_file_skips_it_and_keeps_the_rest) ... ERROR
test_fractional_hours_round_not_truncate (engine.skills.reflect.scripts.tests.test_corpus_scan.TestMtimeMinutes.test_fractional_hours_round_not_truncate) ... ok
test_never_zero (engine.skills.reflect.scripts.tests.test_corpus_scan.TestMtimeMinutes.test_never_zero) ... ok
test_whole_hours (engine.skills.reflect.scripts.tests.test_corpus_scan.TestMtimeMinutes.test_whole_hours) ... ok
test_includes_cursor_agent_transcripts (engine.skills.reflect.scripts.tests.test_corpus_scan.TestRemoteScanCommand.test_includes_cursor_agent_transcripts) ... ok
test_pattern_is_embedded_verbatim (engine.skills.reflect.scripts.tests.test_corpus_scan.TestRemoteScanCommand.test_pattern_is_embedded_verbatim) ... ok
test_read_only_no_write_commands (engine.skills.reflect.scripts.tests.test_corpus_scan.TestRemoteScanCommand.test_read_only_no_write_commands) ... ok
test_ssh_args_from_target_config (engine.skills.reflect.scripts.tests.test_corpus_scan.TestRemoteScanCommand.test_ssh_args_from_target_config) ... ok
test_stderr_discarded_not_a_real_write (engine.skills.reflect.scripts.tests.test_corpus_scan.TestRemoteScanCommand.test_stderr_discarded_not_a_real_write) ... ok
test_uses_mmin_not_mtime (engine.skills.reflect.scripts.tests.test_corpus_scan.TestRemoteScanCommand.test_uses_mmin_not_mtime) ... ok
======================================================================
ERROR: test_default_cap_is_64_mb (engine.skills.reflect.scripts.tests.test_corpus_scan.TestGrepMatchFilesSkips.test_default_cap_is_64_mb)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/edbertchan/Documents/GitHub/catstack/.worktrees/reflect-pstack-premise/engine/skills/reflect/scripts/tests/test_corpus_scan.py", line 201, in test_default_cap_is_64_mb
self.assertEqual(corpus_scan.DEFAULT_MAX_FILE_BYTES, 64 * 1024 * 1024)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'corpus_scan' has no attribute 'DEFAULT_MAX_FILE_BYTES'
======================================================================
ERROR: test_file_above_max_bytes_is_skipped_with_stderr_line (engine.skills.reflect.scripts.tests.test_corpus_scan.TestGrepMatchFilesSkips.test_file_above_max_bytes_is_skipped_with_stderr_line)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/edbertchan/Documents/GitHub/catstack/.worktrees/reflect-pstack-premise/engine/skills/reflect/scripts/tests/test_corpus_scan.py", line 194, in test_file_above_max_bytes_is_skipped_with_stderr_line
matched, err = self._run(max_file_bytes=50)
~~~~~~~~~^^^^^^^^^^^^^^^^^^^
File "/Users/edbertchan/Documents/GitHub/catstack/.worktrees/reflect-pstack-premise/engine/skills/reflect/scripts/tests/test_corpus_scan.py", line 179, in _run
matched = corpus_scan._grep_match_files(["find", self.tmp.name], "hit", 24, **kwargs)
TypeError: _grep_match_files() got an unexpected keyword argument 'max_file_bytes'
======================================================================
ERROR: test_timeout_on_one_file_skips_it_and_keeps_the_rest (engine.skills.reflect.scripts.tests.test_corpus_scan.TestGrepMatchFilesSkips.test_timeout_on_one_file_skips_it_and_keeps_the_rest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/edbertchan/Documents/GitHub/catstack/.worktrees/reflect-pstack-premise/engine/skills/reflect/scripts/tests/test_corpus_scan.py", line 184, in test_timeout_on_one_file_skips_it_and_keeps_the_rest
matched, err = self._run()
~~~~~~~~~^^
File "/Users/edbertchan/Documents/GitHub/catstack/.worktrees/reflect-pstack-premise/engine/skills/reflect/scripts/tests/test_corpus_scan.py", line 179, in _run
matched = corpus_scan._grep_match_files(["find", self.tmp.name], "hit", 24, **kwargs)
File "/Users/edbertchan/Documents/GitHub/catstack/.worktrees/reflect-pstack-premise/engine/skills/reflect/scripts/corpus_scan.py", line 112, in _grep_match_files
r = subprocess.run(["grep", "-l", "-i", "-E", pattern, p], capture_output=True, text=True, timeout=15)
File "/Users/edbertchan/Documents/GitHub/catstack/.worktrees/reflect-pstack-premise/engine/skills/reflect/scripts/tests/test_corpus_scan.py", line 171, in fake_run
raise sp.TimeoutExpired(cmd, kwargs.get("timeout"))
subprocess.TimeoutExpired: Command '['grep', '-l', '-i', '-E', 'hit', '/var/folders/rq/1lflpr1n1jzgkvdc4f6cqfw80000gn/T/tmpjcoqt1qo/slow.jsonl']' timed out after 15 seconds
======================================================================
FAIL: test_find_failure_logs_to_stderr_and_returns_empty (engine.skills.reflect.scripts.tests.test_corpus_scan.TestGrepMatchFilesSkips.test_find_failure_logs_to_stderr_and_returns_empty)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/edbertchan/Documents/GitHub/catstack/.worktrees/reflect-pstack-premise/engine/skills/reflect/scripts/tests/test_corpus_scan.py", line 212, in test_find_failure_logs_to_stderr_and_returns_empty
self.assertIn("find", err)
~~~~~~~~~~~~~^^^^^^^^^^^^^
AssertionError: 'find' not found in ''
----------------------------------------------------------------------
Ran 17 tests in 0.008s
FAILED (failures=1, errors=3)
exit=1
  • Same command AFTER the fix:
test_audit_one_accepts_cursor_kind_without_tokens (engine.skills.reflect.scripts.tests.test_corpus_scan.TestAuditOneCursor.test_audit_one_accepts_cursor_kind_without_tokens) ... ok
test_detects_concurrent_multi_host_burst (engine.skills.reflect.scripts.tests.test_corpus_scan.TestBucketSummary.test_detects_concurrent_multi_host_burst) ... ok
test_ignores_entries_without_timestamp (engine.skills.reflect.scripts.tests.test_corpus_scan.TestBucketSummary.test_ignores_entries_without_timestamp) ... ok
test_discover_local_docstring_names_cursor (engine.skills.reflect.scripts.tests.test_corpus_scan.TestDiscoverLocalKinds.test_discover_local_docstring_names_cursor) ... ok
test_default_cap_is_64_mb (engine.skills.reflect.scripts.tests.test_corpus_scan.TestGrepMatchFilesSkips.test_default_cap_is_64_mb) ... ok
test_file_above_max_bytes_is_skipped_with_stderr_line (engine.skills.reflect.scripts.tests.test_corpus_scan.TestGrepMatchFilesSkips.test_file_above_max_bytes_is_skipped_with_stderr_line) ... ok
test_find_failure_logs_to_stderr_and_returns_empty (engine.skills.reflect.scripts.tests.test_corpus_scan.TestGrepMatchFilesSkips.test_find_failure_logs_to_stderr_and_returns_empty) ... ok
test_timeout_on_one_file_skips_it_and_keeps_the_rest (engine.skills.reflect.scripts.tests.test_corpus_scan.TestGrepMatchFilesSkips.test_timeout_on_one_file_skips_it_and_keeps_the_rest) ... ok
test_fractional_hours_round_not_truncate (engine.skills.reflect.scripts.tests.test_corpus_scan.TestMtimeMinutes.test_fractional_hours_round_not_truncate) ... ok
test_never_zero (engine.skills.reflect.scripts.tests.test_corpus_scan.TestMtimeMinutes.test_never_zero) ... ok
test_whole_hours (engine.skills.reflect.scripts.tests.test_corpus_scan.TestMtimeMinutes.test_whole_hours) ... ok
test_includes_cursor_agent_transcripts (engine.skills.reflect.scripts.tests.test_corpus_scan.TestRemoteScanCommand.test_includes_cursor_agent_transcripts) ... ok
test_pattern_is_embedded_verbatim (engine.skills.reflect.scripts.tests.test_corpus_scan.TestRemoteScanCommand.test_pattern_is_embedded_verbatim) ... ok
test_read_only_no_write_commands (engine.skills.reflect.scripts.tests.test_corpus_scan.TestRemoteScanCommand.test_read_only_no_write_commands) ... ok
test_ssh_args_from_target_config (engine.skills.reflect.scripts.tests.test_corpus_scan.TestRemoteScanCommand.test_ssh_args_from_target_config) ... ok
test_stderr_discarded_not_a_real_write (engine.skills.reflect.scripts.tests.test_corpus_scan.TestRemoteScanCommand.test_stderr_discarded_not_a_real_write) ... ok
test_uses_mmin_not_mtime (engine.skills.reflect.scripts.tests.test_corpus_scan.TestRemoteScanCommand.test_uses_mmin_not_mtime) ... ok
----------------------------------------------------------------------
Ran 17 tests in 0.006s
OK
exit=0
  • python3 -m unittest discover -s engine/skills/reflect/scripts/tests -> Ran 155 tests in 17.119s / OK
  • bash scripts/run_all_tests.sh -> exit 0; per-suite results:
=== ./engine/hooks/agent-relay-attribution/tests ===
Ran 6 tests in 0.002s
OK
=== ./engine/hooks/auto-pr/tests ===
Ran 24 tests in 10.222s
OK
=== ./engine/hooks/bug-complaint-leak/tests ===
Ran 7 tests in 0.003s
OK
=== ./engine/hooks/build-the-lever/tests ===
Ran 11 tests in 0.006s
OK
=== ./engine/hooks/cat-mode-default/tests ===
Ran 41 tests in 0.042s
OK
=== ./engine/hooks/demo-freeze/tests ===
Ran 8 tests in 0.002s
OK
=== ./engine/hooks/diu-stop/tests ===
Ran 67 tests in 0.004s
OK
=== ./engine/hooks/explicit-failures/tests ===
Ran 28 tests in 0.006s
OK
=== ./engine/hooks/frustration-watchdog/tests ===
Ran 15 tests in 0.004s
OK
=== ./engine/hooks/hedge-runs-prove-it/tests ===
Ran 9 tests in 0.002s
OK
=== ./engine/hooks/named-verb-guard/tests ===
Ran 21 tests in 0.004s
OK
=== ./engine/hooks/narrow-the-scope/tests ===
Ran 7 tests in 0.012s
OK
=== ./engine/hooks/new-file-callout/tests ===
Ran 8 tests in 1.255s
OK
=== ./engine/hooks/no-comments/tests ===
Ran 10 tests in 0.000s
OK
=== ./engine/hooks/pr-schema-gate/tests ===
Ran 54 tests in 0.032s
OK
=== ./engine/hooks/prove-it-ship-gate/tests ===
Ran 10 tests in 0.003s
OK
=== ./engine/hooks/reflect-on-thrash/tests ===
Ran 24 tests in 0.022s
OK
=== ./engine/hooks/repeat-error-stop/tests ===
Ran 29 tests in 0.015s
OK
=== ./engine/hooks/restart-risk-check/tests ===
Ran 25 tests in 0.003s
OK
=== ./engine/hooks/restated-constraint/tests ===
Ran 16 tests in 0.003s
OK
=== ./engine/hooks/scope-lock/tests ===
Ran 31 tests in 0.023s
OK
=== ./engine/hooks/scratchpad-collision/tests ===
Ran 10 tests in 0.009s
OK
=== ./engine/hooks/skill-usage-log/tests ===
Ran 7 tests in 0.004s
OK
=== ./engine/hooks/wait-needs-wakeup/tests ===
Ran 22 tests in 0.005s
OK
=== ./engine/hooks/wrong-check-reflect/tests ===
Ran 36 tests in 0.406s
OK
=== ./engine/skills/draft-pr/tests ===
Ran 2 tests in 0.471s
OK
=== ./engine/skills/make-pr/tests ===
Ran 12 tests in 0.141s
OK
=== ./engine/skills/reflect/scripts/tests ===
Ran 155 tests in 16.875s
OK
=== CODEX token audit: codex_thrash_session.jsonl ===
=== ./product/skills/independent-judge-swarm/tests ===
Ran 6 tests in 0.004s
OK (skipped=1)
=== ./product/skills/land-stack/tests ===
Ran 11 tests in 0.715s
OK
=== ./tests ===
Ran 228 tests in 166.461s
OK
  • python3 engine/skills/make-pr/scripts/preflight.py --base origin/main -> ok preflight passed
  • python3 scripts/check_no_new_comments.py --base origin/main -> ok no new comments
  • python3 scripts/check_no_dated_provenance.py --base origin/main -> ok no dated provenance
  • python3 scripts/check_skill_test_coverage.py --base origin/main --head HEAD -> ok skill test coverage
  • node engine/skills/draft-pr/scripts/lint-diff-atomicity.mjs --base origin/main -> Diff atomicity validation passed.

Revert Plan

Revert Plan
  • Safe to revert? Yes
  • Revert command: git revert 467f5ec
  • Post-revert steps: None
  • Data migration? No

🤖 Generated with Claude Code

https://claude.ai/code/session_014NSMiMsTPuon8otdhRaJEq

@EdbertChan@claude
[auto] reflect: corpus_scan skips slow or oversized transcripts inste…
467f5ec
…ad of crashing
_grep_match_files ran `grep -l` per file with timeout=15 and no handler, so
one slow transcript (a 146 MB Codex rollout) raised subprocess.TimeoutExpired
out of the loop and killed the whole corpus scan. Catch TimeoutExpired per
file, print an explicit stderr skip line naming the path and the timeout,
and continue. Add a size cap (--max-file-bytes, default 64 MB) that skips
oversized files before grep with the same explicit stderr line. The find
failure path keeps returning [] but now logs the command and error to
stderr instead of swallowing it silently.
Tests: TestGrepMatchFilesSkips monkeypatches subprocess.run to raise
TimeoutExpired for one path and asserts the other paths still return and
the skipped path is named on stderr; also covers the size cap, the 64 MB
default, and the find-failure log line.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014NSMiMsTPuon8otdhRaJEq
Change-Id: I6b763dad53f9f35f7e4ebe3154816c54ee00fe08
@EdbertChanEdbertChan mentioned this pull request Sep 8, 2026
Open
4 tasks
@EdbertChan

EdbertChan commented Sep 8, 2026

Copy link
Copy Markdown
OwnerAuthor

This pull request is part of a Mergify stack:

#Pull RequestLink
1[auto] reflect: corpus_scan skips slow or oversized transcripts instead of crashing#288👈
2[auto] reflect: narrow-the-scope names the shared-premise step when one gate rejects twice#289

@cursor

cursorBot commented Sep 8, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_063a7956-1c3d-45a8-b552-629b411dcf54)

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@EdbertChan