perf(read): .xcstrings (and similar JSON-ish) files bypass filter; add unknown-extension token cap

[thehoff]

Summary

Single rtk read Resources/Localizable.xcstrings call produced 605,386 input tokens with 0% savings — full passthrough. This one call is 77% of all rtk read input volume across 180 calls in the history DB. Without it, rtk read would achieve 54.5% weighted savings instead of 12.5%.

Reproduction

sqlite3 ~/Library/Application\ Support/rtk/history.db \
  "SELECT original_cmd, input_tokens, saved_tokens FROM commands WHERE rtk_cmd LIKE 'rtk read%' ORDER BY input_tokens DESC LIMIT 5; "

Top result: cat Resources/Localizable.xcstrings | 605386 | 0

Root cause

.xcstrings is Xcode localization (JSON-structured) but the extension isn't registered, so the read filter falls through to passthrough.

Other observed 0% reads with similar extension-gap pattern: mock-fixtures.js (3,598), .out files in /private/tmp/claude-*/tasks/ (multiple ~2-3K each — separate handling in #11).

Fix (refined design)

Two changes in src/cmds/system/read.rs (or wherever the read filter dispatcher lives):

1. Register JSON-ish extensions

Add to the existing JSON filter path: .xcstrings, .geojson, .ipynb, .webmanifest, .code-workspace. Use whatever extension-mapping mechanism the existing code uses — do not invent a new one.

2. Unknown-extension symmetric head/tail cap

When a file's tokenized input exceeds the threshold AND no filter matched the extension, return:

<first HEAD lines of file>
[━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━]
[ContextCrawler: omitted N lines (T tokens). Full file: contextcrawler proxy cat <path>]
<last TAIL lines of file>

Defaults:

• Threshold: 5000 tokens
• Head: 80 lines
• Tail: 80 lines (symmetric — equal weight to start and end of file)
• Marker: 2 lines, visually unmissable, includes escape hatch (contextcrawler proxy cat <path>) so any reader (human or LLM) knows how to recover full content
• Add an allowlist for known source-code extensions that should bypass the cap entirely (default empty; user populates in config)

Configurable in ~/.config/rtk/config.toml under new [read] section:

[read]
unknown_ext_token_threshold = 5000
unknown_ext_head_lines = 80
unknown_ext_tail_lines = 80
unknown_ext_passthrough_extensions = []  # e.g. [".svelte", ".astro", ".zig"]

Why symmetric 80:80 (not 80:20 or threshold bump)

• Files that exceed the threshold often have meaningful content at the END too (return values, conclusions, summaries, final assertions in tests). Equal head/tail preserves both.
• Keeps threshold aggressive (5K) so we still catch the 600K-token monsters early.
• Marker is loud enough to be obvious yet only 2 lines, minimal noise overhead.
• The escape hatch in the marker text means readers can self-recover when full content is needed.

Empirical impact estimate

Pushing rtk read from 12.5% weighted → 50% weighted lifts global savings ~24.8% → ~31% on the current dataset.

Risk assessment (re: client/harness context loss)

• Source files with uncommon extensions (.svelte, .astro, .zig, etc.): mitigated by the unknown_ext_passthrough_extensions allowlist + escape hatch in marker
• Large config files with distributed meaning: 80:80 head/tail captures most structurally important content; full read available via escape hatch
• Filtered files (recognized extension): unaffected
• Subagent transcripts under claude-*/tasks/: handled positively — harness explicitly warns against full-reading these; cap protects against accidental context blowup. See discussion under perf(read): claude task .out logs pass through unfiltered (0% savings, frequent) #11.

Test plan

• Snapshot + token-savings test for .xcstrings (synthetic 1K-line JSON-like fixture). Assert ≥60% savings.
• Token-savings test for unknown-extension cap: synthetic 500-line .unknowntype file with distinct content at top, middle, bottom. Assert head/tail/marker structure correct, middle content absent, marker contains escape-hatch text, savings >50%.
• Threshold edge: file exactly at threshold passes through fully; one line over → capped.
• Allowlist test: extension added to unknown_ext_passthrough_extensions bypasses cap regardless of size.
• Existing read tests pass: full cargo test --bin contextcrawler.