test(bench): Tier 1 standalone binary harness with isolated DB + JSON/MD reports - #30
Merged
Merged
Conversation
…/MD reports (#29) First tier of the three-tier bench harness scoped in #29. Standalone test that invokes the built contextcrawler binary as a subprocess with `RTK_DB_PATH` pointed at a fresh tempfile, so the user's real tracking DB (`~/Library/Application Support/rtk/history.db`) is never touched. Verified isolation: live DB row count stayed at 4202 across two harness runs that internally invoked the binary four times each. **Cases (against `tests/fixtures/bench/*`):** - `read_short_passthrough` — small file, exercises the no-cap-fires path - `read_large_unknownext_cap_fires` — 500-line .unknownext, exercises the #15 symmetric 80/80 cap. Empirical: 7000 → 2262 tokens, 67.7% savings. Assertion floor 50%. - `read_xcstrings_json_route` — synthetic Xcode strings file, exercises JSON-like extension routing through the JSON compactor (NOT the unknown-ext cap). Empirical: 1807 → 389 tokens, 78.5% savings. Assertion floor >0%. - `read_rust_minimal_filter` — code file with comments. Currently reports 0% (default FilterLevel doesn't strip on this case); kept as a baseline so a future change that enables aggressive filtering can measure the lift. **Aggregate first run on develop tip 5cc6da3: 69.1% weighted savings across 8912 input tokens.** This is the baseline future runs compare against. **Output:** - `bench/results-<git-sha>.json` — machine-readable, for pre/post diff - `bench/results-<git-sha>.md` — human-readable, PR-diff-friendly - Both gitignored under `bench/results-*.{json,md}` so accumulating reports don't pollute the tree **Reproducibility:** same fixtures → same numbers within tolerance. Bisectable across binary commits. Hard regression gates: - Every case must exit 0 - Cap case savings must stay ≥50% (currently 67.7% — 17pp headroom) - xcstrings case must save tokens (currently 78.5% — far above floor) Softer gates (delta against `bench/baseline.json`) deferred to a follow-up so this PR stays focused on the harness scaffolding. **Tier 2 (Claude Code) + Tier 3 (Codex) deferred** to follow-up PRs per the #29 build order. Tier 3 will subsume issue #28's `codex-compliance` measurement. Tests: 1 new harness + 2 branding-lint + 2027 bin = all pass. Refs #29
Three findings raised by `codex review`, all valid:
1. **[P1] Resolve binary via Cargo.** Was hardcoding
`target/debug/contextcrawler` — brittle to `CARGO_TARGET_DIR`,
release builds, workspace layouts. Switched to
`env!("CARGO_BIN_EXE_contextcrawler")` which Cargo sets
automatically for integration tests.
2. **[P2] Use the real telemetry opt-out env var.** Was setting
`RTK_NO_TELEMETRY=1` which nothing reads. The runtime checks
`RTK_TELEMETRY_DISABLED=1` (see src/core/telemetry.rs:27). Fixed —
harness invocations now genuinely skip the daily telemetry ping
instead of just looking like they do.
3. **[P2] Use the production token estimator.** Was using
whitespace-split; the crate's `tracking::estimate_tokens` is
`(text.len() / 4).ceil()`. Bench numbers were diverging from the
live `contextcrawler gain` output and from the SQLite history DB,
making pre/post comparison across tiers inconsistent. Now matches.
Side effect of the estimator change: aggregate savings re-measured
from 69.1% (whitespace) to 65.1% (production estimator). All
assertions still pass; the new numbers are the correct baseline
for future bench/baseline.json comparisons.
Per-case post-fix:
- read_short_passthrough: 26 → 26, 0% (passthrough, expected)
- read_large_unknownext_cap_fires: 8473 → 2790, 67.1% (cap firing)
- read_xcstrings_json_route: 3083 → 1135, 63.2% (JSON compaction)
- read_rust_minimal_filter: 134 → 134, 0% (default FilterLevel
doesn't strip; kept as a baseline for the future case where a
follow-up enables aggressive filtering)
Added a doc comment on `count_tokens()` noting it MUST track
production's `estimate_tokens` formula to keep the numbers
comparable. If the production estimator ever changes, this function
needs to follow.
Co-Authored-By: Codex review <noreply@openai.com>
This was referenced May 18, 2026
Merged
noogalabs
pushed a commit
to noogalabs/contextcrawler
that referenced
this pull request
Jun 4, 2026
… branding-lint extension **Version bump 0.39.0 → 0.1.7.** The upstream rebase pulled in rtk's `0.39.0` version; restoring the downstream `0.1.x` numbering line per the maintainer's call. `.release-please-manifest.json` updated to match. Last downstream tag was `v0.1.6` (2026-05-15); this is the next in the sequence. **release-please-config.json `package-name: "rtk"` → "contextcrawler".** Caught by manual audit during release prep. Same regression-family as thehoff#19 / thehoff#20 / thehoff#22: a rebrand gap the prior branding-lint (src/-scoped) didn't catch because release-please-config.json lives at repo root. Without this fix, release-please would have produced `rtk-vX.Y.Z` tags and release-PR titles. **Cargo.toml `extended-description` rebrand.** Line 77 still read "rtk filters and compresses..." — same upstream-leak family. Fixed. **Extended `tests/branding_lint.rs`** with a third test — `branding_lint_config_files_pin_canonical_package_name` — that: - Parses Cargo.toml and asserts `[package].name == "contextcrawler"` - Parses release-please-config.json and asserts `packages["."]["package-name"] == "contextcrawler"` This closes the gap that let release-please-config.json silently drift to `"rtk"` post-rebase. Any future rebase that re-introduces an upstream package-name fails `cargo test --test branding_lint` with an explicit error pointing at issues thehoff#19/thehoff#20/thehoff#22 history. **CHANGELOG.md** entry for v0.1.7 added, summarizing all PRs merged today (thehoff#10 thehoff#15 thehoff#16 thehoff#21 thehoff#24 thehoff#25 thehoff#30) plus the deferred follow-ups (thehoff#26 thehoff#27 thehoff#28 thehoff#29). Tests: 2027 bin + 3 branding-lint (1 new) + 1 harness = all pass. Verified: `contextcrawler --version` now prints `contextcrawler 0.1.7`. Refs thehoff#19 thehoff#20 thehoff#22 thehoff#29
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
First tier of the three-tier bench harness scoped in #29. A Rust integration test that invokes the built
contextcrawlerbinary as a subprocess withRTK_DB_PATHpointed at an isolated tempfile, runs a fixed battery of cases againsttests/fixtures/bench/*, and produces JSON + Markdown reports for pre/post comparison.Baseline numbers (develop tip 5cc6da3)
Numbers use the production
(len/4).ceil()token estimator so they're directly comparable tocontextcrawler gainand the SQLite history DB.Isolation verified
Live DB row count stayed at 4202 across two consecutive harness runs that internally invoked the binary 4× each. The harness never touches
~/Library/Application Support/rtk/history.db.Hard regression gates
read_large_unknownext_cap_firessavings must stay ≥50% (currently 67.1%, 17pp headroom)read_xcstrings_json_routesavings must be >0% (currently 63.2%)Softer baseline.json diffs deferred to a follow-up so this PR stays focused on the scaffolding.
Reviewed by Codex
target/debug/contextcrawler→ switched toenv!("CARGO_BIN_EXE_contextcrawler")(Cargo idiom)RTK_NO_TELEMETRY(non-existent) → corrected toRTK_TELEMETRY_DISABLED(the actual env var)(len/4).ceil()formula; aggregate moved 69.1% → 65.1% (numbers now consistent withgainoutput)Tier 2 / 3 deferred
Per #29 build order. Tier 3 will subsume #28's
codex-compliancemeasurement deliverable.Test plan
cargo test --test harness_standalone— 1 pass, produces reportcargo test --bin contextcrawler— 2027 pass (no regressions)cargo test --test branding_lint— 2 passRefs #29