feat(gh): lossless csv+schema packing for gh --json and gh api output

[rbotte]

What

gh <cmd> --json … and gh api … are currently full passthrough (#196), tracked at 0% savings — exactly the outputs where most bytes are field names repeated on every row. The old run_api comment states why: "Converting JSON to a schema destroys all values and forces Claude to re-fetch." That objection is to lossy conversion.

This PR adds core/jsonpack: a lossless re-encoding that keeps every value and removes only the repetition:

• Top-level arrays of objects → CSV+schema:

  [3]{id:int,author.login:string,title:string,tag:string?}
  1,alice,First title,x
  2,bob,"Title, with a comma",
  3,carol,Third title,null
  

  [N] declares the row count (truncation is detectable), ? marks nullable columns, an empty cell means the key was absent, bare null means JSON null, lookalike strings ("null", "42" in string columns) stay quoted, mixed-type columns carry JSON literals so 42 and "42" can never blur. Uniform nested objects flatten recursively into dotted columns (commit.author.name).

• Envelopes (gh api objects) stay valid JSON: dense inner arrays of objects become {"_cols":[…],"_rows":[[…],…]} tables (uniform nested objects flatten into dotted _cols, marked "_flat":1), and the body is minified.

• --jq / --template (long, short -q/-t, = and attached forms) keep full passthrough — the caller asked for a projection. The registry rewrites bare --json to rtk gh again.

Losslessness is verified at runtime, not assumed

pack() decodes its own output and requires exact value equality with the parsed input before emitting a single byte. Any mismatch — encoder bug, or adversarial data that collides with the notation itself (a real {"_cols":…,"_rows":…} object in the payload) — returns the raw bytes unchanged. Tests include those adversarial cases. This honors Never Block, composes with the existing never_worse token guard, and means the worst case of this feature is the status quo (raw passthrough).

The only normalization is serde_json number parsing (1.10 → 1.1), shared by every JSON path in rtk today.

Measured (real data, rtk-ai/rtk)

command	before	after
gh api repos/rtk-ai/rtk/actions/runs?per_page=12	0% (passthrough)	20% (7.2K tokens)
gh api repos/rtk-ai/rtk/commits?per_page=25	0%	18% (5.2K tokens)
gh pr list --limit 50 --json number,title,state,author,updatedAt,url	0%	17%
gh issue list --limit 30 --json …	0%	4%
gh api repos/rtk-ai/rtk (single object)	0%	~0% (GitHub already minifies; falls back raw)

Real payloads are value-dominated (URLs, ids, timestamps), so the honest lossless ceiling here is well below the synthetic-fixture numbers — the two committed real-capture fixtures (produced by gh, emails sanitized) pin round-trip + ≥10% savings on actual bytes, precisely so synthetic fixtures can't overstate.

Design notes

• Ported from the lossless compaction stage of headroom-core (Apache-2.0, attribution in the module docs) with every lossy path removed: no retrieval pointers, no row dropping under budget, no stringified-JSON rewriting. A dependency was ruled out — the crate pulls ONNX Runtime, HF tokenizers, tree-sitter grammars and SQLite; the lossless algorithm is ~500 lines with zero new dependencies.
• Transparency trade-off, stated plainly: this emits a format the model didn't literally ask for. The declaration line is self-describing, envelopes stay valid JSON, and every value is verbatim — but if you prefer keeping --json passthrough and only packing gh api, the gate is one line in gh_cmd::run.
• Known future work: arrays-of-objects inside cells (e.g. issue labels) still render as JSON cells and pay CSV quote-doubling — nested sub-tables would lift gh issue list beyond 4%; glab api could reuse the packer as-is.

Capture is byte-exact on the packed paths — "raw fallback" means raw bytes

An adversarial pre-submission review (fake gh on PATH, end-to-end through the hook) drove the third commit. The shared line-oriented capture would have made "fall back to raw" untrue for gh api: stdin cut (--input - sending empty bodies), single lines beyond the 10MiB cap dropped entirely (empty stdout, exit 0), binary bodies decoded lossily. The packed paths therefore use their own byte-exact runner: stdin/stderr inherited, strict UTF-8 or verbatim bytes, and outputs beyond 32MiB stream through untouched with bounded memory. Additionally: a file redirect (--json … > f.json / gh api … > f.json) skips the rewrite at the registry (a program will parse that file); pflag fused shorts (-dq 'expr') are detected as projections; the flatten pass has a hard 512-column ceiling (its width cost is quadratic); and serde_json gains float_roundtrip so cited float digits are exactly the digits the API sent.

Tests

• 44 unit tests in jsonpack (round-trips for commas/quotes/CRLF/unicode, null vs absent vs empty vs lookalikes, extreme numbers, recursive flatten with missing parents, adversarial notation collisions, strict decoder rejections, column-cap bailout, float digit fidelity) + gh_cmd gate tests + registry tests (incl. redirect skip).
• tests/gh_pack_process_fidelity_test.rs: 6 integration tests driving the real binary against a fake gh on PATH — stdin reaches gh, a >10MiB single-line body survives whole, a body past the pack cap streams verbatim, a gzip body stays byte-exact, tabular output still packs, error bodies pass through with gh's exit code. These cover the plumbing, where unit tests are blind and where all three HIGH bugs lived.
• Those six are mutation-verified: restoring the previous line-oriented capture makes exactly the four fidelity tests fail with the original symptoms (stdin_bytes 0, 0x8b decoded to U+FFFD, 0 bytes emitted for a 34MB body) while both controls keep passing.
• Review fuzz: 800k randomized round-trip iterations, zero verify mismatches, zero panics.
• Full suite: 2645 tests green across unit + integration; cargo fmt --check, clippy --all-targets (deny warnings), scripts/check-test-presence.sh origin/develop all pass.

🤖 Generated with Claude Code

[CLAassistant]

All committers have signed the CLA.

[rbotte]

Ready for review whenever it fits your queue — posting status so triage is cheap:

• CLA signed ✅
• DCO sign-off on all commits ✅
• Still merges cleanly on current develop (checked today) ✅
• Full suite green locally, plus integration tests driving the real binary against a fake gh ✅

The only thing outstanding is the first-time-contributor workflow approval, which needs a maintainer click. Happy to rebase or split the PR if that helps.