Skip to content

Local-only directory selection: enrollment picker + export-seam withholding (LLP 0069/0080/0081) - #270

Merged
philcunliffe merged 22 commits into
masterfrom
integration/local-only-dir-selection
Jul 7, 2026
Merged

Local-only directory selection: enrollment picker + export-seam withholding (LLP 0069/0080/0081)#270
philcunliffe merged 22 commits into
masterfrom
integration/local-only-dir-selection

Conversation

@philcunliffe

@philcunliffephilcunliffe commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Implements the local-only-dir-selection change set — LLP 0080 (design), LLP 0081 (plan); covers request LLP 0069 and decisions LLP 0070/0071/0072.

Lets a user mark directories as local-only — recorded locally but never forwarded to a central sink — via a login-time enrollment picker and a durable hyp ignore --local-only CLI, enforced at the export seam (drop-but-advance) so excluded rows stay locally queryable while never leaving the machine. Closes a real format-iceberg leak where captured directories could be forwarded before the user could withhold them.

Tasks (all merged, verified as real ancestors of the integration branch):

  • T1 machine-local list store (src/core/usage-policy/local_only.js)
  • T2 resolver gains the local-only list as a second source (most-restrictive wins)
  • T3 export-seam enforcement — drop-but-advance (readRowsSince + sink/incremental consumers)
  • T4 format-iceberg reroute off the filtered seam
  • T5 enumeration + login-time picker (src/core/commands/local_only.js)
  • T6 durable CLI authoring (hyp ignore/unignore --local-only, --check)
  • T7 login wiring (picker before enrollCentralSink)
  • T8hyp status surface (withholding count)
  • T9 hermetic smoke local_only_export_withhold

Generated-by: neutral

Change-Set: local-only-dir-selection

philcunliffeand others added 20 commits July 6, 2026 14:56
…vers 0069)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add src/core/usage-policy/local_only.js: localOnlyListPath(stateDir),
readLocalOnlyDirs (missing => [], corrupt => throws
LocalOnlyListUnreadableError with error_kind
'local_only_list_unreadable', the LLP 0080 fail-safe), and
writeLocalOnlyDirs (normalize/dedupe/sort absolute paths, mkdir -p,
atomic tmp+rename write). Exported from the usage-policy barrel.
@ref LLP 0071 [implements]
@ref LLP 0080#fail-safe [implements]
Task-Id: T1
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
collectHypAwareStatus best-effort reads the machine-local local-only
list and reports usagePolicy.localOnlyDirCount; a corrupt list
degrades overall with a local_only_list_unreadable diagnostic instead
of silently showing zero. Text renders "local-only: withholding N
directories from forwarding (recorded locally)" when N > 0; JSON
carries the count under usage_policy.
Task-Id: T8
createUsagePolicyResolver accepts an optional localOnlyListPath: resolve(cwd)
now merges the existing .hypignore ancestor walk with segment-aware
equal-or-descendant membership in the machine-local local-only list, and
returns the most-restrictive class (ignore > local-only > full). A
list-governed result names the list file as governedBy/declared. The parsed
list is memoized with the existing 5s TTL, independent of per-cwd caching, and
a corrupt list throws the same LocalOnlyListUnreadableError the store uses
rather than silently resolving to "no exclusions".
format.js's IMPLEMENTED set gains 'local-only', so a committable .hypignore
declaring it resolves to that class instead of clamping to ignore
(LLP 0070 #resolver). Updates the two existing tests that exercised
'local-only' as a stand-in for an unimplemented token to use a still-
unimplemented token instead, and adds coverage for the new merge behavior.
Task-Id: T2
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds `src/core/commands/local_only.js`: `listCapturedDirectories`
(LLP 0069 #enumerate) runs one best-effort `executeQuerySql` GROUP BY
over `ai_gateway_messages` and collapses to one candidate per distinct
`cwd`, most-recently-active first. `runLocalOnlyPicker` (LLP 0072) is
the stdin+stderr TTY-gated checkbox multi-select (reusing the existing
`src/core/cli/tui` `multiselect`, rendered on stderr) that pre-checks
directories already on the machine-local list, preserves entries not
offered this round, persists via `writeLocalOnlyDirs`, prints the
never-silent "withholding N..." line, and emits
`local_only.picker_result` telemetry. Cancellation and a failed/empty
enumeration both proceed with zero *new* exclusions without touching
whatever was already on disk.
Task-Id: T5
runBrowserLogin now calls runLocalOnlyPicker once the login-minted
gateway credential is seeded, strictly before enrollCentralSink
provisions or re-seeds a central sink - covering both the fresh-enroll
and re-login forks. --no-forward and query-only (no gateway) logins
never reach it, and a non-cancellation picker error (e.g. a corrupt
existing list) is caught and warned rather than failing enrollment.
Task-Id: T7
`hyp ignore --local-only [path]` / `hyp unignore --local-only [path]` add
and remove entries in the machine-local `local-only` list (LLP 0071) as an
alternative to the committable `.hypignore` dotfile, giving a non-login
authoring path for directories excluded from forwarding (LLP 0072 #cli).
Both verbs reuse the shared usage-policy resolver and its equal-or-ancestor
matching rather than duplicating path logic (R8): add is a no-op when the
target is already governed (by an existing list entry or a stricter
`.hypignore`), and remove drops every governing list entry. `hyp ignore
--check` now constructs the resolver with the list wired in, so it reports
`local-only` alongside `ignore`, names whichever source (dotfile or list
file) governs, and computes the residual cached-row count for either class.
Task-Id: T6
Withhold `local-only` rows at the shared export read (`storage.readRowsSince`)
while still advancing the per-(sink, partition) watermark across them, so a
directory the user marked local-only is recorded in the cache and stays locally
queryable but never forwarded — and its tail is durably passed rather than
re-scanned each tick.
- storage.js: `createQueryStorageService` gains an optional `usagePolicyResolver`;
`readRowsSince` yields `{ after, dropped: true }` (no payload) for a row whose
non-empty `cwd` resolves to class != `full`, passes cwd-less rows untouched,
and lets a corrupt-list throw fail the partition read (fail-safe). Emits the
`usage_policy.export_drop` aggregate with hashed cwds.
- activation.js: every kernel boot defaults the resolver from the machine-local
list at `localOnlyListPath(<stateDir>)`.
- kernel contract: `readRowsSince` element type is now a discriminated union
(row-carrying | drop-only).
- central sink: advance `lastAfter` across drops, count `droppedRowCount`, widen
the end-of-partition watermark gate to `shipped>0 || dropped>0`
(ship-first/advance-second preserved; a failed chunk still never checkpoints).
- openIncrementalRows: skip drops in `rows()`, advance `lastAfter` incl. trailing
drops, expose `droppedRowCount`, peek past leading drops for `empty`; s3 and
local-fs write the watermark on an empty-but-dropped tick.
Tests: cache-but-never-forward for a local-only cwd, drop-only tick checkpoints,
un-excluded dir not re-sent past a checkpoint, failed chunk never checkpoints,
cwd-less datasets unaffected, corrupt list fails the tick with the watermark
untouched.
Task-Id: T3
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…0070/0081 T4)
The Iceberg table-format sink read committed rows via `storage.readRows`, a
plain full scan that never went through `storage.readRowsSince` — the shared,
usage-policy-filtered export seam every other sink (central forward, blob/
local-fs incremental) already reads through. That made it the last export
path off the filtered seam: a directory the user marked `local-only` could
still leak into a committed Iceberg snapshot even though no other sink would
ever forward it.
`openRows` now reads through `readRowsSince` instead: `since` omitted (full
scan) with `includeLegacy: true`, skipping drop-only entries and discarding
the `after` continuation `readRowsSince` yields — Iceberg commits are
snapshot/marker-based (`state.js` load/writeMarker), not cursor-based, so
there is no watermark here to advance.
Tests: a committed snapshot excludes rows from a directory on the
machine-local `local-only` list; a partition with no list configured commits
every row unaffected, exactly as before.
Task-Id: T4
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…81 T9)
Adds `hyp smoke local_only_export_withhold`, closing the local-only-dir-
selection change set: seeds cache rows for two cwds into a fixture dataset,
marks one directory local-only via the durable CLI (`hyp ignore
--local-only`), then drives the REAL central forward sink through the REAL
sink driver.
Asserts end to end: the clean cwd's rows reach the fake central server; the
excluded cwd's rows never do, but stay locally queryable via
`executeQuerySql`; the sink watermark advances across the withheld rows (a
second forced tick ships zero bytes and POSTs nothing new); `hyp status`
(text and --json) reports the withholding count; and `usage_policy.export_
drop` fires exactly once, naming the dropped-row and distinct-cwd counts.
Registers the new flow in the README release checklist.
Task-Id: T9
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…nly-dir-selection
# Conflicts:
#	src/core/cli/remote_commands.js
@philcunliffe

Copy link
Copy Markdown
ContributorAuthor

🔍 neutral review (Codex + 4 Claude lenses) — request_changes

Round 1, head 6fd00b1 (after the union-merge with #259). This is a thorough review of the whole 9-task change set — export-seam enforcement, resolver + list store, CLI/picker/login, and status/smoke/LLP — plus Codex as the independent family. The core holds: the privacy-critical drop-but-advance invariant was traced end-to-end (dropped payload never emitted; watermark advances past drops without re-send or stall; a failed forward chunk never checkpoints; a corrupt list fails the tick with the watermark untouched — fail-safe, not fail-open), the resolver's segment-aware matching is correct (/a/bc/a/b) with most-restrictive precedence and atomic list writes, the picker can't break enrollment (cancel persists nothing), the #259 union-merge is correct (deps seam fully unioned, order login → seed → picker → enroll → attach-wait, enrollCentralSinkenroll applied), and the hermetic smoke genuinely proves the invariant (not tautological). One substantive finding to fix before merge, plus one nit:

Major — readRowsSince({ columns }) can bypass the privacy filter (src/core/cache/storage.js)

The export seam decides withholding from row.cwd. If a caller passes a columns projection that omits cwd, the projected row has no cwd, the filter treats it as cwd-less, and a local-only row would be forwarded. No current in-repo export caller passes such a projection — central, incremental, and format-iceberg all omit columns, so cwd is present and withholding works (verified by all reviewers, so there's no active leak today). But readRowsSince's columns option is public, so the seam's privacy guarantee is conditional on every present and future caller (in-repo or external plugin) remembering to include cwd. For a control meant to be the guaranteed choke point (LLP 0070), that conditionality is a real robustness hole.
Fix: when a usagePolicyResolver is configured, force cwd into the internal scan projection regardless of the caller's columns, evaluate the policy before yielding, then omit cwd from the yielded row if the caller did not request it — so withholding is unconditional. Add a regression test: readRowsSince({ columns: ['id'] }) over a local-only row is still dropped.
(Codex rated this a blocker; downgraded to major here because no current caller triggers it — but it's a privacy seam, so it's fixed rather than deferred.)

Nit — inline import('...') type (test/core/local-only-command.test.js:151)

@param {import('node:stream').PassThrough} stdin violates CLAUDE.md's no-inline-import-types rule; the file already value-imports PassThrough (bare name in scope). CI won't catch it. Use @param {PassThrough}.

Verdict: request_changes — neutral is applying both fixes and will re-review the new head (round 2).

Automated neutral review (Codex = independent second family; 4 parallel Claude review lenses).

philcunliffeand others added 2 commits July 7, 2026 11:44
…'t bypass withholding + fix inline import type (PR #270 review round 1)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…nly-dir-selection
# Conflicts:
#	src/core/commands/clients.js
@philcunliffe

Copy link
Copy Markdown
ContributorAuthor

🔍 neutral review round 2 — approve · ready for merge

Round-1 findings fixed and verified on head e463998:

  • Major (a columns projection could bypass the privacy filter) ✓ — readRowsSince now forces cwd into the internal scan whenever a usage-policy resolver is configured, evaluates the policy, then strips cwd from the yielded row if the caller's columns didn't ask for it. Withholding is now unconditional — it no longer rides on callers remembering to include cwd. The default (no-columns) path is byte-unchanged; the drop branch and corrupt-list throw are preserved. New regression test: readRowsSince({ columns: ['id'] }) over a local-only row still drops.
  • Nit ✓ — the inline import('...') type is now the bare name.

Also merged up to current master (which landed #256/#266/#271/#273): the only conflict was the clients.jsimport block, unioned so #271's manual-attach disk-probe and #270's local-only CLI coexist (independent functions). Both suites green — attach 5/5, local-only 73/73; full run 2086 pass / 0 fail. CI green, mergeable.

Verdict: approve. Review rounds complete (N=2). Flipping to ready and holding for a human to merge — neutral does not merge.

Automated neutral review.

@philcunliffe
philcunliffe marked this pull request as ready for review July 7, 2026 18:51
@philcunliffe
philcunliffe merged commit 0bafd5f into masterJul 7, 2026
4 checks passed
@philcunliffe
philcunliffe deleted the integration/local-only-dir-selection branch July 7, 2026 18:56
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@philcunliffe