Skip to content

Context-graph plugin: T0 activity-graph projection over ai_gateway_messages - #97

Merged
philcunliffe merged 2 commits into
masterfrom
context-graph-plugin
Jun 12, 2026
Merged

Context-graph plugin: T0 activity-graph projection over ai_gateway_messages#97
philcunliffe merged 2 commits into
masterfrom
context-graph-plugin

Conversation

@philcunliffe

@philcunliffephilcunliffe commented Jun 12, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

Was stacked on #96 (icebird 0.8.10 pin); #96 has merged and this branch is rebased onto master — the diff here is graph-only.

What

The first implemented slice of the context-graph direction designed in the cgproto LLP corpus (LLP 0006, "Projection pipeline"): a T0 deterministic projection that turns recorded gateway traffic into a queryable node/edge activity graph. No models — T0 is pure relational projection over data the gateway already structures. The in-repo design rationale is LLP 0023, with @ref annotations linking the code to its decisions.

@hypaware/context-graph plugin

  • Datasets node / edge — derived Iceberg tables fronted by the kernel cache, queryable via hyp query sql. Multi-partition queries go through a union source that keeps limit/offset at the engine level (never pushed into sub-scans).
  • hyp graph project [--dry-run] — 9 hand-authored contract rules over ai_gateway_messages materialize 5 node types (Session, App, Model, Tool, File) and 4 edge types (via, used_model, used, touched). Ids are content-addressed (SHA-256 of type + NUL-delimited natural key, digests pinned in tests); every row carries inline provenance (source_dataset, source_keys, projector, projector_version). Re-projection is idempotent: pre-write dedup filters already-committed ids (a real query/storage failure aborts the run rather than weakening the guarantee), so a re-run writes zero rows. Duplicate-row merges are order-independent: earliest first_seen wins, props union with per-key earliest-seen-wins and a value tie-break.
  • hyp graph compact [--dry-run] — merges duplicate rows that slip past pre-write dedup (concurrent projections, partial failures), including across source= partitions: each duplicate group folds into one row (same mergeRow projection uses) kept in the earliest-seen partition; affected partitions rewrite via the cache's generation swap (new table-<seq> dir, cursor repoint, .retired marker for the kernel's grace-period sweep). The swap is conditional: cursors are positively read (tryReadCursorSync, source-table layout required) and re-checked before repointing — any concurrent append or an unreadable cursor aborts the swap, removes the staged table, and reports the partition skipped (stderr + span attributes; unreadable cursors exit nonzero). Home partitions rewrite before copy-droppers and copies are only dropped once their merged row landed, so a partial run can leave duplicates but never lose rows or props.

Kernel-side enablers

  • AppendOptions.sortOrder — column-name sort declaration applied at table creation; icebird ≥ 0.8.9 then sorts every appended data file by the table's default sort order. Graph rewrites declare (node_type, node_id) / (edge_type, src_id, dst_id) so type scans and id lookups prune after the first compaction.
  • Cache compaction carries sort orders over — compactSourceTable's generation swap previously recreated the replacement table without the source table's default sort order, silently dropping it.
  • compactExportTable telemetry — the format-iceberg export rewrite now runs inside a sink.export.compact span recording reason, data-file counts, commit-verification outcome, staged-file reclamation, and error_kind on conflict/error paths.
  • @hypaware/context-graph added to the bundled-plugin allowlist.

Deliberately not here

The declarative contract→SQL compiler, T1/T2 enrichment, entity resolution, per-snapshot triggering, and a partition-level maintenance lock (the conditional swap's failure mode is "duplicate persists", never data loss — see LLP 0023#graph-compaction) are later slices.

Testing

  • npm test: 871 passing, including new context-graph-ids (pinned digests, delimiter collision-freedom), context-graph-contract (every rule's toRow: null-skips, file-tool gating, string/malformed-JSON tool_args, notebook_path fallback, timestamp normalization), context-graph-project (mergeRow determinism across all merge orders, firstSeenTime), context-graph-datasets (union limit/offset isolation), and expanded context-graph-maintenance (corrupt-cursor refusal, concurrent-write swap abort with staged-table cleanup, CLI skip reporting + exit codes)
  • Smokes: context_graph_projects_rows (projection counts, node_type breakdown, idempotent re-run, graph compact round-trip, plus graph.project/graph.compact span assertions per the log-driven-development policy) and core_boot_noop — ok
  • Typecheck and lint: clean

🤖 Generated with Claude Code

Base automatically changed from iceberg-export-compaction-and-smoke-fixes to masterJune 12, 2026 19:02
@philcunliffe

philcunliffe commented Jun 12, 2026

Copy link
Copy Markdown
ContributorAuthor

Dual-agent review — request_changes

  • Verdict:request_changes
  • Risk class:medium
  • Auto-merge advisory: 👎 thumbs down — verdict is request_changes; needs human-gated follow-up

Advisory only: no merge was attempted.

Risk capstone

Cross-reference: reviewer findings vs high-risk surfaces

SourceFinding (severity, evidence)Intersects
CodexCompaction loses rows appended during rewrite window (major, context-graph/src/maintenance.js:207)Concurrency surface: generation swap without lock/CAS; Risk 1
ClaudereadCursorSync used for destructive swap, violating PR #82 guard (major, context-graph/src/maintenance.js:208)Concurrency surface: cursor read; Risk 1
ClaudeunionSources double-applies LIMIT/OFFSET (major, context-graph/src/datasets.js:167)Cross-package usage: query-engine scan contract; Risk 2
ClaudePR head fails npm run typecheck (major, format-iceberg/src/maintenance.js:237)Risk 3: stacked-branch release safety
CodexDedup failure path swallowed, duplicates on storage error (major, context-graph/src/project.js:119)Risk 4; Concurrency surface: dedup→append non-atomicity
CodexmergeRow props conflict resolution non-deterministic (major, context-graph/src/project.js:153)Targets: mergeRow/projectGraph
Codex + ClaudeManifest omits graph compact command (minor, hypaware.plugin.json)Targets: plugin manifest / V1_BUNDLED_PLUGIN_ALLOWLIST exposure
ClaudecompactExportTable destructive path has no spans/logs (minor, format-iceberg/src/maintenance.js:1249)Direct callers: sink maintain --compact path
Claudeids.js literal NUL bytes make file binary (major, context-graph/src/ids.js:25)Targets: nodeId/edgeId determinism contract
Codex review

Fix Validations

Cache compaction preserves sort order

  • Status: correct
  • Evidence: src/core/cache/maintenance.js:471, src/core/cache/maintenance.js:491, src/core/cache/iceberg/store.js:110
  • Assessment: The compactor now recovers the existing table sort order and passes it into the replacement table create path. The store create path applies AppendOptions.sortOrder, so identity column sort orders survive generation swaps.

Findings

4) Concurrency, Ordering & State Safety

  • Severity: major
  • Confidence: high
  • Evidence: hypaware-core/plugins-workspace/context-graph/src/maintenance.js:207, hypaware-core/plugins-workspace/context-graph/src/maintenance.js:251, hypaware-core/plugins-workspace/context-graph/src/maintenance.js:267, src/core/cache/partition.js:108
  • Why it matters:graph compact can lose graph rows appended during the rewrite window: writers still append to the old cursor table while compaction scans it, then compaction repoints the cursor to a new table and retires the old one.
  • Suggested fix: Make the partition swap conditional on the cursor still matching the table read at the start, or take a partition-level maintenance lock; on mismatch, retry from the updated live table instead of retiring the old generation.

1) Behavioral Correctness

  • Severity: major
  • Confidence: high
  • Evidence: hypaware-core/plugins-workspace/context-graph/src/contract.js:81, hypaware-core/plugins-workspace/context-graph/src/project.js:47, hypaware-core/plugins-workspace/context-graph/src/project.js:153
  • Why it matters: Session projection is not deterministic when duplicate rows have conflicting props, because the source SELECT has no stable ordering and mergeRow lets later props overwrite earlier props.
  • Suggested fix: Define a deterministic conflict policy for props, such as earliest-row wins with stable tie-breakers, append-only value sets, or explicit ordered SQL plus tests for conflicting cwd / git_branch values.

5) Error Handling & Resilience

  • Severity: major
  • Confidence: medium
  • Evidence: hypaware-core/plugins-workspace/context-graph/src/project.js:119, hypaware-core/plugins-workspace/context-graph/src/project.js:131
  • Why it matters: A real query/storage failure during pre-write dedup is treated as “no committed partitions,” so graph project can append duplicates and report success while the cache is unreadable.
  • Suggested fix: Only swallow known missing-table / missing-dataset cases; rethrow other query errors so the command fails instead of weakening the idempotence guarantee.

2) Contract & Interface Fidelity

  • Severity: minor
  • Confidence: high
  • Evidence: hypaware-core/plugins-workspace/context-graph/hypaware.plugin.json:12, hypaware-core/plugins-workspace/context-graph/src/index.js:42
  • Why it matters: The plugin registers graph compact, but the manifest only declares graph project, so command metadata and plugin traceability are incomplete.
  • Suggested fix: Add { "name": "graph compact" } to contributes.commands.

No Finding

  1. Change Impact / Blast Radius; 6) Security Surface; 7) Resource Lifecycle & Cleanup; 8) Release Safety; 9) Test Evidence Quality; 10) Architectural Consistency; 11) Debuggability & Operability.

Evidence Bundle

  • Changed hot paths:projectGraph, dedupExisting, mergeRow, compactGraphTables, rewritePartition, cache appendRows / source-table flush, cache sort-order create path, sink maintain --compact.
  • Impacted callers: hypaware-core/plugins-workspace/context-graph/src/command.js:16, hypaware-core/plugins-workspace/context-graph/src/command.js:47, src/core/cache/storage.js:137, src/core/cache/storage.js:111, src/core/cli/core_commands.js:2333.
  • Impacted tests: hypaware-core/smoke/flows/context_graph_projects_rows.js:83, hypaware-core/smoke/flows/context_graph_projects_rows.js:102, test/plugins/context-graph-maintenance.test.js:44, test/plugins/context-graph-maintenance.test.js:105, test/core/sink-maintain-command.test.js:174, test/core/sink-maintain-command.test.js:193, test/core/sink-maintain-command.test.js:221, test/plugins/iceberg-maintenance.test.js:119, test/plugins/iceberg-maintenance.test.js:164.
  • Unresolved uncertainty: Static review only; I did not run tests. The compaction race depends on concurrent projection/flush while graph compact is running, but the current code has no guard against that interleaving.
Claude review

Claude review

ids.js embeds literal NUL bytes, making the file binary and the id-determinism invariant unpinned

  • Severity: major
  • Confidence: 95
  • Evidence: hypaware-core/plugins-workspace/context-graph/src/ids.js:25
  • Why it matters: The hash separators in nodeId/edgeId are raw 0x00 bytes inside template literals, so git renders the file as "Binary files differ" (unreviewable in the PR diff), grep-based tooling (including the in-repo /ref-check LLP validator) silently skips it, the invisible delimiters render as spaces in editors where any formatter normalization would silently change every graph id, and no unit test pins nodeId/edgeId output to catch that. (Flagged independently by 4 of 5 reviewers.)
  • Suggested fix: Replace the literal NUL bytes with the two-character \0 (or �) escape sequences — the runtime hash input is byte-identical, so no id migration is needed; document the delimiter choice in the comment and add a unit test pinning known nodeId/edgeId hex values.

PR head fails npm run typecheck — the fix commit exists but is not on this branch

  • Severity: major
  • Confidence: 97
  • Evidence: hypaware-core/plugins-workspace/format-iceberg/src/maintenance.js:237; test/core/sink-maintain-command.test.js:49
  • Why it matters:tsc --noEmit reports 5 errors, all in PR-added code (optional resolver.writer possibly undefined; memory BlobStore mock's getObject/listObjects return shapes don't satisfy the BlobStore interface); commit a183759 fixes exactly these but lives only on sibling branch iceberg-export-compaction-and-smoke-fixes and is not an ancestor of this PR's head.
  • Suggested fix: Cherry-pick or rebase a183759 into PR Context-graph plugin: T0 activity-graph projection over ai_gateway_messages #97 so the PR head passes npm run typecheck (runtime tests and smokes already pass: 847 tests, context_graph_projects_rows ok).

unionSources forwards LIMIT/OFFSET to every sub-source while claiming it didn't apply them

  • Severity: major
  • Confidence: 85
  • Evidence: hypaware-core/plugins-workspace/context-graph/src/datasets.js:167
  • Why it matters:scan(options) passes the full hints (including limit/offset) to each underlying iceberg source, which performs offset/limit pushdown per source, yet the union reports appliedLimitOffset: false, so the engine skips offset rows again on the concatenated stream — any query on node/edge with an OFFSET silently drops rows once the dataset has more than one committed partition.
  • Suggested fix: Forward { ...options, limit: undefined, offset: undefined } to the sub-source scans (keep where/columns/signal), or honor each sub-scan's appliedLimitOffset and only report true when coherent across the union.

Context-graph generation swap reads cursors with readCursorSync, violating the corrupt-cursor guard PR #82 established

  • Severity: major
  • Confidence: 82
  • Evidence: hypaware-core/plugins-workspace/context-graph/src/maintenance.js:208
  • Why it matters: Commit 085af71 (fix: make cache compaction memory-safe and reclaim orphaned generations #82) added tryReadCursorSync with an explicit contract (src/core/cache/partition.js:32) that callers taking destructive action must use it so a corrupt cursor.json is never mistaken for a synthetic default; the new rewritePartition/liveTableDir use readCursorSync, so a corrupt cursor synthesizes {epoch: 0} with no tableDir, making hyp graph compact rewrite from the stale pre-compaction table dir, repoint the cursor (resetting epoch and dropping retention), and leave the real live generation cursor-unreferenced — which the fix: make cache compaction memory-safe and reclaim orphaned generations #82 orphan sweep then reclaims after its grace period, losing every graph row projected since the last compaction.
  • Suggested fix: In rewritePartition and liveTableDir, use tryReadCursorSync and skip the partition (surfacing an error in the compact report) when it returns null; only rewrite when cursor.layout === 'source-table', and preserve epoch/retention only from a positively-read cursor.

New context-graph subsystem ships with no in-repo LLP document

  • Severity: major
  • Confidence: 85
  • Evidence: hypaware-core/plugins-workspace/context-graph/src/index.js:29
  • Why it matters: CLAUDE.md requires design rationale in numbered LLP docs landed with the code; this PR adds ~1,200 lines of non-obvious design (T0 contract semantics, content-addressed ids, inline-provenance simplification, graph-semantic compaction) whose rationale lives only in an external repo, leaving the in-code pointer "see the design notes" dangling and the LLP 0000 subsystem map stale — a break from precedent (the ai-gateway plugin has LLP 0016).
  • Suggested fix: Land an LLP doc covering the context-graph T0 projection decisions, add it to LLP 0000's doc map, and convert the "see the design notes" and provenance-simplification comments into @ref LLP NNNN#anchor annotations.

Manifest omits graph compact from contributes.commands while activate() registers it

  • Severity: minor
  • Confidence: 92
  • Evidence: hypaware-core/plugins-workspace/context-graph/hypaware.plugin.json:13
  • Why it matters: The kernel's own contract checker flags it (hyp plugin doctor emits contribution_undeclared for command 'graph compact'), and the manifest's contributes feeds help/discovery surfaces, so the new command is invisible there.
  • Suggested fix: Add { "name": "graph compact" } to contributes.commands in hypaware.plugin.json.

No traditional tests for the new projection transform logic (contract.js / ids.js)

  • Severity: minor
  • Confidence: 85
  • Evidence: hypaware-core/plugins-workspace/context-graph/src/contract.js:311
  • Why it matters: CLAUDE.md requires traditional tests for deterministic transforms (sibling projectors have them), yet the toRow mappers, the string-JSON/malformed-JSON/notebook_path branches of filePathFrom, and firstSeenTime normalization are covered only by one hermetic smoke happy path.
  • Suggested fix: Add test/plugins/context-graph-contract.test.js exercising each rule's toRow (null-skip, file-tool vs non-file-tool, tool_args as JSON string and malformed JSON, notebook_path fallback) plus firstSeenTime/mergeRow edge cases.

New smoke flow asserts no internal telemetry signal, contrary to the log-driven-development policy

  • Severity: minor
  • Confidence: 85
  • Evidence: hypaware-core/smoke/flows/context_graph_projects_rows.js:79
  • Why it matters: CLAUDE.md says a smoke should assert the user-visible result and the internal signal proving the intended path ran; projectGraph emits a graph.project span and the harness exposes expect.traces(), but this flow asserts only SQL counts and CLI stdout, so it would still pass if span emission silently broke.
  • Suggested fix: Assert via expect.traces() that a graph.project span exists with nodes_written === 7, edges_written === 6, and status: 'ok', mirroring gateway_codex_capture's trace assertions.

compactExportTable emits no structured telemetry for a high-stakes new workflow

  • Severity: minor
  • Confidence: 82
  • Evidence: hypaware-core/plugins-workspace/format-iceberg/src/maintenance.js:1249
  • Why it matters: The new rewrite path makes corruption-relevant decisions (landed/lost/unknown commit verification, best-effort deletes of staged files) with zero spans or logs, while sibling new code in this same PR (compactGraphTables, projectGraph) and the existing export path use withSpan with component/operation/status/error_kind attributes.
  • Suggested fix: Wrap compactExportTable in withSpan (e.g. sink.export.compact) recording reason, data_files_before/after, commit-verification outcome, and error_kind on conflict/error paths.

Reports: /Users/phil/workspace/hypaware/.git/worktrees/dual-review-pr-97/dual-review/pr-97

philcunliffe added a commit that referenced this pull request Jun 12, 2026
@philcunliffe@claude
Address dual-review findings for PR#97
6303fb6
- maintenance.js: positively read cursors (tryReadCursorSync) and require
source-table layout before any rewrite; make the generation swap
conditional on the cursor matching the scan-time read so rows appended
during the rewrite window are never lost — on mismatch the staged table
is removed and the partition is reported skipped, never retired. Home
partitions rewrite before copy-droppers and copies are only dropped
once the merged row landed. Skips surface in the report, on stderr,
and as span attributes; unreadable cursors exit nonzero.
- project.js: dedup query failures now abort the projection unless the
dataset is genuinely missing; mergeRow resolves props conflicts
deterministically (per-key earliest-seen wins, value tie-break) so
merge order can never change the result.
- datasets.js: unionSources no longer forwards limit/offset to
sub-sources (offsets were applied twice on multi-partition datasets).
- ids.js: literal NUL bytes replaced with \0 escapes (byte-identical
hash input, file is plain text again); delimiter choice documented.
- format-iceberg maintenance.js: compactExportTable wrapped in a
sink.export.compact span recording reason, file counts, commit
verification outcome, staged-file reclamation, and error_kind.
- hypaware.plugin.json: declare 'graph compact' in contributes.commands.
- smoke: context_graph_projects_rows now asserts the graph.project /
graph.compact spans alongside the SQL counts.
- tests: pinned nodeId/edgeId digests, contract toRow rules, mergeRow
determinism, union limit/offset, corrupt-cursor and concurrent-write
compaction safety, CLI skip reporting.
- LLP 0023 documents the context-graph T0 projection decisions; doc map
updated and design comments converted to @ref annotations.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@philcunliffe

philcunliffe commented Jun 12, 2026

Copy link
Copy Markdown
ContributorAuthor

Review findings addressed (6303fb6, db624af)

All findings from the dual-agent review are addressed on this branch.

Finding (source)Resolution
Compaction loses rows appended during rewrite window (Codex, major)Generation swap is now conditional on the cursor matching the scan-time read; on mismatch the staged table is removed and the partition is skipped + reported — never retired. Home partitions rewrite first; a duplicate's copies are only dropped after its merged row landed. Chose skip-and-report over retry: reruns are cheap and the failure mode is "duplicate persists", never data loss (rationale in LLP 0023#graph-compaction). Tests: concurrent-write abort, staged-table cleanup, row preservation.
readCursorSync for destructive swap violates #82 guard (Claude, major)All compaction reads use tryReadCursorSync and require layout: 'source-table'; an unreadable cursor skips the partition (report + stderr + nonzero exit), epoch/retention only preserved from a positively-read cursor. Test: corrupt-cursor refusal leaves cursor, table, and rows untouched.
unionSources double-applies LIMIT/OFFSET (Claude, major)Sub-scans receive limit: undefined, offset: undefined; where/columns/signal still forwarded. Regression test added.
PR head fails npm run typecheck (Claude, major)a183759 cherry-picked (db624af); typecheck clean on this head.
Dedup failure path swallowed (Codex, major)Only a genuinely missing dataset is benign; any other query/storage error aborts graph project instead of appending duplicates.
mergeRow props conflict non-deterministic (Codex, major)Deterministic policy: per-key earliest-seen-wins with value tie-break on equal/unknown timestamps; order-independent (verified across all 6 merge orders, conflicting cwd/git_branch cases included). Shared by projection and compaction.
ids.js literal NUL bytes (Claude, major)Replaced with \0 escapes — hash input byte-identical, no id migration; delimiter documented; known digests pinned in context-graph-ids.test.js.
No in-repo LLP doc (Claude, major)LLP 0023 covers the T0 contract, content-addressed ids, inline provenance, merge policy, and compaction safety; doc map updated (incl. the missing 0022 row); 9 @ref annotations replace the "see the design notes" pointers (ref-check clean).
Manifest omits graph compact (both, minor)Declared in contributes.commands.
compactExportTable no telemetry (Claude, minor)Wrapped in sink.export.compact span: reason, data-file counts, commit-verification outcome, staged-file reclamation, error_kind on conflict/error.
No traditional tests for contract.js (Claude, minor)context-graph-contract.test.js: every rule's toRow (null-skips, file-tool gating, string + malformed JSON tool_args, notebook_path fallback), timestamp normalization, numeric keys.
Smoke asserts no internal signal (Claude, minor)context_graph_projects_rows now asserts graph.project spans (7 nodes / 6 edges written, idempotent re-run wrote 0) and a graph.compact span with nothing skipped.

Verification on this head: 871 tests passing, typecheck + lint clean, context_graph_projects_rows and core_boot_noop smokes ok.

Known residual: a small TOCTOU window remains between the cursor re-read and cursor write; closing it needs a partition-level maintenance lock, deliberately deferred in LLP 0023 (manual command, cheap reruns, no-data-loss failure mode).

🤖 Generated with Claude Code

philcunliffeand others added 2 commits June 12, 2026 12:58
@philcunliffe@claude
Context-graph plugin: T0 activity-graph projection over ai_gateway_me…
662c2f5
…ssages
The first slice of the context-graph direction (designed in the cgproto
LLP corpus, LLP 0006 "Projection pipeline"): a deterministic T0
projection that turns recorded gateway traffic into a queryable
node/edge activity graph. No models involved — T0 is pure relational
projection over data the gateway already structures.
The @hypaware/context-graph plugin registers:
- Datasets `node` and `edge` — derived Iceberg tables fronted by the
kernel cache, queryable via `hyp query sql`.
- `hyp graph project [--dry-run]` — runs 9 hand-authored contract rules
over ai_gateway_messages, materializing 5 node types (Session, App,
Model, Tool, File) and 4 edge types (via, used_model, used, touched).
Ids are content-addressed (SHA-256 of type + natural key) and every
row carries inline provenance (source_dataset, source_keys, projector,
projector_version), so re-projection is idempotent: pre-write dedup
filters ids already committed, and a re-run writes zero rows.
- `hyp graph compact [--dry-run]` — merges duplicate node/edge rows
that slip past pre-write dedup (concurrent projections, partial
failures), possibly across `source=` partitions: each duplicate group
folds into one row (earliest first_seen, unioned props — the same
mergeRow projection uses) kept in the earliest-seen partition, and
affected partitions are rewritten via the cache's generation swap
(new table dir, cursor repoint, .retired marker for the kernel sweep).
Kernel-side enablers (shared, not graph-specific in shape):
- `AppendOptions.sortOrder`: a column-name sort declaration applied at
table creation; icebird >= 0.8.9 then sorts every appended data file
by the table's default sort order. Graph rewrites declare
(node_type, node_id) / (edge_type, src_id, dst_id) so type scans and
id lookups prune after the first compaction.
- Cache compaction (compactSourceTable) now carries an existing default
sort order over to its replacement table — previously the generation
swap silently dropped it.
- `@hypaware/context-graph` added to the bundled-plugin allowlist.
The contract is hand-authored per rule for now; the declarative
contract -> SQL compiler is a later slice (cgproto LLP 0006
"projection contracts compile to SQL").
Covered by test/plugins/context-graph-maintenance.test.js (cross-
partition dedup merge, sort-order declaration on rewrite, idempotence)
and the context_graph_projects_rows hermetic smoke (projection counts,
node_type breakdown, idempotent re-run, clean compaction round-trip
through real plugin registration).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@philcunliffe@claude
Address dual-review findings for PR#97
4ef828d
- maintenance.js: positively read cursors (tryReadCursorSync) and require
source-table layout before any rewrite; make the generation swap
conditional on the cursor matching the scan-time read so rows appended
during the rewrite window are never lost — on mismatch the staged table
is removed and the partition is reported skipped, never retired. Home
partitions rewrite before copy-droppers and copies are only dropped
once the merged row landed. Skips surface in the report, on stderr,
and as span attributes; unreadable cursors exit nonzero.
- project.js: dedup query failures now abort the projection unless the
dataset is genuinely missing; mergeRow resolves props conflicts
deterministically (per-key earliest-seen wins, value tie-break) so
merge order can never change the result.
- datasets.js: unionSources no longer forwards limit/offset to
sub-sources (offsets were applied twice on multi-partition datasets).
- ids.js: literal NUL bytes replaced with \0 escapes (byte-identical
hash input, file is plain text again); delimiter choice documented.
- format-iceberg maintenance.js: compactExportTable wrapped in a
sink.export.compact span recording reason, file counts, commit
verification outcome, staged-file reclamation, and error_kind.
- hypaware.plugin.json: declare 'graph compact' in contributes.commands.
- smoke: context_graph_projects_rows now asserts the graph.project /
graph.compact spans alongside the SQL counts.
- tests: pinned nodeId/edgeId digests, contract toRow rules, mergeRow
determinism, union limit/offset, corrupt-cursor and concurrent-write
compaction safety, CLI skip reporting.
- LLP 0023 documents the context-graph T0 projection decisions; doc map
updated and design comments converted to @ref annotations.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@philcunliffe
philcunliffe merged commit 322aaed into masterJun 12, 2026
6 checks passed
@philcunliffe
philcunliffe deleted the context-graph-plugin branch June 12, 2026 20:01
@philcunliffephilcunliffe mentioned this pull request Jun 12, 2026
Merged
philcunliffe added a commit that referenced this pull request Jun 12, 2026
@philcunliffe@claude
Merge origin/master; renumber remote-config spec to LLP 0024
dcc63a5
Master's context-graph work (#97) took the 0023 slot, so the
remote-config spec moves to llp/0024-remote-config-join-flow.spec.md.
All @ref annotations and prose references updated; context-graph's own
LLP 0023 references are untouched.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
philcunliffe added a commit that referenced this pull request Jun 13, 2026
@philcunliffe@claude
Vector search plugins: @hypaware/vector-search + @hypaware/embedder-o…
a6fca1f
…penai (LLP 0024) (#99)
* Add @hypaware/vector-search + @hypaware/embedder-openai plugins (LLP 0024)
Vector similarity search over cached datasets via hypvector, as designed
in LLP 0024:
- @hypaware/embedder-openai provides hypaware.embedder: an
OpenAI-compatible POST /v1/embeddings client with configurable
base_url (covers OpenAI, proxies, and localhost servers), model, and
key-env-var name. The key resolves from the environment at call time
and never reaches logs; requests without the env var set go out
unauthenticated so local servers need zero credential config.
- @hypaware/vector-search provides hypaware.vector-search and requires
hypaware.embedder. Indexes are declared in config; artifacts are one
hypvector parquet shard per cache partition (plus a JSON sidecar
carrying model + source row count) under the plugin state dir.
Freshness rides the cache-maintenance pattern: a daemon refresh
source with interval + max_tick_ms + max_rows_per_tick budgets, and
search-time auto refresh with an upfront row estimate (--no-refresh
opts out; a model mismatch there is a hard error). Orphaned shards
sweep when retention evicts their partition. CLI: hyp vector
search / hyp vector status through the command registry, formatted
by the intrinsic formatter.
- hypvector 0.1.1 lands as a root optionalDependency mirroring
hyparquet-writer (its hyparquet pin matches ours exactly; the
existing hyparquet-writer override covers its 0.15.3 range). The
plugin lazy-imports it and degrades gracefully when absent.
- Both plugins are bundled but excluded from default activation:
enabling an API-backed embedder is the explicit opt-in that lets
captured text leave the machine.
- Kernel types gain EmbedderCapability / VectorSearchCapability and
capability registry doc entries.
LLP changes landed with the code: LLP 0024 (new, Active) records the
design and resolves the cost-visibility open question as the per-tick
row budget; LLP 0000 gains the subsystem map row; LLP 0003 gains the
sharpened "query is intrinsic means the SQL/dataset surface" wording
this decision rests on. (The design session numbered this doc 0023;
it lands as 0024 because 0022/0023 were taken on master by the
iceberg-partitioning and context-graph specs.)
Tests: config validation, embedder client (auth/batching/error
mapping/secret hygiene), shard staleness + model-mismatch + orphan
classification, top-K merge, manifest validation. Hermetic smoke
vector_search_local_fixture drives a localhost fake embedder through
populate -> activate -> search-time build -> staleness -> timer
rebuild -> --no-refresh search, asserting results and the telemetry
that proves each path ran (and that neither key material nor indexed
text leaks into telemetry).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix cli_bundled_plugins_activated smoke for context-graph allowlisting
The config-profile boot's plugins_skipped counts allowlist plugins the
flow's config does not name. #97 added @hypaware/context-graph to
V1_BUNDLED_PLUGIN_ALLOWLIST without updating this assertion, so the
smoke has expected 3 (and observed 4) since then. Update to 4 and
document what the number is composed of, so the next allowlist
addition fails with a self-explanatory message.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Address dual-review findings on the vector-search plugins
Behavioral fixes, all classified major by the review:
- Shard file names append a short hash of the canonical partition JSON:
the sanitized label alone was lossy (source=a/b vs source=a_b, and
values containing , or = could mimic another partition's entry list),
so distinct partitions could overwrite each other's shards.
- Shard sidecars now record id_column, and computeShardStates validates
the full identity (index, dataset, column, id_column, exact
partition) as stale_config — reusing an index name over a different
dataset/column can no longer classify the old vectors fresh.
- Dimension drift is staleness (stale_dimension), not a brick: the
embedder capability exposes its configured dimensions, search embeds
the query before refreshing so the query's dimension feeds the
staleness check, and auto-refresh re-embeds instead of hard-failing
on shards it just declared fresh. The hard error (with mode-correct
wording) is reserved for --no-refresh and non-deterministic embedders.
- Same-shard builds serialize through an in-process lock and temp files
carry pid + UUID, so concurrent search-time and daemon refresh can
no longer race on shared temp paths; the final rename stays atomic.
- Provider error bodies are no longer copied into errors/logs — a
provider or proxy may echo indexed text or credentials in its error
detail; failures surface as status + endpoint + error kind only.
Minor findings:
- Inline import('...') types replaced with @import / top-level
import type per the repo style.
- New tests: refresh budget enforcement (deadline + row budget, skip
accounting, orphan sweep under exhausted budget), content-hash dedup
and id_column keying, --no-refresh model/dimension mismatch hard
errors, and shard-name collision regressions.
- LLP 0024 corrected to match the implementation (sidecar carries
model/dimension, declarations carry no model field, layout is keyed
by index name) and extended with the decisions above.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@philcunliffe