context-graph-enrich: scan signal not plumbing (row-selection filter)

[philcunliffe]

Why

T1 propose scanned every part of ai_gateway_messages with no content filter, concatenating them into the per-session text handed to Haiku; T2 then re-deref'd the same source by message id for the curator. Measured on a real ~634k-part corpus:

Category	Share of fed text
tool_result (raw file/command output)	60.4%	the real cost/noise lever
main-thread conversational text	37.3%	the signal
sub-agent (is_sidechain) rows	11.8%	~80% is already tool_result
thinking (part_type='reasoning')	0.04%	empty in the proxy — already free

On top of that, ~33% of all scanned rows carry zero content_text (every tool_call part + the signature-only thinking parts). They were already dropped from the model text but still consumed the 200-row/tick budget and crowded the per-session char cap.

What

A shared content filter (contentFilterClauses) applied in both the T1 scan (buildProposeQuery) and the T2 source deref (safeDeref), behind two source-config knobs:

• require_text (default true) — drop rows whose text column is null/empty. This already excludes "thinking tokens" (empty in the proxy), so they need no dedicated filter.
• exclude_part_types (default ['tool_result']) — drop whole part kinds; an explicit [] disables it. part_type_column (default 'part_type') names the column it reads.

Filtering the T2 deref is the non-obvious bit: it reads source by message_id, so a kept text part could otherwise re-admit its message's tool_result siblings into the expensive Opus call. The filter keeps T2 consistent with T1.

Watermark stays safe: filtered-out rows carry nothing to process, and the cursor advances over the rows the tick did see, so the next ts >= cursor scan naturally starts past them. The filter is applied before LIMIT, so each tick's row budget is spent on signal.

Deferred (documented, not dropped)

Sub-agent (is_sidechain) exclusion — the original ask — is left out and written up in LLP 0028#row-selection: only ~2.3% incremental once tool_result is gone, a leaky signal (NULL on 37% of rows; only 150/3940 conversations flagged), and a recall policy call rather than a pure cost lever. exclude_part_types can't express it (row flag, not part type); a sidechain predicate is noted as a future knob.

Notes / caveat

The per-session 12k-char cap bounds the absolute token bill, so the biggest win is signal density inside that window + freed row budget; pure $ savings land on under-cap ticks.

Test plan

• 68 tests in the enrich suite pass (new coverage for contentFilterClauses, the config knobs incl. explicit-[] and validation errors, and buildProposeQuery's WHERE)
• tsc -p tsconfig.json --noEmit clean
• node scripts/check-syntax.js clean

🤖 Generated with Claude Code

[philcunliffe]

Dual-agent review — request_changes

• Verdict:request_changes
• Risk class:medium
• Auto-merge advisory: 👎 thumbs down — verdict is request_changes; needs human-gated follow-up

Advisory only: no merge was attempted.

Risk capstone

Cross-reference: reviewer findings vs high-risk surfaces

Source	Finding (severity, evidence)	Intersects
Codex	Contract & Interface Fidelity — default part-type filter applied to custom sources (major, config.js:85/92/97, sql.js:106)	Config field chain; Risks #1
Claude	safeDeref filtered WHERE untested (minor, curate.js:267)	Direct callers (safeDeref); Risks #2

Codex review

Fix Validations

T1 propose scanned empty/tool-result rows

• Status: correct
• Evidence: hypaware-core/plugins-workspace/context-graph-enrich/src/propose.js:113, hypaware-core/plugins-workspace/context-graph-enrich/src/sql.js:103
• Assessment:buildProposeQuery now ANDs the shared content predicates into the source scan before LIMIT. Existing groupSourceRows already skipped empty text after fetch at propose.js:145, but that did not solve row-budget crowding.

T2 source deref re-admitted excluded sibling rows

• Status: correct
• Evidence: hypaware-core/plugins-workspace/context-graph-enrich/src/curate.js:267, hypaware-core/plugins-workspace/context-graph-enrich/src/curate.js:270
• Assessment:safeDeref applies the same contentFilterClauses(cfg) while dereferencing by id_column, which closes the message-id sibling leak described in the PR.

Findings

2) Contract & Interface Fidelity

• Severity: major
• Confidence: high
• Evidence: hypaware-core/plugins-workspace/context-graph-enrich/src/config.js:85, hypaware-core/plugins-workspace/context-graph-enrich/src/config.js:92, hypaware-core/plugins-workspace/context-graph-enrich/src/config.js:97, hypaware-core/plugins-workspace/context-graph-enrich/src/sql.js:106
• Why it matters: Existing custom source_dataset configs that only satisfy the previous source contract now get a default part_type NOT IN (...) predicate against a column they may not have, turning valid configs into failing propose queries.
• Suggested fix: Either make the default part-type exclusion conditional on the default ai_gateway_messages source, or validate custom sources/configs so users must explicitly provide part_type_column or exclude_part_types: [] before activation.

No Finding

1. Behavioral Correctness
2. Change Impact / Blast Radius
3. Concurrency, Ordering & State Safety
4. Error Handling & Resilience
5. Security Surface
6. Resource Lifecycle & Cleanup
7. Release Safety
8. Test Evidence Quality
9. Architectural Consistency
10. Debuggability & Operability

Evidence Bundle

• Changed hot paths:validateEnrichConfig, contentFilterClauses, buildProposeQuery, runProposeTick source scan, safeDeref T2 source excerpt.
• Impacted callers:runProposeTick calls buildProposeQuery at hypaware-core/plugins-workspace/context-graph-enrich/src/propose.js:41; runCurateTick calls safeDeref at hypaware-core/plugins-workspace/context-graph-enrich/src/curate.js:97; plugin validation uses validateEnrichConfig at hypaware-core/plugins-workspace/context-graph-enrich/src/index.js:36 and :40.
• Impacted tests: config defaults/overrides at test/plugins/context-graph-enrich-config.test.js:16 and :78; propose SQL filter coverage at test/plugins/context-graph-enrich-propose.test.js:43 and :67; SQL helper coverage at test/plugins/context-graph-enrich-sql.test.js:27 and :34.
• Unresolved uncertainty: I did not run the suite; this is a targeted diff review. No direct execution test for the private T2 safeDeref query was visible in the changed test set.

Claude review

Claude review

safeDeref's new content-filter WHERE clause has no test assertion

• Severity: minor
• Confidence: 88
• Evidence: hypaware-core/plugins-workspace/context-graph-enrich/src/curate.js:267
• Why it matters: The single behavior-changing line in curate.js — ANDing contentFilterClauses(cfg) into the deref WHERE so an excluded part sharing a message_id with a kept text part is not re-admitted into the curator excerpt — is exercised but never asserted (the curate test's fakeQuery matches only on FROM <table> and ignores the WHERE), so a regression in the deref filter assembly would not be caught.
• Suggested fix: Add a curate test that captures the deref query string (or seeds the source dataset with a kept text part and an excluded tool_result part sharing one message_id) and asserts the excluded part is absent from the curator excerpt, mirroring the buildProposeQuery filter assertions.

---

Four of five review passes (guidance compliance, shallow bug scan, historical/git-history, contract & callers) found no issues at or above the confidence threshold. Notable confirmations from those passes:

• No-semicolon / JSDoc-type / @import style rules all satisfied; the @ref LLP 0028#row-selection annotations resolve to the new ## Row selection heading.
• cfg in safeDeref is runtime.config (the full validated EnrichConfig); all contentFilterClauses call sites pass a fully-populated cfg, so .length/.map never hit undefined.
• The watermark-safety claim holds: the cursor advances only over rows the query returned (built into rowMeta), and the content filter is deterministic, so filtered higher-ts rows are re-filtered each tick and never cause an unfiltered row to be skipped.
• Schema confirmed: part_type (non-null) and content_text (nullable) exist at schema v6; mapPartType emits 'tool_result', matching the default exclude_part_types.
• No EnrichConfig is constructed outside validateEnrichConfig in any runtime path; full suite (1189 tests) passes.

---

Reports: .git/dual-review/pr-119

[philcunliffe]

Addressed dual-review findings — 877b31c

• Codex (major, contract fidelity) — the default exclude_part_types: ['tool_result'] no longer applies to a non-default source_dataset. A custom source defaults to [] (no part_type predicate, so the scan can't reference a column the source lacks) and opts into part-type filtering explicitly. require_text is left on for any source since it only reads the always-configured text_column. Documented in LLP 0028#row-selection; the gating site carries the @ref.
• Claude (minor, coverage) — added two runCurateTick tests that capture the safeDeref SQL and assert the shared content filter is ANDed into the deref WHERE (default source) and absent (custom source with the filter disabled).

npm test (1192 pass / 1 pre-existing skip), typecheck, and lint all green; @refs validate.