Skip to content

Replace any-typed annotations with real types - #196

Merged
platypii merged 1 commit into
masterfrom
replace-any-with-real-types
Jun 29, 2026
Merged

Replace any-typed annotations with real types#196
platypii merged 1 commit into
masterfrom
replace-any-with-real-types

Conversation

@platypii

@platypiiplatypii commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Replaces about fifteen any annotations with real types, surfaced by auditing JSDoc and .d.ts type annotations.

Changes

  • Daemon installer callbacks (daemon/types.d.ts): the four isLaunchAgentInstalled / launchAgentStatus / isSystemdUnitInstalled / systemdUnitStatus callbacks now declare their actual option shapes instead of (opts: any).
  • Registry casts (cli/core_commands.js): ctx.sources / ctx.sinks cast straight to the extended source/sink registry types rather than hopping through any. This also lets the iceberg table-format handle filter infer its element type, dropping a @param {any} h.
  • TUI runtime (cli/tui/runtime.js): stdin, stdout, isTTY/columns access, and the keypress handler now use Node's ReadStream / WriteStream and readline.Key types.

Left as any (justified)

  • JSON-RPC / MCP message handling, which is genuinely dynamic.
  • The picker skills / agents arguments, where the registry client list carries an "all" value the picker type intentionally excludes (narrowing there would be a behavior change, not a type fix).

Verification

  • npm run typecheck clean
  • npm test — 1485 pass, 1 skipped

@platypii
Replace any-typed annotations with real types
04e12a8
Daemon installer callbacks in daemon/types.d.ts now declare their actual
option shapes instead of `any`. Registry casts in cli/core_commands.js
cast straight to the extended source/sink registry types rather than
hopping through `any`, which also lets the iceberg handle filter infer
its element type. The TUI runtime types stdin, stdout, and the keypress
event via Node's stream and readline types.
Leaves genuinely dynamic JSON-RPC/MCP message handling as `any`, and the
picker skills/agents arguments where the registry client list carries an
"all" value the picker type intentionally excludes.
@platypii
platypii merged commit 81ed466 into masterJun 29, 2026
6 checks passed
@platypii
platypii deleted the replace-any-with-real-types branch June 29, 2026 16:45
philcunliffe added a commit that referenced this pull request Jun 29, 2026
@philcunliffe@claude
Fix typecheck: root-anchor type-module imports in feature .js files
c94e6b6
The conflict-resolution merge of master (PR #196 type-surface cleanup)
left four feature .js files importing type modules via sibling-relative
paths (`./types.d.ts`). tsc copies the generated declaration into
`types/` but does not emit input `.d.ts` siblings there, so
`types/core/sinks/incremental.d.ts` had a dangling `import('./types.d.ts')`
→ TS2307 in `npm run typecheck` after `build:types` (CI-only; local
`npm test` does not run tsc emit).
Match master's convention (e.g. sinks/driver.js, cache/spool.js): .js
files reference type modules by a repo-root-anchored path into src/
(`../../../src/core/sinks/types.js`), which resolves identically from
both the source location and the emitted types/ location, plus `.js`
extension for the kernel-types barrel.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
philcunliffe added a commit that referenced this pull request Jun 29, 2026
@philcunliffe@claude
Fix build:types: import ClientDescriptor from the src/core/types barrel
d421761
Master's PR #196 relocated ClientDescriptor into the src/core/types.d.ts
barrel. The merge left four files still importing it from
src/core/plugin_catalog.js, which itself only @imports the type (for local
use) and does not re-export it → TS2459 in `npm run build:types`, run via
`prepare` on `npm i`. That broke all three CI jobs (lint/typecheck/test)
at the install step before any tests ran, while the resolver's local
`npm test` passed because its symlinked node_modules skipped prepare.
Repoint all four to the canonical barrel, matching sibling files
(config/validate.js, cli/walkthrough.js): root-anchored `src/core/types.js`
for the emitted .js files, `.d.ts` for the input config/types.d.ts.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This was referenced Jun 29, 2026
Merged
Merged
philcunliffe added a commit that referenced this pull request Jun 30, 2026
@philcunliffe@claude
Client attach on join: auto-attach clients when the daemon binds (LLP…
96cd19d
… 0044/0045/0046) (#179)
* Plan: client attach on join (LLP 0046, implements 0045)
* Attach policy reader: tri-state readAttachPolicy (LLP 0044/0045)
New src/core/config/attach_policy.js — the backfill_policy.js twin and single
source of truth for interpreting a client adapter plugin entry's
`config.attach.on_join` block, shared by the attach handler (T6) and the
declared-attach status surface (T9) so the two can never disagree.
Tri-state, mirroring readBackfillPolicy:
- absent block → default-on (onJoin: undefined)
- on_join: true/false → that value
- present-but-malformed (non-boolean on_join) → fail-safe opt-out (false);
silently editing a user-owned client settings file on a malformed opt-out is
the wrong default, so do not fail open.
Pure; unit-tested over the full tri-state plus non-object attach blocks and the
`onJoin !== false` off-switch contract both consumers rely on.
@ref LLP 0044#where-attach-is-declared
@ref LLP 0045
Task-Id: T2
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Client attach: reconcile-context seam (clientDescriptors/clients/endpoint) (LLP 0045)
Extend ActionContext and ReconcileInput in src/core/config/types.d.ts with the
three optional fields the attach handler reads, mirroring LLP 0045 §Part 1:
- clientDescriptors?: Map<string, ClientDescriptor> — the static client→plugin
map (from buildPluginCatalog) used to enumerate desired() and to hand the
disk-driven undo each descriptor's attachProbe.
- clients?: AiGatewayCapability — runtime gateway capability used only to invoke
a client's attach effect (getClient(name).attach).
- endpoint?: string — the local gateway base URL clients attach to.
All optional and daemon-only by construction (a plain CLI boot leaves them
unset). Type-only; no behaviour change until a handler (T6) or the daemon (T7)
reads them. Imports AiGatewayCapability from the kernel types and ClientDescriptor
from plugin_catalog.js.
typecheck, lint, and the full test suite (1455 pass) stay green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Task-Id: T3
* Per-plugin attach config validation: validateAttachSection (LLP 0045 T8)
Add validateAttachSection beside validateBackfillSection in the claude and
codex config.js, wired into validateClaudeConfig / validateCodexConfig. It
validates the optional config.attach.on_join (boolean, default true) and
rejects unknown attach keys, mirroring the backfill section validator.
Plugin-local and additive: no top-level/core schema change. Realizes
LLP 0045 Part 4 (attach.on_join rides the client adapter's own config block,
validated by that plugin's config-section validator). Inert until the attach
handler (T6) and daemon wiring (T7) read the policy.
Tests cover accept/reject for both plugins, pointer mounting, and the
LLP 0031 central-lock merge-drop (a local attach.on_join cannot flip a
central-locked entry).
Task-Id: T8
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* T1: attach() writes a self-describing undo record (LLP 0045/0046)
Make each client adapter's attach marker carry everything the single
core undo (task 4) needs to reverse the attach from disk alone, with the
plugin unloaded.
claude (`_hypaware` marker): record `prev_base_url` (the restore target)
plus the managed `env.ANTHROPIC_BASE_URL` and session-context hook
entries. An idempotent re-attach preserves the marker's recorded
original `prev_base_url` instead of backing up our own gateway URL over
it, and reports that original via the attach result.
codex (`# BEGIN/END hypaware` marked block): already self-delimiting and
already records the prior `model_provider` as `# previous_model_provider`
— locked in with tests (block is self-describing; re-attach keeps the
original pointer).
Unit-tested via the marker contents (no plugin/gateway loaded).
Adapter-local; no core contract change (the `json: true` one-line attach
output shape is unchanged).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Task-Id: T1
* The single core undo (= detach): disk-driven client_detach_disk.js (LLP 0045/0046 T4)
Add `src/core/config/client_detach_disk.js` — the one detach
implementation. It reverses a client's attach from disk state alone: the
descriptor's `attachProbe` locates the settings file and the settings-file
marker is the self-describing undo record `attach()` wrote (LLP 0045 Part 3,
the contract task T1 landed).
Format-aware but plugin-agnostic, reusing the same `resolveClientSettingsPath`
+ `attachProbe` format dispatch (`json` marker-key / `toml` managed-block) the
read-side `probeClientAttached` uses, and importing no plugin code (which would
not survive the plugin being unloaded at reverse time):
- json (claude `_hypaware`): restore-or-remove each managed `env` key from the
recorded `prev_base_url`, strip the recorded managed hook entries (matched by
event/matcher/exact command so no orphaned `hyp claude-hook` survives), and
delete the marker. A pre-existing foreign base URL round-trips byte-for-byte;
the no-pre-existing-URL fixture round-trips to empty; an externally-overridden
value is left in place with a warning.
- toml (codex `# BEGIN/END hypaware …`): strip the self-delimiting managed
blocks and restore the prior `model_provider` recorded as
`# previous_model_provider`. The managed-block convention is now a
core-understood format contract, subsuming the codex adapter's old marked-block
strip + model_provider restore.
Atomic, mtime-gated writes preserve the adapters' concurrent-edit guarantee.
Unit-tested on fixture settings files with no plugin loaded (incl. hand-written
markers), proving reverse never depends on `ctx.clients`; cross-checked against
`probeClientAttachFromDescriptor` for both formats. No second caller is wired
yet (T5 reroutes `hyp detach`, T6 the reconciler `reverse()`), so this lands
inert and green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Task-Id: T4
* T9: status surface — declared-attach-targets derivation (LLP 0044/0045)
buildClientActionsReport (src/core/daemon/status.js) gains an attach
declared-targets derivation symmetric to backfill's. It reuses the
clientDescriptors-derived client-adapter plugin set and the shared
readAttachPolicy (T2) so status can never disagree with the attach handler
about what an `attach.on_join` block means.
Interpretation, per LLP 0044 §status-surface:
- enabled client adapter on a joined host, no marker -> pending
- attach.on_join:false, or a non-joined host -> n/a
- a `done` marker -> attached (the done rendering)
Attach targets are keyed by *client* name (the handler's request key,
descriptor.name) — not the owning plugin — so a `done` attach marker
collapses with the declared target instead of doubling it. A failed/pending
attach is its own status line and never flips `overall` to degraded
(client-action state is not a diagnostic). It reads markers only; it never
runs a reconcile pass.
Tests: mixed done/failed/pending/n-a read cleanly; on_join:false and a
non-joined explicit target both render n/a while a bare local install keeps
the V1 surface; a failed attach stays healthy; a done marker collapses with
its declared target (no double row).
Task-Id: T9
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Attach handler: action_attach.js (createAttachHandler + attachHandler) (LLP 0044/0045 T6)
The reversible instance of the client-action reconciler — the action_backfill.js
twin and the first handler to implement reverse(). New
src/core/config/action_attach.js exporting createAttachHandler(opts) and the
default attachHandler:
- desired() iterates ctx.clientDescriptors ∩ enabled plugins ∩ readAttachPolicy
(the attach_policy.js opt-out), guarded on ctx.clients.getClient(name) being
present so it never names a client perform() can't reach. Daemon-only by
construction: inert when clientDescriptors or clients is absent (a plain CLI
boot).
- perform() is in-process (a bounded settings write, not a subprocess): resolves
the runtime registration, calls attach({ endpoint: ctx.endpoint, config:{},
stdout, stderr, json:true }), parses the one-line JSON, and records the marker
detail (settings_path, prev_value). A throw becomes a failed outcome the
reconciler retries.
- reverse() is disk-driven, not adapter-driven: it runs after the staged restart
has unloaded the adapter, so it replays the single core undo
(detachClientFromDisk) from the descriptor's attachProbe + the self-describing
marker, never touching ctx.clients.
Adds the attach handler's type surface (ClientDetachFromDisk seam +
CreateAttachHandlerOptions) to config/types.d.ts, the direct analog of
CreateBackfillHandlerOptions. Unit-tested (22 cases) with injected fake
clientDescriptors + clients + fs, including a real fs round-trip proving reverse
runs with no adapter loaded. The reconciler is left untouched (threading the new
context fields onto ctx is T7's daemon wiring).
Task-Id: T6
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Client attach: retire adapter detach(), reroute manual detach to the core undo (LLP 0045 T5)
Drop `detach` from `AiGatewayClientRegistration` (kernel type) and from the
claude/codex `registerClient()` calls, and point `runClientLifecycle`'s detach
branch at the single T4 core disk-driven undo (`detachClientFromDisk`),
resolved per client via its `clientDescriptor`. There is now exactly one undo,
shared by manual `hyp detach` and the daemon reconciler's `reverse()`, so the
two cannot drift.
- collectivus-plugin-kernel-types.d.ts: remove `detach()` from
`AiGatewayClientRegistration` and the orphaned `AiGatewayClientDetachContext`;
refresh the registerClient/getClient doc comments to attach-only.
- claude/codex adapters: drop the registration `detach` property and the
now-unused `writeDetachOutput`/type import (the settings-file `detach()`
writers stay as exported utilities).
- ai-gateway api.js: `registerClient` now requires only `attach()`.
- core_commands.js: new `detachClientViaCore` + `writeCoreDetachOutput` emit the
`client.detach` span and a plugin-agnostic `done`/no-op output via the core
undo; the descriptor map is built once per detach invocation.
- Tests/smoke updated for the rerouted path: ai-gateway-api (attach-only
validation), command-dispatch + integration (detach is the disk undo, HOME
isolated), and the claude_attach_detach smoke prose. claude_attach_detach /
client_attach_idempotent smokes stay green.
Task-Id: T5
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Client attach: daemon wiring — resolve client seam + register [attachHandler, backfillHandler] (LLP 0045 T7)
Wire the attach handler into the daemon's client-action reconciler
(LLP 0045 §Part 1 / Module-seam-breakdown item 7):
- runtime.js: resolve the client-action seam from boot — clientDescriptors from
the plugin catalog (now surfaced on BootKernelResult), and clients/endpoint
from boot.runtime.capabilities when the ai-gateway capability is present, via
localEndpoint() with the configuredGatewayEndpoint fallback. Thread all three
onto each reconcile() input and register [attachHandler, backfillHandler] —
attach first (exported DEFAULT_ACTION_HANDLERS so the order is a testable
invariant).
- action_reconciler.js: thread the new context fields (clientDescriptors /
clients / endpoint) onto the ActionContext so a client handler can read them.
- boot.js / runtime types: surface the boot catalog's clientDescriptors on
BootKernelResult.
- gateway_endpoint.js: extract configuredGatewayEndpoint/endpointFromListen into
a shared module (was duplicated in core_commands.js and walkthrough.js; the
daemon is the third consumer).
Tests: daemon-reconcile.test.js gains the handler-order invariant and a
gateway-enabled boot asserting clientDescriptors/clients/endpoint are threaded
(and undefined on a non-gateway boot). New hermetic smoke client_attach_on_join
drives join -> auto-attach -> no-re-attach -> config-drop reverse end to end.
typecheck, lint, full test suite (1522 pass), and the new + existing attach/join
smokes are green. (Pre-existing unrelated red: walkthrough_to_first_query, also
red on the untouched integration branch.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Task-Id: T7
* LLP 0045: flip Status Accepted -> Active (client attach shipped)
Marks the client-attach implementation design as built-and-merged now that
the whole change set (T1-T9) has landed. Active = shipped marker per the
project lifecycle (sibling design LLP 0041 is already Active).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Task-Id: T10
* client-attach: legacy-marker detach fallback; descriptor-map manual detach; bundled+installed descriptors; type+doc fixes (#179 review)
Dual-review request_changes fixes for PR #179 (integration/client-attach):
1. Legacy-marker half-reversal: detachJsonMarker now falls back to the
pre-self-describing convention when marker.managed is absent — removes
ANTHROPIC_BASE_URL at the recorded port and strips claude-hook
session-context hooks, so a manual `hyp detach` after upgrade fully
reverses with nothing orphaned. Logic moved from the now-retired
claude-adapter detach() into the one core undo.
2. Manual detach no longer gated on the live gateway: runClientLifecycle
resolves/validates the detach target from the bundled+installed
descriptor map, so a dropped/unloaded adapter's on-disk attach still
reverses.
3. buildClientDescriptorMap now built from bundled+installed (matching
boot/status), so installed (non-bundled) client adapters resolve.
4. DetachFromDiskResult moved from a @typedef to an interface in
config/types.d.ts, imported via @import.
5. Relabel "§Part 7" prose to "§Module / seam breakdown item 7".
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0178wfuBrVQHwPbsHAtcE7YS
* client-attach: detach works without gateway; TOCTOU stat-before-read; drop dead types; hoist import types; document stable client seam (#179 round-2)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0178wfuBrVQHwPbsHAtcE7YS
* client-attach: unlink temp file on partial write; daemon auto-attach requires a bound endpoint (#179 round-3 hardening)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0178wfuBrVQHwPbsHAtcE7YS
* Fix build:types: import ClientDescriptor from the src/core/types barrel
Master's PR #196 relocated ClientDescriptor into the src/core/types.d.ts
barrel. The merge left four files still importing it from
src/core/plugin_catalog.js, which itself only @imports the type (for local
use) and does not re-export it → TS2459 in `npm run build:types`, run via
`prepare` on `npm i`. That broke all three CI jobs (lint/typecheck/test)
at the install step before any tests ran, while the resolver's local
`npm test` passed because its symlinked node_modules skipped prepare.
Repoint all four to the canonical barrel, matching sibling files
(config/validate.js, cli/walkthrough.js): root-anchored `src/core/types.js`
for the emitted .js files, `.d.ts` for the input config/types.d.ts.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
philcunliffe added a commit that referenced this pull request Jun 30, 2026
@philcunliffe@claude
Incremental sink reads: per-row ingest-seq watermark for forward + bl…
63efc23
…ob sinks (LLP 0039/0040) (#159)
* Design: incremental sink reads (LLP 0040)
Cover LLP 0039 with a neutral-minted design for a per-(sink, partition)
watermark so the central forward sink and the core blob sink read and ship
only rows added since their last successful export.
Recommends a monotonic per-row _hyp_ingest_seq column over snapshot ancestry
(does not survive a compaction generation swap) and a content-addressed
seen-set (cannot meet the bounded-read goal). Specifies the readRows since/
continuation extension, the persisted watermark contract keyed by the
generation-stable logical partition path, application to both sinks, and the
exactly-once argument across retention prunes and compaction swaps.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Plan: incremental sink reads (LLP 0042)
Refine LLP 0040 into six small, independently-mergeable tasks along the
producer -> read-API -> persistence -> consumer seam:
T1 stamp _hyp_ingest_seq at the decorateRow chokepoint (deps: [])
T2 readRows since/continuation + readRowsSince (deps: T1)
T3 per-(sink,partition) watermark store keyed by logical path (deps: T2)
T4 wire the central forward sink (deps: T2,T3)
T5 wire the core blob sink (deps: T2,T3)
T6 exactly-once tests across retention prune + compaction swap (deps: T4,T5)
Verified with `neutral ready incremental-sink-reads --json`.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* T1: stamp internal monotonic _hyp_ingest_seq at the decorateRow flush chokepoint
Adds the row-resident, append-monotonic int64 watermark column that the
incremental-sink-reads design (LLP 0040, Candidate B) is built on. This is
the producer half of the seam: nothing reads the column yet, and it is
stripped by INTERNAL_FIELDS from every existing readRows consumer, so it
merges with zero behavioural change.
- New `createIngestSeqAllocator` (src/core/cache/ingest-seq.js): a crash-safe,
never-regressing monotonic int64 allocator. Reserve-before-stamp — a block
of seqs is durably persisted (nextSeq advanced via atomic write-rename)
before any seq in it is handed to a row, so a resumed flush never re-issues
a seq <= one already stamped/exported. Gaps are tolerated; regressions are
not. The counter is cache-global (<cacheRoot>/_hyp_ingest_seq.json), not a
per-partition cursor.json, because decorateRow runs before rows are grouped
into source= partitions and two spool paths (live + backfill) can feed one
partition — only a cache-wide counter keeps every partition's seq subsequence
strictly increasing. (LLP 0040 §7 records this refinement of risk #2.)
- streaming-reader.js: decorateRow stamps `_hyp_ingest_seq` (the cache_row_id
hash is still computed over the original row, so seq does not perturb dedup);
the chunk's columns gain the additive nullable INT64 column so it lands in
the Iceberg schema and rides a compaction generation swap verbatim; the field
joins INTERNAL_FIELDS.
- spool.js wires one cache-global allocator into the flush loop.
- Tests: allocator monotonicity / never-regress-across-restart / reserve-
before-stamp / concurrency; streamFlushFile stamping; and a storage round-trip
proving the seq persists in Iceberg, increases per row, and is stripped from
readRows. Verified separately that the column survives a real compaction swap.
Task-Id: T1
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* T2: extend storage read contract with cursor-aware incremental reads
Add a back-compatible `opts.since` to `readRows` and a cursor-aware
`readRowsSince` sibling that pairs each internal-stripped row with its
`after` continuation token, so the forward and blob sinks can read only
rows added since their last durable export.
- `scanRowsFromTable` gains an `opts.since` (bigint `_hyp_ingest_seq`
watermark) and applies a `seq > since` predicate as a yielded-row
filter. It is NOT pushed into icebird's `scan({ where })`: icebird
couples file/row-group pruning with a per-row match that drops nulls
(`null > since` is false), which would silently skip the legacy
null-seq rows the migration must preserve (LLP 0040 risk #1). The
design names this yielded-row filter as the fallback; a future
null-aware icebird filter can add the file-skip optimization on top.
- null-seq = new: a row whose `_hyp_ingest_seq` is null/absent
(pre-upgrade) is always yielded, so the one-time migration is at worst
a full re-export, never silent data loss. A table that never carried
the seq column yields everything.
- `after` is a monotonic high-water of real seqs, so a null-seq row
carries the prior watermark forward and progress never regresses even
when the scan visits seqs out of order (interleaved sources).
- `opts` absent ⇒ byte-for-byte the pre-existing full scan, so every
current caller is untouched until it opts in.
- Update the kernel-types decl: `SinkContinuation`, `ReadRowsOptions`,
the extended `readRows`, and the new `readRowsSince`.
Tests cover back-compat, after-token monotonicity, no-new-rows ≈0,
incremental new rows, the null-seq migration contract, the pure-legacy
(no seq column) table, and invalid-token rejection.
Task-Id: T2
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* T3: persisted per-(sink,partition) watermark store keyed by logical path
Add `src/core/sinks/watermarks.js`, the shared incremental-read watermark
store for the forward and blob sinks (LLP 0042 task T3). Files live at
`<stateDir>/watermarks/<dataset>/<partition-key>.json` and carry the
versioned `{ continuation, exportedRowCount, updatedAt }` record.
The key is derived from the partition's stable LOGICAL path
(`datasets/<dataset>/<partition...>` relative to cacheRoot), never the
physical `tableDir` inside it — the hinge of design constraint (B): the
watermark reads straight through a compaction generation swap and a
retention front-prune. `write` is atomic write-rename (the
`writeCursor`/`writeProgress`/`ingest-seq.js` idiom); a corrupt or absent
record reads back as null so a sink re-exports from the start
(at-least-once + downstream dedup) rather than silently skipping rows.
Adds `SinkWatermarkKey`/`SinkWatermarkRecord`/`SinkWatermarkStore` types,
anchors LLP 0040 §3 (`#watermark-contract`), and a unit suite covering key
derivation (logical-not-tableDir, nesting, sanitize, sentinel, escape
guard), round-trip, in-place advance, atomic-no-temp, malformed-token
rejection, and corrupt-file null.
@ref LLP 0040#watermark-contract
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Task-Id: T3
* Wire central forward sink to incremental readRowsSince watermark
Switch `forwardPartition` from a full-partition `storage.readRows(tablePath)`
scan to the cursor-aware `storage.readRowsSince(tablePath, { since })`, driven
by a per-(sink instance, partition) watermark loaded from the sink's stateDir
watermark store (T3). Each acked chunk advances the watermark to that chunk's
last `after` token (ship first, advance second), so a crash re-sends at most one
chunk and the server idempotency ledger now backstops only a bounded in-flight
suffix instead of the whole partition. A tick with no new rows yields zero rows,
sends zero chunks, and writes zero bytes.
Chunking (MAX_CHUNK_ROWS/MAX_CHUNK_BYTES), the Retry-After backpressure loop,
and `batchIdForChunk` derivation are unchanged. A missing/unreadable watermark
or underivable key falls back to a full scan (at-least-once + server dedup),
never a silent skip.
Implements task T4 of incremental-sink-reads (LLP 0040 §3-4, 0042).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Task-Id: T4
* T5: wire core blob sink (local-fs + s3) to incremental readRowsSince
Wire the local-fs and s3 blob destinations to the cursor-aware
readRowsSince surface (T2) and the persisted per-(sink instance,
partition) watermark store (T3), so each tick exports only rows added
since the sink's last durable PUT.
- New shared helpers in src/core/sinks/incremental.js (exported via
hypaware/core/sinks): openIncrementalRows (peek-to-decide-empty,
self-tracking rowCount + high-water lastAfter, feeds the unchanged
encoder.encodePartition contract), withSeqRangeFilename (embeds
[sinceSeq,lastSeq] before the extension), watermarkKeyFor, and
createInstanceWatermarkStore.
- Empty new-row set writes no blob (skip, 0 bytes).
- The output filename/object key embeds the [sinceSeq,lastSeq] range so a
crash-retry re-PUTs the same key (idempotent overwrite) — the blob
sink's stand-in for the central server ledger.
- The watermark advances only after the durable write/PUT.
- PluginPaths.stateDir is per-plugin, not per sink instance, so the
watermark store is scoped under the instance to honor the per-(sink
instance, partition) contract.
- Tests: helper unit tests, a local-fs end-to-end incremental test
(ranged filename, watermark advance, skip-empty, new range, cumulative
count), and rewritten s3-export-batch tests (skip-empty, ranged key,
watermark advance, idempotent re-PUT on lost watermark) preserving the
prior retry-semantics coverage.
- Updated the two local-fs blob-sink smokes to match the ranged filename
(note: both are pre-existing red on the integration branch for an
unrelated reason — the driver hands the sink the drained spool path).
Task-Id: T5
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* T6: exactly-once acceptance suite + smoke for incremental sink reads
Add the LLP 0040 exactly-once proof the T4/T5 unit suites can't give
(they stub storage): a deterministic acceptance test driving the REAL
kernel cache, retention enforcer, maintainCache compaction, BOTH sinks
(central forward + core local-fs blob), and the driver outbox respool.
Covers: ~0 bytes on a no-new-rows tick, ~N on an N-new tick, exactly-once
across a retention front-prune and a compaction generation swap for both
sinks, and watermark vs. driver-outbox respool composition (suffix-only
replay + idempotent batch-id / re-PUT). Adds a hermetic smoke proving the
blob sink reads straight through a compaction generation swap via the real
driver. Anchors LLP 0040 section 5 so the @ref resolves.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Task-Id: T6
* fix(central-forward): keep chunk batch-id stable across watermark advance
The forward sink derived its X-Hyp-Batch-Id from the per-tick chunk ordinal
(chunkIndex). After an earlier chunk was acked and the (sink, partition)
watermark advanced, a respool re-read only the un-acked suffix and re-numbered
it from 0 — minting a NEW batch-id for a chunk that may already have committed
on the server (ambiguous ack / commit-then-5xx). The server idempotency ledger
could not dedupe the redelivery, double-storing rows and breaking the spec's
'ledger covers mid-batch retries' guarantee.
Key the batch-id on the chunk's start seq (the watermark it resumes from)
instead. A respooled suffix reproduces the same [startSeq, body] and thus the
same id, so the ledger dedupes it; distinct chunks still differ because each
row's _hyp_ingest_seq is unique. Adds a cross-tick regression test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(incremental-sink-reads): close exactly-once holes in forward+blob sinks (#159 review)
Dual-review found a BLOCKER plus majors against the exactly-once claim. Fixes,
each with a regression test that fails before and passes after:
[BLOCKER] Forward sink advanced the watermark per acked chunk to the scan's
running-max `after`. Because the cache scan is NOT seq-ordered (interleaved
live+backfill spool; post-compaction sortOrder re-sort), an early acked chunk
could checkpoint past lower-seq rows still un-acked in a later chunk; a
between-chunk failure then dropped them forever (`seq <= since`). Fix: advance
the forward watermark ONCE at end-of-partition (as the blob sink already does),
so a partial partition never checkpoints — a failure re-reads the whole
partition and the server ledger dedupes the acked prefix (stable
chunkStartSeq batch ids).
test: central-forward-chunking "an unordered scan never skips a lower-seq row
when a later chunk fails".
[MAJOR] Legacy null-seq rows re-exported every tick (never reached steady
state). Add `includeLegacy` to the storage read API (default true). Both sinks
pass `includeLegacy = (no durable watermark)`: a fresh sink exports the
pre-upgrade backlog once, then excludes null-seq rows. Safe because no new
null-seq row can appear post-upgrade (decorateRow stamps every flushed row).
tests: sink-incremental-acceptance pure-legacy + mixed, forward + blob
(~0 bytes on 2nd tick, no row in two artifacts).
[MAJOR] Central forward sink scoped watermarks per-PLUGIN, not per-instance —
two @hypaware/central instances clobbered one watermark file. Switch to
createInstanceWatermarkStore({ paths, instanceName }) (matching local-fs/s3);
correct the @ref.
test: sink-incremental-acceptance "two instances keep independent
watermarks (no cross-instance skip)".
[MAJOR, doc] Bounded-reads constraint (C): the read is a full scan over the
surviving partition with a yielded-row filter, not the file/row-group skip the
design implied. Correct LLP 0040 §1(C)/§5 to state reads are bounded by
surviving-partition size, with O(N_new) reads pending null-aware icebird
pushdown.
[MINOR/NIT, doc+style] Correct LLP 0040 §4 (batchIdForChunk keys by
chunkStartSeq, not chunkIndex). Hoist inline import('...') types to @import in
sink-watermarks/acceptance tests and the compaction smoke.
[MAJOR, ESCALATED — left unfixed] Mid-retry duplication when a unit commits,
its watermark write is lost, AND new rows append before the retry: the resumed
in-flight unit grows past what committed and the dedup net no longer recognizes
it. The common in-flight retry (no new arrivals) IS covered. A correct fix needs
a pre-commit intent (read `until` upper bound + persisted intent record) across
both sinks — a design-level change to the watermark contract, deferred rather
than half-patched under the exactly-once claim. Documented as LLP 0040 §6 risk
#7 / §5 known gap.
LLP 0040/0042 updated in this commit to match the new behavior.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Fix typecheck: root-anchor type-module imports in feature .js files
The conflict-resolution merge of master (PR #196 type-surface cleanup)
left four feature .js files importing type modules via sibling-relative
paths (`./types.d.ts`). tsc copies the generated declaration into
`types/` but does not emit input `.d.ts` siblings there, so
`types/core/sinks/incremental.d.ts` had a dangling `import('./types.d.ts')`
→ TS2307 in `npm run typecheck` after `build:types` (CI-only; local
`npm test` does not run tsc emit).
Match master's convention (e.g. sinks/driver.js, cache/spool.js): .js
files reference type modules by a repo-root-anchored path into src/
(`../../../src/core/sinks/types.js`), which resolves identically from
both the source location and the emitted types/ location, plus `.js`
extension for the kernel-types barrel.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@platypii