Uh oh!
There was an error while loading. Please reload this page.
[codex] Remove OpenTelemetry npm dependencies - #2
Merged
Conversation
philcunliffe
marked this pull request as ready for review
May 21, 2026 16:55
This was referenced May 21, 2026
Merged
This was referenced May 29, 2026
Merged
This was referenced Jun 8, 2026
This was referenced Jun 15, 2026
philcunliffe added a commit
that referenced
this pull request
Jun 18, 2026
…outing) Fixes the dual-review findings on the two-regime enrichment rework: - propose: select sessions on the EXACT (ts, tiebreak) tuple, not MAX(ts) alone. buildSessionAggregateQuery now ranks parts with ROW_NUMBER() OVER (PARTITION BY anchor ORDER BY ts DESC, tiebreak DESC) and keeps rn=1, so a same-millisecond part that advanced a settled session past its mark is reselected instead of silently dropped (Codex #1). selectSessions compares the full tuple via cmpMark; the exact match also avoids re-selecting already-enriched sessions every tick. - state: add updateState() read-modify-write helper (atomic w.r.t. the event loop). propose's final write and batch submit/collect now merge into the latest on-disk state, so a curate_job submitted during propose's await window is preserved rather than clobbered to null — no orphaned batch / double spend (Codex #4 / Claude). - curate: an under-specified merge (missing merge_into or item_type) can't be routed to the right content-addressed node, so routeDecision returns it pending (no commit, no resolution) and routeClusterDecisions leaves it in the queue, rather than attaching the produced edge to the wrong node (Codex #2). - commands: export + test parseBackfillArgv (bare argv, each flag, mutual exclusion, unknown flag, stray positional, flag ordering). - llp 0028: describe the actual deterministic (timestamp, tiebreak) ordering chosen over message-graph columns for source portability, so the orderSessionParts [implements] ref is honest. - batch/curate: move VectorSearchHit/CompletionRequest to top-of-file @import blocks (no inline import('...') types). npm test 1264 pass / 1 skipped; tsc --noEmit clean; lint clean (379 files). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
philcunliffe added a commit
that referenced
this pull request
Jun 18, 2026
…tering, Batch API) (#124) * context-graph-enrich: two-regime rework (full-session T1, curate clustering, Batch API) Implements the LLP 0028 redesign; the spec edit lands with the code. - Per-session high-water mark replaces the global (timestamp, tiebreak) cursor (state.js schema v4). T1 propose now extracts a whole DAG-ordered session in a single call — closing the 12k-char truncation defect (silent loss on 47% of sessions). Two regime selectors: ongoing (settled + past-watermark, capped) and backfill (all sessions). - T2 curate clusters by recall-region + embedding-cosine the no-recall remainder (hypaware.embedder, best-effort), with content-based shared context replacing the structural one-hop neighborhood. merge now writes a committed row under the canonical key with the merging session's anchor → a `produced` edge per contributing session; the node dedups by content-addressed id. - @hypaware/completion-anthropic gains an optional `batch` surface (Anthropic Message Batches: submit/poll/results/cancel; refusal = success; provider error messages never surfaced). New `hyp enrich backfill` command, and the ongoing curate daemon source is now submit-and-collect (job state in the sidecar) so frontier work never blocks a tick. npm test 1251 pass / 0 fail; tsc 0 errors; lint clean; all @refs resolve. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * context-graph-enrich: address PR 124 review (race, watermark, merge routing) Fixes the dual-review findings on the two-regime enrichment rework: - propose: select sessions on the EXACT (ts, tiebreak) tuple, not MAX(ts) alone. buildSessionAggregateQuery now ranks parts with ROW_NUMBER() OVER (PARTITION BY anchor ORDER BY ts DESC, tiebreak DESC) and keeps rn=1, so a same-millisecond part that advanced a settled session past its mark is reselected instead of silently dropped (Codex #1). selectSessions compares the full tuple via cmpMark; the exact match also avoids re-selecting already-enriched sessions every tick. - state: add updateState() read-modify-write helper (atomic w.r.t. the event loop). propose's final write and batch submit/collect now merge into the latest on-disk state, so a curate_job submitted during propose's await window is preserved rather than clobbered to null — no orphaned batch / double spend (Codex #4 / Claude). - curate: an under-specified merge (missing merge_into or item_type) can't be routed to the right content-addressed node, so routeDecision returns it pending (no commit, no resolution) and routeClusterDecisions leaves it in the queue, rather than attaching the produced edge to the wrong node (Codex #2). - commands: export + test parseBackfillArgv (bare argv, each flag, mutual exclusion, unknown flag, stray positional, flag ordering). - llp 0028: describe the actual deterministic (timestamp, tiebreak) ordering chosen over message-graph columns for source portability, so the orderSessionParts [implements] ref is honest. - batch/curate: move VectorSearchHit/CompletionRequest to top-of-file @import blocks (no inline import('...') types). npm test 1264 pass / 1 skipped; tsc --noEmit clean; lint clean (379 files). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * context-graph-enrich: don't unref the inter-poll delay (fixes CI test flake) The pre-existing batch tests failed in CI (ubuntu-latest's Node 20) while passing on local Node 24: `pollUntilEnded` awaits `delay()`, whose setTimeout was `unref()`'d, so the test-runner event loop could drain before the timer fired — reported as "Promise resolution is still pending but the event loop has already resolved", poisoning the whole batch test file. Removing the unref also fixes a latent bug: this delay is awaited inside the backfill command's run-to-completion poll loop, so it must keep the loop alive — an unref'd timer could let `hyp enrich backfill` exit mid-poll. The daemon source intervals keep their unref (correct: never block shutdown); the abort signal still clears this timer for prompt cancellation. npm test 1264 pass / 1 skipped; tsc clean; lint clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This was referenced Jun 19, 2026
philcunliffe added a commit
that referenced
this pull request
Jun 30, 2026
…ob sinks (LLP 0039/0040) (#159) * Design: incremental sink reads (LLP 0040) Cover LLP 0039 with a neutral-minted design for a per-(sink, partition) watermark so the central forward sink and the core blob sink read and ship only rows added since their last successful export. Recommends a monotonic per-row _hyp_ingest_seq column over snapshot ancestry (does not survive a compaction generation swap) and a content-addressed seen-set (cannot meet the bounded-read goal). Specifies the readRows since/ continuation extension, the persisted watermark contract keyed by the generation-stable logical partition path, application to both sinks, and the exactly-once argument across retention prunes and compaction swaps. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Plan: incremental sink reads (LLP 0042) Refine LLP 0040 into six small, independently-mergeable tasks along the producer -> read-API -> persistence -> consumer seam: T1 stamp _hyp_ingest_seq at the decorateRow chokepoint (deps: []) T2 readRows since/continuation + readRowsSince (deps: T1) T3 per-(sink,partition) watermark store keyed by logical path (deps: T2) T4 wire the central forward sink (deps: T2,T3) T5 wire the core blob sink (deps: T2,T3) T6 exactly-once tests across retention prune + compaction swap (deps: T4,T5) Verified with `neutral ready incremental-sink-reads --json`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * T1: stamp internal monotonic _hyp_ingest_seq at the decorateRow flush chokepoint Adds the row-resident, append-monotonic int64 watermark column that the incremental-sink-reads design (LLP 0040, Candidate B) is built on. This is the producer half of the seam: nothing reads the column yet, and it is stripped by INTERNAL_FIELDS from every existing readRows consumer, so it merges with zero behavioural change. - New `createIngestSeqAllocator` (src/core/cache/ingest-seq.js): a crash-safe, never-regressing monotonic int64 allocator. Reserve-before-stamp — a block of seqs is durably persisted (nextSeq advanced via atomic write-rename) before any seq in it is handed to a row, so a resumed flush never re-issues a seq <= one already stamped/exported. Gaps are tolerated; regressions are not. The counter is cache-global (<cacheRoot>/_hyp_ingest_seq.json), not a per-partition cursor.json, because decorateRow runs before rows are grouped into source= partitions and two spool paths (live + backfill) can feed one partition — only a cache-wide counter keeps every partition's seq subsequence strictly increasing. (LLP 0040 §7 records this refinement of risk #2.) - streaming-reader.js: decorateRow stamps `_hyp_ingest_seq` (the cache_row_id hash is still computed over the original row, so seq does not perturb dedup); the chunk's columns gain the additive nullable INT64 column so it lands in the Iceberg schema and rides a compaction generation swap verbatim; the field joins INTERNAL_FIELDS. - spool.js wires one cache-global allocator into the flush loop. - Tests: allocator monotonicity / never-regress-across-restart / reserve- before-stamp / concurrency; streamFlushFile stamping; and a storage round-trip proving the seq persists in Iceberg, increases per row, and is stripped from readRows. Verified separately that the column survives a real compaction swap. Task-Id: T1 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * T2: extend storage read contract with cursor-aware incremental reads Add a back-compatible `opts.since` to `readRows` and a cursor-aware `readRowsSince` sibling that pairs each internal-stripped row with its `after` continuation token, so the forward and blob sinks can read only rows added since their last durable export. - `scanRowsFromTable` gains an `opts.since` (bigint `_hyp_ingest_seq` watermark) and applies a `seq > since` predicate as a yielded-row filter. It is NOT pushed into icebird's `scan({ where })`: icebird couples file/row-group pruning with a per-row match that drops nulls (`null > since` is false), which would silently skip the legacy null-seq rows the migration must preserve (LLP 0040 risk #1). The design names this yielded-row filter as the fallback; a future null-aware icebird filter can add the file-skip optimization on top. - null-seq = new: a row whose `_hyp_ingest_seq` is null/absent (pre-upgrade) is always yielded, so the one-time migration is at worst a full re-export, never silent data loss. A table that never carried the seq column yields everything. - `after` is a monotonic high-water of real seqs, so a null-seq row carries the prior watermark forward and progress never regresses even when the scan visits seqs out of order (interleaved sources). - `opts` absent ⇒ byte-for-byte the pre-existing full scan, so every current caller is untouched until it opts in. - Update the kernel-types decl: `SinkContinuation`, `ReadRowsOptions`, the extended `readRows`, and the new `readRowsSince`. Tests cover back-compat, after-token monotonicity, no-new-rows ≈0, incremental new rows, the null-seq migration contract, the pure-legacy (no seq column) table, and invalid-token rejection. Task-Id: T2 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * T3: persisted per-(sink,partition) watermark store keyed by logical path Add `src/core/sinks/watermarks.js`, the shared incremental-read watermark store for the forward and blob sinks (LLP 0042 task T3). Files live at `<stateDir>/watermarks/<dataset>/<partition-key>.json` and carry the versioned `{ continuation, exportedRowCount, updatedAt }` record. The key is derived from the partition's stable LOGICAL path (`datasets/<dataset>/<partition...>` relative to cacheRoot), never the physical `tableDir` inside it — the hinge of design constraint (B): the watermark reads straight through a compaction generation swap and a retention front-prune. `write` is atomic write-rename (the `writeCursor`/`writeProgress`/`ingest-seq.js` idiom); a corrupt or absent record reads back as null so a sink re-exports from the start (at-least-once + downstream dedup) rather than silently skipping rows. Adds `SinkWatermarkKey`/`SinkWatermarkRecord`/`SinkWatermarkStore` types, anchors LLP 0040 §3 (`#watermark-contract`), and a unit suite covering key derivation (logical-not-tableDir, nesting, sanitize, sentinel, escape guard), round-trip, in-place advance, atomic-no-temp, malformed-token rejection, and corrupt-file null. @ref LLP 0040#watermark-contract Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Task-Id: T3 * Wire central forward sink to incremental readRowsSince watermark Switch `forwardPartition` from a full-partition `storage.readRows(tablePath)` scan to the cursor-aware `storage.readRowsSince(tablePath, { since })`, driven by a per-(sink instance, partition) watermark loaded from the sink's stateDir watermark store (T3). Each acked chunk advances the watermark to that chunk's last `after` token (ship first, advance second), so a crash re-sends at most one chunk and the server idempotency ledger now backstops only a bounded in-flight suffix instead of the whole partition. A tick with no new rows yields zero rows, sends zero chunks, and writes zero bytes. Chunking (MAX_CHUNK_ROWS/MAX_CHUNK_BYTES), the Retry-After backpressure loop, and `batchIdForChunk` derivation are unchanged. A missing/unreadable watermark or underivable key falls back to a full scan (at-least-once + server dedup), never a silent skip. Implements task T4 of incremental-sink-reads (LLP 0040 §3-4, 0042). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Task-Id: T4 * T5: wire core blob sink (local-fs + s3) to incremental readRowsSince Wire the local-fs and s3 blob destinations to the cursor-aware readRowsSince surface (T2) and the persisted per-(sink instance, partition) watermark store (T3), so each tick exports only rows added since the sink's last durable PUT. - New shared helpers in src/core/sinks/incremental.js (exported via hypaware/core/sinks): openIncrementalRows (peek-to-decide-empty, self-tracking rowCount + high-water lastAfter, feeds the unchanged encoder.encodePartition contract), withSeqRangeFilename (embeds [sinceSeq,lastSeq] before the extension), watermarkKeyFor, and createInstanceWatermarkStore. - Empty new-row set writes no blob (skip, 0 bytes). - The output filename/object key embeds the [sinceSeq,lastSeq] range so a crash-retry re-PUTs the same key (idempotent overwrite) — the blob sink's stand-in for the central server ledger. - The watermark advances only after the durable write/PUT. - PluginPaths.stateDir is per-plugin, not per sink instance, so the watermark store is scoped under the instance to honor the per-(sink instance, partition) contract. - Tests: helper unit tests, a local-fs end-to-end incremental test (ranged filename, watermark advance, skip-empty, new range, cumulative count), and rewritten s3-export-batch tests (skip-empty, ranged key, watermark advance, idempotent re-PUT on lost watermark) preserving the prior retry-semantics coverage. - Updated the two local-fs blob-sink smokes to match the ranged filename (note: both are pre-existing red on the integration branch for an unrelated reason — the driver hands the sink the drained spool path). Task-Id: T5 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * T6: exactly-once acceptance suite + smoke for incremental sink reads Add the LLP 0040 exactly-once proof the T4/T5 unit suites can't give (they stub storage): a deterministic acceptance test driving the REAL kernel cache, retention enforcer, maintainCache compaction, BOTH sinks (central forward + core local-fs blob), and the driver outbox respool. Covers: ~0 bytes on a no-new-rows tick, ~N on an N-new tick, exactly-once across a retention front-prune and a compaction generation swap for both sinks, and watermark vs. driver-outbox respool composition (suffix-only replay + idempotent batch-id / re-PUT). Adds a hermetic smoke proving the blob sink reads straight through a compaction generation swap via the real driver. Anchors LLP 0040 section 5 so the @ref resolves. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Task-Id: T6 * fix(central-forward): keep chunk batch-id stable across watermark advance The forward sink derived its X-Hyp-Batch-Id from the per-tick chunk ordinal (chunkIndex). After an earlier chunk was acked and the (sink, partition) watermark advanced, a respool re-read only the un-acked suffix and re-numbered it from 0 — minting a NEW batch-id for a chunk that may already have committed on the server (ambiguous ack / commit-then-5xx). The server idempotency ledger could not dedupe the redelivery, double-storing rows and breaking the spec's 'ledger covers mid-batch retries' guarantee. Key the batch-id on the chunk's start seq (the watermark it resumes from) instead. A respooled suffix reproduces the same [startSeq, body] and thus the same id, so the ledger dedupes it; distinct chunks still differ because each row's _hyp_ingest_seq is unique. Adds a cross-tick regression test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(incremental-sink-reads): close exactly-once holes in forward+blob sinks (#159 review) Dual-review found a BLOCKER plus majors against the exactly-once claim. Fixes, each with a regression test that fails before and passes after: [BLOCKER] Forward sink advanced the watermark per acked chunk to the scan's running-max `after`. Because the cache scan is NOT seq-ordered (interleaved live+backfill spool; post-compaction sortOrder re-sort), an early acked chunk could checkpoint past lower-seq rows still un-acked in a later chunk; a between-chunk failure then dropped them forever (`seq <= since`). Fix: advance the forward watermark ONCE at end-of-partition (as the blob sink already does), so a partial partition never checkpoints — a failure re-reads the whole partition and the server ledger dedupes the acked prefix (stable chunkStartSeq batch ids). test: central-forward-chunking "an unordered scan never skips a lower-seq row when a later chunk fails". [MAJOR] Legacy null-seq rows re-exported every tick (never reached steady state). Add `includeLegacy` to the storage read API (default true). Both sinks pass `includeLegacy = (no durable watermark)`: a fresh sink exports the pre-upgrade backlog once, then excludes null-seq rows. Safe because no new null-seq row can appear post-upgrade (decorateRow stamps every flushed row). tests: sink-incremental-acceptance pure-legacy + mixed, forward + blob (~0 bytes on 2nd tick, no row in two artifacts). [MAJOR] Central forward sink scoped watermarks per-PLUGIN, not per-instance — two @hypaware/central instances clobbered one watermark file. Switch to createInstanceWatermarkStore({ paths, instanceName }) (matching local-fs/s3); correct the @ref. test: sink-incremental-acceptance "two instances keep independent watermarks (no cross-instance skip)". [MAJOR, doc] Bounded-reads constraint (C): the read is a full scan over the surviving partition with a yielded-row filter, not the file/row-group skip the design implied. Correct LLP 0040 §1(C)/§5 to state reads are bounded by surviving-partition size, with O(N_new) reads pending null-aware icebird pushdown. [MINOR/NIT, doc+style] Correct LLP 0040 §4 (batchIdForChunk keys by chunkStartSeq, not chunkIndex). Hoist inline import('...') types to @import in sink-watermarks/acceptance tests and the compaction smoke. [MAJOR, ESCALATED — left unfixed] Mid-retry duplication when a unit commits, its watermark write is lost, AND new rows append before the retry: the resumed in-flight unit grows past what committed and the dedup net no longer recognizes it. The common in-flight retry (no new arrivals) IS covered. A correct fix needs a pre-commit intent (read `until` upper bound + persisted intent record) across both sinks — a design-level change to the watermark contract, deferred rather than half-patched under the exactly-once claim. Documented as LLP 0040 §6 risk #7 / §5 known gap. LLP 0040/0042 updated in this commit to match the new behavior. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Fix typecheck: root-anchor type-module imports in feature .js files The conflict-resolution merge of master (PR #196 type-surface cleanup) left four feature .js files importing type modules via sibling-relative paths (`./types.d.ts`). tsc copies the generated declaration into `types/` but does not emit input `.d.ts` siblings there, so `types/core/sinks/incremental.d.ts` had a dangling `import('./types.d.ts')` → TS2307 in `npm run typecheck` after `build:types` (CI-only; local `npm test` does not run tsc emit). Match master's convention (e.g. sinks/driver.js, cache/spool.js): .js files reference type modules by a repo-root-anchored path into src/ (`../../../src/core/sinks/types.js`), which resolves identically from both the source location and the emitted types/ location, plus `.js` extension for the kernel-types barrel. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This was referenced Jun 30, 2026
This was referenced Jul 23, 2026
This was referenced Jul 29, 2026
Merged
This was referenced Aug 6, 2026
This was referenced Aug 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Impact
The package no longer installs the OpenTelemetry npm stack. The developer telemetry contract remains available through the existing HypAware observability helpers.
Validation