Uh oh!
There was an error while loading. Please reload this page.
[codex] Add Codex metadata projection - #21
Merged
Conversation
philcunliffe
marked this pull request as ready for review
May 22, 2026 19:28
Uh oh!
There was an error while loading. Please reload this page.
2 tasks
philcunliffe added a commit
that referenced
this pull request
Jun 11, 2026
…91) * feat(sinks): partition iceberg exports by day with conversation sort Lay out @hypaware/format-iceberg exports for an archive's job, not the cache's: partition by day(primaryTimestampColumn) — a writer-owned default, not the cache's conversation_id-identity cachePartitioning, which sets an unbounded ~1-file-per-conversation floor compaction can't beat — and sort each day partition by the dataset's lookup columns (conversation_id-led) so a conversation lookup prunes row groups by min/max instead of needing a partition per conversation. - Promote partitionSpecForDeclaration + validatePartitionSpecStability (and the declaration type) from src/core/cache/iceberg to a shared src/core/iceberg home, re-exported from src/core/index.js: they are core surface consumed by the registry, cache, plugin types, and now the export (LLP 0003). - format-iceberg derives the day grain + sort order per dataset at commit time, creates the table with both, and rejects partition-spec drift on append (iceberg_partition_spec_drift). Emits hyp_partition_spec and hyp_sort_order on commit spans. - Reframe maintenance compaction: available via icebergRewrite but not run in-daemon and not needed for a day grain (was "blocked by icebird"). Spec: LLP 0022 (rewritten from the abandoned cache-parity decision); xrefs in LLP 0014 and 0003. Tests: 10 (derivation + drift through the real icebird write path) plus a passing iceberg_export_partitioned_local_fs smoke asserting the layout and hyp_partition_spec. Clustering (icebird #22) and read pruning (#20/#21) require a published icebird containing commit 3edb15b; the package.json pin must move off 0.8.5 before those benefits land. The code degrades gracefully on 0.8.5 — partitioning and drift work; the sort order is recorded but inert. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(deps): bump icebird 0.8.5 -> 0.8.9 (sort-on-write + scan pruning) Activates the conversation sort within day partitions and read-side scan pruning that the partitioned export records in metadata. Clears the merge blocker: 0.8.9 contains icebird 3edb15b. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(sinks): address PR #91 review — reverse drift guard, on-disk sort assertions, CLI text - commitBatch now rejects reverse partition-spec drift: appending with no derived partitioning onto an already-partitioned table throws iceberg_partition_spec_drift instead of silently skipping the guard (and mislabeling spans as unpartitioned). LLP 0022#drift-rejection updated to record the guard as bidirectional; new test alongside the forward-drift case. - The conversation sort is now asserted on disk, not just in metadata: both the commitBatch integration test and the partitioned smoke read a day-partition parquet file back with hyparquet and assert conversation_id row order — fails on an icebird that records the sort order but writes unsorted. Verified against icebird 0.8.9. - hyp sink maintain CLI text reframed to match maintenance.js / LLP 0022: compaction is not run by this sink (out-of-band via icebergRewrite), not 'unsupported by icebird'; action label is now compaction_out_of_band. - Removed stray </content> artifact from the end of LLP 0022. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Validation