Uh oh!
There was an error while loading. Please reload this page.
Bump hyparquet-writer to 0.16.6 for the dictionary-encoding fix - #712
Merged
Conversation
…olumns Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bumps the optional
hyparquet-writerdep 0.16.5 -> 0.16.6, the release with the dictionary-encoding fix (hyparam/hyparquet-writer#35): the writer now keeps any dictionary that at least halves the encoded bytes instead of busting to PLAIN at the 1 MiB page cap. Server-side counterpart already merged (hyparam/hypaware-server#309); this covers the client's cache-tier writes.The override:
icebird0.8.17 pinshyparquet-writerat exactly 0.16.5, and the cache write path goes through icebird, so a direct bump alone would leave a nested 0.16.5 where it matters. The override is scoped to icebird only;hypvectorkeeps its own 0.16.1 pin untouched. Drop the entry when icebird releases with its own bump.Test updates: two
format-parquet-clusteringtests asserted the old writer's dictionary bust (it was the documented motivation for row-group clustering). They now assert the fixed behavior: the wide per-conversation column staysRLE_DICTIONARYwith or without clustering, and clustering costs no material size (its remaining jobs are the peak-heap byte cap and read locality, per the updated rationale comment in the plugin). The old bust shape is kept as a regression canary.Validation: typecheck green; 3,939/3,942 tests pass. The one failure (
leave tears down a central layer whose active-slot pointer does not resolve (#623)) fails identically on unmodified master in two separate local checkouts, so it is pre-existing and unrelated (possibly macOS-specific).🤖 Generated with Claude Code