Add experimental VECTOR repetition type for fixed-size vectors - #1
Draft
rok wants to merge 1 commit into
Draft
Conversation
rokforce-pushed
the
vector-repetition-type
branch
2 times, most recently
from
June 22, 2026 15:04
0dacf96 to
513b777Comparerokforce-pushed
the
vector-repetition-type
branch
4 times, most recently
from
June 30, 2026 14:06
b8ac05c to
ba4c2c3CompareFixed-shape vector data (embeddings, image/tensor patches, fixed-size decimal vectors) currently round-trips through Parquet's 3-level LIST encoding, paying per-element repetition levels for a shape that is already known from the schema. This adds the proposed opt-in VECTOR logical and repetition types for fixed-size vectors. This is aligned with the current parquet-format proposal (apache/parquet-format PR apache#592) and the arrow-go prototype (apache/arrow-go PR apache#854): <required|optional> group v (VECTOR) { vector group list [N] { <required|optional> <element-type> element; } } Format (parquet-format-structures): - Vendor parquet.thrift (parquet-format 2.13.0) under src/main/thrift and generate from it instead of unpacking the parquet-format jar, adding FieldRepetitionType.VECTOR = 3, VectorType / LogicalType.VECTOR = 19, and SchemaElement.vector_length (thrift field id 11). These are not yet ratified in apache/parquet-format, so files written with VECTOR are not readable by implementations that do not understand the repetition type. Schema (parquet-column): - Add LogicalTypeAnnotation.vectorType() and Type.Repetition.VECTOR for the middle group. GroupType carries vectorLength for VECTOR groups; the outer group carries the VECTOR logical type. - Types builder gains vector(type, length), optionalVector(type, length), and group vectorLength(int) helpers. MessageTypeParser and the schema printer support the middle-group "vector group list [N]" syntax. - VECTOR contributes no repetition or definition level; nullable vectors are represented by the optional outer group and nullable elements by the optional element field. - The current experimental implementation intentionally limits VECTOR elements to primitive types and rejects bare VECTOR repetition groups outside the outer VECTOR logical type. Read/write (parquet-column): - The column writer counts a row every vector_length element slots and keeps pages on whole-vector boundaries. Statistics, encodings and bloom filters remain element-level. - The record writer expands null vectors to vector_length null slots. The record reader materializes vector_length contiguous element groups per vector, and column-index/row-range seeking advances by the vector stride. - VECTOR chunk/page value counts are validated against row_count * vector_length where row counts are available, and partial-vector record writes are rejected even when validation is disabled. - The example object model (SimpleGroup) supports VECTOR group cardinality. Metadata (parquet-hadoop): - ParquetMetadataConverter round-trips the VECTOR logical type on the outer group and vector_length on the VECTOR middle group. Arrow (parquet-arrow): - Opt-in SchemaConverter(convertInt96, vectorEncoding) maps an eligible top-level non-nullable Arrow FixedSizeList of a non-nullable fixed-width primitive to a 3-level VECTOR group, and back; ineligible cases fall back to LIST. Tests cover schema parse/print, builder validation, level computation, metadata round-trip, malformed VECTOR schema rejection, end-to-end write/read round-trips (including nullable vectors/elements, page alignment, element-level statistics, FLBA elements and filtered reads), and Arrow FixedSizeList <-> VECTOR mapping.
This pull request has been automatically marked as stale because it has had no activity for at least 2 months. If you are still working on this change or plan to move it forward, please leave a comment or push a new commit so we know to keep it open. Otherwise, this PR will be closed automatically in about one month. Thank you for your contribution to Apache Parquet! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixed-shape vector data (embeddings, image/tensor patches, fixed-size decimal vectors) currently round-trips through Parquet's 3-level LIST encoding, paying per-element repetition and definition levels for a shape that is already known from the schema. This adds a new, opt-in, self-describing VECTOR repetition type that stores a dense, non-nullable fixed-size vector as a single primitive leaf with no inner levels.
This is a faithful port of the reference prototypes in arrow-rs (PR apache#7) and arrow-go (PR apache#854).
Format (parquet-format-structures):
Schema (parquet-column):
Read/write (parquet-column):
Metadata (parquet-hadoop):
Arrow (parquet-arrow):
Tests cover schema parse/print, builder validation, level computation, metadata round-trip, end-to-end write/read round-trips (incl. page alignment, statistics suppression, FLBA elements and filtered reads), and the Arrow FixedSizeList <-> VECTOR mapping.