Uh oh!
There was an error while loading. Please reload this page.
allow filters referencing parquet struct columns to be pushed down - #19556
Closed
adriangb wants to merge 17 commits into
Closed
allow filters referencing parquet struct columns to be pushed down#19556adriangb wants to merge 17 commits into
adriangb wants to merge 17 commits into
Conversation
adriangb
commented
Jan 7, 2026
| ScalarValue::try_new_null(&DataType::Map(map_field_ref, false)).unwrap(), | ||
| ScalarValue::try_new_null(&DataType::Union( | ||
| UnionFields::new(vec![42], vec![field_ref]), | ||
| UnionFields::try_new(vec![42], vec![field_ref]).unwrap(), |
ContributorAuthor
There was a problem hiding this comment.
Note that this is just handling upstream changes in arrow, not actually related to this PR
adriangbforce-pushed
the
parquet-struct-filter
branch
2 times, most recently
from
January 7, 2026 21:04
8b0bd72 to
7568dedCompareadriangbforce-pushed
the
parquet-struct-filter
branch
from
January 8, 2026 14:07
7568ded to
39b37afCompareadriangb
commented
Jan 8, 2026
ContributorAuthor
There's some test failures but I think those are all mostly unrelated to this change and are issues that need fixing with the update to arrow-rs/main. |
this does *not* benchmark (or implement) reading a single field of a struct column. this only tests being able to late materialize a struct columns *as a whole*
adriangbforce-pushed
the
parquet-struct-filter
branch
from
January 11, 2026 20:46
f56c2ad to
65426eaCompareadriangb
commented
Jan 27, 2026
ContributorAuthor
Closing in favor of #19538 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A step towards #11745.
Depends on upstream arrow fixes..
Note that this does not do anything fancy with parquet struct access like actually generating the leaf mask to read only some fields, it reads the entire struct column and process it in memory much like currently happens. That should be an easy followup though.
Benchmark numbers: