Skip to content

wip struct field pushdown parquet - #18966

Closed
adriangb wants to merge 2 commits into
apache:mainfrom
pydantic:parquet-struct-pushdown
Closed

wip struct field pushdown parquet#18966
adriangb wants to merge 2 commits into
apache:mainfrom
pydantic:parquet-struct-pushdown

Conversation

@adriangb

Copy link
Copy Markdown
Contributor

Next step in #14993

need to figure out what to do with SchemaAdapter; there's a couple failing tests
Should we do this transformation before or after ExprAdapter?
@github-actionsgithub-actionsBot added core Core DataFusion crate datasource Changes to the datasource crate labels Nov 27, 2025
@adriangb

Copy link
Copy Markdown
ContributorAuthor

I have a better next step in #18998.

Essentially we migrate ParquetOpener to use PhysicalExprAdapterFactory fully and then all we have left is to implement the specialization for struct pushdown as a separate self-contained change.

github-merge-queueBot pushed a commit that referenced this pull request Dec 5, 2025
…8998)
Chips away at #14993 and
#16800
Changes made in this PR:
- Update DefaultPhysicalExprAdapter to handle struct column evolution
- Remove use of SchemaAdapter from row filter / predicate pushdown /
late materialization. I believe it was already not doing much here other
than where tests checked specific behavior and had not been updated to
use PhysicalExprAdapter
- Changed projection handling to use `PhysicalExprAdapter` instead of
`SchemaAdapter`
- Kept intermediary `Vec<usize>` so we can use `ProjectionMask::roots`
and punt the complexity of implementing `ProjectionExprs` ->
`ProjectionMask` until a later PR (there is a draft in #18966 of what
that might look like).
@github-actions

Copy link
Copy Markdown

Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.

@github-actionsgithub-actionsBot added the Stale PR has not had any activity for some time label Feb 3, 2026
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

coreCore DataFusion cratedatasourceChanges to the datasource crateStalePR has not had any activity for some time

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@adriangb