Uh oh!
There was an error while loading. Please reload this page.
pipe column orderings into pruning predicate creation - #15821
Conversation
| let pruning_predicate = PruningPredicate::try_new( | ||
| Arc::clone(predicate), | ||
| self.schema(), | ||
| vec![ColumnOrdering::Unknown; self.schema().fields().len()], | ||
| )?; |
There was a problem hiding this comment.
We could add a new signature to avoid API churn, but I wanted to make it explicit for now to see all of the callsites
3e15afe to
8c2ceb1Compareetseidl
commented
Apr 25, 2025
@adriangb please check out pydantic#28 |
| return expr; | ||
| } | ||
| // Special handlng for floats. Because current Parquet statistics do not allow NaN, and |
There was a problem hiding this comment.
This block is why we need to get the column ordering info passed down. Here we know which column is being pruned and with which operation. For floats we can disallow pruning because the Parquet stats are incomplete. We can also skip pruning if the stats are not valid because an ordering is not defined for the type.
adriangb
commented
Jul 20, 2025
@etseidl I'm sorry we haven't made any progress here. I see there are a lot of merge conflicts but we do now finally have all of the building blocks in place since we evaluate predicates against the physical file schema. Should we revive this? |
etseidl
commented
Jul 20, 2025
I'll be able to revisit this early in the week. It's been a back burner issue for me lately because of the slow progress on parquet format changes. It seems the sort order keeps popping up, though, so I think we'll want this change eventually. |
alamb
commented
Sep 5, 2025
marking as draft as this doesn't appear to be waiting on review |
Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days. |
NaNis present #15812