Skip to content

Fix predicate pushdown bugs: project columns within DatafusionArrowPredicate (#4005) (#4006) - #4021

Merged
tustvold merged 5 commits into
apache:masterfrom
tustvold:project-parquet-filter-columns
Oct 30, 2022
Merged

Fix predicate pushdown bugs: project columns within DatafusionArrowPredicate (#4005) (#4006)#4021
tustvold merged 5 commits into
apache:masterfrom
tustvold:project-parquet-filter-columns

Conversation

@tustvold

@tustvoldtustvold commented Oct 29, 2022•
edited
Loading

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes#4005
Closes#4006

Rationale for this change

See tickets

What changes are included in this PR?

Project batch in DatafusionArrowPredicate according to the order expected by the PhysicalExpr

Are there any user-facing changes?

No

@github-actionsgithub-actionsBot added the core Core DataFusion crate label Oct 29, 2022
@tustvold
tustvold marked this pull request as ready for review October 30, 2022 01:50
alamb
alamb approved these changes Oct 30, 2022
View reviewed changes

@alambalamb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM -- thank you @tustvold 🙏 -- I am working on additional test coverage of this feature in #3976 so I feel pretty good about this PR

@alamb

alamb commented Oct 30, 2022

Copy link
Copy Markdown
Contributor

I'll plan to merge this in tomorrow unless anyone else would like to comment

@alambalamb changed the title Project columns within DatafusionArrowPredicate (#4005) (#4006)Fix predicate pushdown bugs: project columns within DatafusionArrowPredicate (#4005) (#4006)Oct 30, 2022
@alamb

alamb commented Oct 30, 2022

Copy link
Copy Markdown
Contributor

This PR had a conflict after merging #3989

Screen Shot 2022-10-30 at 7 44 22 AM

I took the liberty of fixing the merge conflict and pushing the fix.

@tustvold
tustvold merged commit 35f786b into apache:masterOct 30, 2022
@ursabot

ursabot commented Oct 30, 2022

Copy link
Copy Markdown

Benchmark runs are scheduled for baseline = f4d70ac and contender = 35f786b. 35f786b is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ec2-t3-xlarge-us-east-2] ec2-t3-xlarge-us-east-2
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on test-mac-arm] test-mac-arm
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ursa-i9-9960x] ursa-i9-9960x
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ursa-thinkcentre-m75q] ursa-thinkcentre-m75q
Buildkite builds:
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

jimexist pushed a commit to jimexist/arrow-datafusion that referenced this pull request Oct 31, 2022
@tustvold@alamb@jimexist
Fix predicate pushdown bugs: project columns within DatafusionArrowPr…
ed27e31
…edicate (apache#4005) (apache#4006) (apache#4021)
* Project columns within DatafusionArrowPredicate (apache#4005) (apache#4006)
* Add test
* Format
* Fix merge blunder
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
@alamb

alamb commented Oct 31, 2022

Copy link
Copy Markdown
Contributor

I found another related bug: #4046

@tustvoldtustvold mentioned this pull request Oct 31, 2022
Merged
Dandandan pushed a commit to yuuch/arrow-datafusion that referenced this pull request Nov 5, 2022
@tustvold@alamb@Dandandan
Fix predicate pushdown bugs: project columns within DatafusionArrowPr…
f9bff57
…edicate (apache#4005) (apache#4006) (apache#4021)
* Project columns within DatafusionArrowPredicate (apache#4005) (apache#4006)
* Add test
* Format
* Fix merge blunder
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@alambalambalamb approved these changes

Assignees

No one assigned

Labels

coreCore DataFusion crate

Projects

None yet

Milestone

No milestone

3 participants

@tustvold@alamb@ursabot