Skip to content

reuse datafusion physical planner in ballista building from protobuf - #532

Merged
alamb merged 4 commits into
apache:masterfrom
jimexist:reuse-planner-building
Jun 11, 2021
Merged

reuse datafusion physical planner in ballista building from protobuf#532
alamb merged 4 commits into
apache:masterfrom
jimexist:reuse-planner-building

Conversation

@jimexist

@jimexistjimexist commented Jun 10, 2021

Copy link
Copy Markdown
Member

Which issue does this PR close?

Closes #.

Rationale for this change

reuse datafusion physical planner in ballista building from protobuf.

so far there are duplicated code within ballista and datafusion planner because the latter needs to handle alias but for ballista the expr name is flattened in protobuf, resulting two similar codepath in building the aggregation exec.

this tries to unify both and reuse code.

What changes are included in this PR?

Are there any user-facing changes?

@jimexistjimexist changed the title use logical planner in ballista building from protobufuse datafusion physical planner in ballista building from protobufJun 10, 2021
@jimexistjimexist changed the title use datafusion physical planner in ballista building from protobufreuse datafusion physical planner in ballista building from protobufJun 10, 2021
@codecov-commenter

codecov-commenter commented Jun 10, 2021

Copy link
Copy Markdown

Codecov Report

Merging #532 (d0f2804) into master (8f84564) will increase coverage by 0.12%.
The diff coverage is 66.21%.

Impacted file tree graph

@@ Coverage Diff @@## master #532 +/- ##
==========================================
+ Coverage 76.04% 76.17% +0.12% 
==========================================
Files 157 156 -1 Lines 27098 27017 -81 ==========================================
- Hits 20608 20580 -28 + Misses 6490 6437 -53 
Impacted FilesCoverage Δ
...ta/rust/core/src/serde/physical_plan/from_proto.rs39.83% <47.05%> (+2.11%)⬆️
datafusion/src/physical_plan/planner.rs78.79% <61.90%> (-1.41%)⬇️
datafusion/src/physical_plan/windows.rs86.46% <100.00%> (+0.24%)⬆️
...sta/rust/core/src/serde/logical_plan/from_proto.rs35.04% <0.00%> (-0.17%)⬇️
datafusion/src/physical_plan/window_frames.rs
datafusion/src/sql/planner.rs84.85% <0.00%> (+0.23%)⬆️
datafusion/src/logical_plan/window_frames.rs89.39% <0.00%> (+2.78%)⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8f84564...d0f2804. Read the comment docs.

@alambalamb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like a nice change to me -- but I think someone more familiar with Ballista to review it too -- @andygrove / @edrevo can you suggest someone?

@jimexist
jimexistforce-pushed the reuse-planner-building branch from 6ff2361 to d0f2804CompareJune 10, 2021 15:16
@andygrove

andygrove commented Jun 10, 2021 via email

Copy link
Copy Markdown
Member

Comment on lines +68 to +70
_partition_by: &[Arc<dyn PhysicalExpr>],
_order_by: &[PhysicalSortExpr],
_window_frame: WindowFrame,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: once this is merged, it would be cool to add a link to this code in the respective issues that are tracking these missing features to make it easier for new contributors to start contributing.

@edrevoedrevoJun 11, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just realized that this PR changes the behavior when there is a non-empty partition_by/order_by/window_frame: before (at least in ballista) it would error, whereas now it is silently ignored. Maybe it is worth erroring if they aren't empty to make it explicit that there is no support?

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it'll be further developed in #520

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

but that's a good point let me add guard here.

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is fixed now

@jimexist
jimexistforce-pushed the reuse-planner-building branch from d0f2804 to 45b87f9CompareJune 11, 2021 10:02

@andygroveandygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I pulled the branch locally and ran the integration tests.

@alamb
alamb merged commit ad70a1e into apache:masterJun 11, 2021
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants

@jimexist@codecov-commenter@andygrove@alamb@houqp@edrevo