Skip to content

Add partition by constructs in window functions and modify logical planning - #501

Merged
alamb merged 2 commits into
apache:masterfrom
jimexist:add-partition-by
Jun 9, 2021
Merged

Add partition by constructs in window functions and modify logical planning#501
alamb merged 2 commits into
apache:masterfrom
jimexist:add-partition-by

Conversation

@jimexist

@jimexistjimexist commented Jun 4, 2021

Copy link
Copy Markdown
Member

Which issue does this PR close?

Add partition by constructs and modify logical planning.

Partly contributes to #299
Based on #492 so review that first

Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

@codecov-commenter

codecov-commenter commented Jun 4, 2021

Copy link
Copy Markdown

Codecov Report

Merging #501 (595fe99) into master (8495f95) will decrease coverage by 0.06%.
The diff coverage is 57.50%.

Impacted file tree graph

@@ Coverage Diff @@## master #501 +/- ##
==========================================
- Coverage 76.09% 76.03% -0.07% 
==========================================
Files 157 157 Lines 26913 26990 +77 ==========================================
+ Hits 20480 20521 +41 - Misses 6433 6469 +36 
Impacted FilesCoverage Δ
...sta/rust/core/src/serde/logical_plan/from_proto.rs35.20% <0.00%> (-0.21%)⬇️
...lista/rust/core/src/serde/logical_plan/to_proto.rs61.34% <0.00%> (-0.31%)⬇️
...ta/rust/core/src/serde/physical_plan/from_proto.rs37.71% <0.00%> (-0.80%)⬇️
datafusion/src/optimizer/utils.rs45.63% <0.00%> (-2.42%)⬇️
datafusion/src/logical_plan/expr.rs84.28% <70.00%> (-0.28%)⬇️
datafusion/src/sql/planner.rs84.62% <81.57%> (+0.11%)⬆️
datafusion/src/sql/utils.rs68.85% <88.23%> (+1.23%)⬆️
datafusion/src/logical_plan/plan.rs81.50% <100.00%> (+0.43%)⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8495f95...595fe99. Read the comment docs.

@jimexist
jimexist marked this pull request as ready for review June 4, 2021 15:28
@jimexistjimexist changed the title Add partition by constructs and modify logical planningAdd partition by constructs and modify logical planningJun 4, 2021
@jimexistjimexist changed the title Add partition by constructs and modify logical planningAdd partition by constructs in window functions and modify logical planningJun 4, 2021
Comment threaddatafusion/src/sql/planner.rs Outdated
let sql = "SELECT order_id, MAX(qty) OVER (PARTITION BY order_id) from orders";
let expected = "\
Projection: #order_id, #MAX(qty)\
\n WindowAggr: windowExpr=[[MAX(#qty)]] partitionBy=[]\

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems like order_id should appear in the partitionBy list

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me remove the list printing altogether for now to keep parity with psql

I can always add it back later but for now the partition information is already captured by the child sort by plan

@jimexist
jimexistforce-pushed the add-partition-by branch 2 times, most recently from b6c6023 to c2bfd60CompareJune 7, 2021 12:03
Comment threaddatafusion/src/optimizer/utils.rs Outdated
@alamb

alamb commented Jun 7, 2021

Copy link
Copy Markdown
Contributor

This is on my review queue for tomorrow

@alambalamb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @jimexist -- I think this PR looks reasonable. I had two suggestions I think are worth considering (handling errors in planner and including PARTITION BY in the explain plan)

I think it would also be fine to make those changes in a follow on PR if that would be easier for you

Comment threaddatafusion/src/optimizer/utils.rs Outdated
Comment threaddatafusion/src/optimizer/utils.rs Outdated
Comment threaddatafusion/src/sql/planner.rs Outdated
.map(|window_frame| window_frame.clone().try_into())
.transpose()?;
let fun = window_functions::WindowFunction::from_str(&name);
if let Ok(window_functions::WindowFunction::AggregateFunction(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I may be misreading this, but it looks like the error message may have gotten lost.

Stylistically, this might be cleaner using a match rather than an if/else chain (and the compiler will tell you if you missed a case) something like

let fun = window_functions::WindowFunction::from_str(&name)?;// note question markuse window_functions::WindowFunction::*;match fun {AggregateFunction(aggregate_fun) =>{ .. code .. },BuiltInWindowFunction(window_fun) =>{ .. code .. },}

Perhaps?

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's a good point. i was previously thinking maybe deferring the else the subsequent parsing part but then it's with over clause so the allowed listed of function names shall be known to be limited.

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is cool now to see the compiler ensuring all cases are covered.

format!("{:?}", err)
);
fn over_partition_by() {
let sql = "SELECT order_id, MAX(qty) OVER (PARTITION BY order_id) from orders";

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think including the PARTITION BY information somewhere in this plan would be valuable -- maybe it could be added to WindowExpr formatting?

I may be missing a reason to not include it as well

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think it can be added in a subsequent PR, where i can revisit all the planning related printing (to be less verbose perhaps)

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's address in #526

@alamb
alamb merged commit d5bca0e into apache:masterJun 9, 2021
@jimexist
jimexist deleted the add-partition-by branch June 10, 2021 01:49
@houqphouqp added ballista enhancement New feature or request labels Jul 31, 2021
unkloud pushed a commit to unkloud/datafusion that referenced this pull request Mar 23, 2025
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancementNew feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@jimexist@codecov-commenter@alamb@houqp