Skip to content

finish implementation of window accumulation and func row_number() and add sql.rs test - #413

Closed
jimexist wants to merge 1 commit into
apache:masterfrom
jimexist:done-impl-row-number
Closed

finish implementation of window accumulation and func row_number() and add sql.rs test#413
jimexist wants to merge 1 commit into
apache:masterfrom
jimexist:done-impl-row-number

Conversation

@jimexist

@jimexistjimexist commented May 24, 2021

Copy link
Copy Markdown
Member

Which issue does this PR close?

Closes#298 .

Rationale for this change

with #381 we can finally close up this. based on #375 and #403 so review these two first.

What changes are included in this PR?

Are there any user-facing changes?

@codecov-commenter

Copy link
Copy Markdown

Codecov Report

Merging #413 (9ba25a0) into master (7359e4b) will increase coverage by 0.57%.
The diff coverage is 81.81%.

Impacted file tree graph

@@ Coverage Diff @@## master #413 +/- ##
==========================================
+ Coverage 74.85% 75.43% +0.57% 
==========================================
Files 146 148 +2 Lines 24498 24744 +246 ==========================================
+ Hits 18339 18666 +327 + Misses 6159 6078 -81 
Impacted FilesCoverage Δ
datafusion/src/physical_plan/expressions/mod.rs71.42% <ø> (ø)
datafusion/src/physical_plan/hash_aggregate.rs85.21% <ø> (ø)
datafusion/src/physical_plan/sort.rs92.07% <ø> (ø)
datafusion/src/physical_plan/mod.rs80.58% <68.75%> (-2.18%)⬇️
...fusion/src/physical_plan/expressions/row_number.rs69.23% <69.23%> (ø)
datafusion/src/physical_plan/window_functions.rs87.68% <70.58%> (-1.05%)⬇️
...afusion/src/physical_plan/expressions/nth_value.rs70.76% <70.76%> (ø)
datafusion/src/physical_plan/windows.rs84.00% <90.98%> (+84.00%)⬆️
datafusion/src/execution/context.rs92.07% <100.00%> (+0.02%)⬆️
datafusion/src/physical_plan/planner.rs80.45% <100.00%> (+3.94%)⬆️
... and 14 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7359e4b...9ba25a0. Read the comment docs.

commit 7fb3640
Author: Jiayu Liu <jiayu.liu@airbnb.com>
Date: Fri May 21 16:38:25 2021 +0800
row number done
commit 1723926
Author: Jiayu Liu <jiayu.liu@airbnb.com>
Date: Fri May 21 16:05:50 2021 +0800
add row number
commit bf5b8a5
Author: Jiayu Liu <jiayu.liu@airbnb.com>
Date: Fri May 21 15:04:49 2021 +0800
save
commit d2ce852
Author: Jiayu Liu <jiayu.liu@airbnb.com>
Date: Fri May 21 14:53:05 2021 +0800
add streams
commit 0a861a7
Author: Jiayu Liu <jiayu.liu@airbnb.com>
Date: Thu May 20 22:28:34 2021 +0800
save stream
commit a9121af
Author: Jiayu Liu <jiayu.liu@airbnb.com>
Date: Thu May 20 22:01:51 2021 +0800
update unit test
commit 2af2a27
Author: Jiayu Liu <jiayu.liu@airbnb.com>
Date: Fri May 21 14:25:12 2021 +0800
fix unit test
commit bb57c76
Author: Jiayu Liu <jiayu.liu@airbnb.com>
Date: Fri May 21 14:23:34 2021 +0800
use upper case
commit 5d96e52
Author: Jiayu Liu <jiayu.liu@airbnb.com>
Date: Fri May 21 14:16:16 2021 +0800
fix unit test
commit 1ecae8f
Author: Jiayu Liu <jiayu.liu@airbnb.com>
Date: Fri May 21 12:27:26 2021 +0800
fix unit test
commit bc2271d
Author: Jiayu Liu <jiayu.liu@airbnb.com>
Date: Fri May 21 10:04:29 2021 +0800
fix error
commit 880b94f
Author: Jiayu Liu <jiayu.liu@airbnb.com>
Date: Fri May 21 08:24:00 2021 +0800
fix unit test
commit 4e792e1
Author: Jiayu Liu <jiayu.liu@airbnb.com>
Date: Fri May 21 08:05:17 2021 +0800
fix test
commit c36c04a
Author: Jiayu Liu <jiayu.liu@airbnb.com>
Date: Fri May 21 00:07:54 2021 +0800
add more tests
commit f5e64de
Author: Jiayu Liu <jiayu.liu@airbnb.com>
Date: Thu May 20 23:41:36 2021 +0800
update
commit a1eae86
Author: Jiayu Liu <jiayu.liu@airbnb.com>
Date: Thu May 20 23:36:15 2021 +0800
enrich unit test
commit 0d2a214
Author: Jiayu Liu <jiayu.liu@airbnb.com>
Date: Thu May 20 23:25:43 2021 +0800
adding filter by todo
commit 8b486d5
Author: Jiayu Liu <jiayu.liu@airbnb.com>
Date: Thu May 20 23:17:22 2021 +0800
adding more built-in functions
commit abf08cd
Author: Jiayu Liu <Jimexist@users.noreply.github.com>
Date: Thu May 20 22:36:27 2021 +0800
Update datafusion/src/physical_plan/window_functions.rs
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
commit 0cbca53
Author: Jiayu Liu <Jimexist@users.noreply.github.com>
Date: Thu May 20 22:34:57 2021 +0800
Update datafusion/src/physical_plan/window_functions.rs
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
commit 831c069
Author: Jiayu Liu <Jimexist@users.noreply.github.com>
Date: Thu May 20 22:34:04 2021 +0800
Update datafusion/src/logical_plan/builder.rs
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
commit f70c739
Author: Jiayu Liu <Jimexist@users.noreply.github.com>
Date: Thu May 20 22:33:04 2021 +0800
Update datafusion/src/logical_plan/builder.rs
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
commit 3ee87aa
Author: Jiayu Liu <jiayu.liu@airbnb.com>
Date: Wed May 19 22:55:08 2021 +0800
fix unit test
commit 5c4d92d
Author: Jiayu Liu <jiayu.liu@airbnb.com>
Date: Wed May 19 22:48:26 2021 +0800
fix clippy
commit a0b7526
Author: Jiayu Liu <jiayu.liu@airbnb.com>
Date: Wed May 19 22:46:38 2021 +0800
fix unused imports
commit 1d3b076
Author: Jiayu Liu <jiayu.liu@airbnb.com>
Date: Thu May 13 18:51:14 2021 +0800
add window expr
@jimexist
jimexistforce-pushed the done-impl-row-number branch from 9ba25a0 to d567705CompareMay 25, 2021 01:47
@jimexist
jimexist marked this pull request as draft May 25, 2021 05:14
@jimexist

Copy link
Copy Markdown
MemberAuthor

converting back to draft because most changes are already incorporated in #375

.map_err(DataFusionError::into_arrow_external_error)?;

for i in 0..(schema.fields().len() - window_expr.len()) {
let col = concat(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess this is only needed for the empty over clause right, as in other cases we probably only want to access the rows.
Also for functions like RowNumber we could be streaming/emit batches instead of "caching" them?

@jimexist
jimexist deleted the done-impl-row-number branch May 26, 2021 05:28
unkloud pushed a commit to unkloud/datafusion that referenced this pull request Mar 23, 2025
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support window functions with empty OVER clause

3 participants

@jimexist@codecov-commenter@Dandandan