implement lead and lag built-in window function

[jimexist]

Which issue does this PR close?

implement lead and lag built-in window function.

based on #520 so review that first

Closes#553

Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

[codecov-commenter]

Codecov Report

Merging #429 (f763205) into master (321fda4) will decrease coverage by 0.01%.
The diff coverage is 65.47%.

@@ Coverage Diff @@## master #429 +/- ##
==========================================
- Coverage 75.16% 75.15% -0.02% 
==========================================
Files 150 152 +2 Lines 25144 25357 +213 ==========================================
+ Hits 18899 19056 +157 - Misses 6245 6301 +56 

Impacted Files	Coverage Δ
ballista/rust/client/src/columnar_batch.rs	0.00% <ø> (ø)
ballista/rust/core/src/serde/scheduler/mod.rs	14.28% <ø> (ø)
datafusion/src/physical_plan/expressions/mod.rs	71.42% <ø> (ø)
...tafusion/src/physical_plan/expressions/lead_lag.rs	38.29% <38.29%> (ø)
...afusion/src/physical_plan/expressions/nth_value.rs	70.76% <70.76%> (ø)
datafusion/src/physical_plan/windows.rs	78.32% <72.11%> (+4.54%)	⬆️
datafusion/src/physical_plan/mod.rs	79.09% <100.00%> (+0.38%)	⬆️
datafusion/tests/sql.rs	99.89% <100.00%> (ø)
... and 2 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 321fda4...f763205. Read the comment docs.

[alamb]

I plan to review this PR tomorrow

[jimexist]

Actually let's park this pull request for a while - I plan to implement sort and partition first and then window frame, after which the window shift approach might not be relevant.

[jimexist]

Actually let's park this pull request for a while - I plan to implement sort and partition first and then window frame, after which the window shift approach might not be relevant.

now that #520 is implemented, this PR is ready

[jimexist]

putting this back to draft as this relies on apache/arrow-rs#388 which is not yet in arrow 4.3

[alamb]

putting this back to draft as this relies on apache/arrow-rs#388 which is not yet in arrow 4.3

Oh no!

Can we possibly use the API that is in Arrow 4.3 (and then we can upgrade datafusion to use the new api when the next version of Arrow comes out)?

[jimexist]

putting this back to draft as this relies on apache/arrow-rs#388 which is not yet in arrow 4.3

Oh no!

Can we possibly use the API that is in Arrow 4.3 (and then we can upgrade datafusion to use the new api when the next version of Arrow comes out)?

I don't mind parking this one here for a while since there would be many other window frame stuff to be done before revisiting this and by that time newer version would be released

[alamb]

I don't mind parking this one here for a while since there would be many other window frame stuff to be done before revisiting this and by that time newer version would be released

Ok, thank you. The plan is to do a 4.4 release in ~ 2 weeks

[jimexist]

@alamb and @Dandandan this pull request is ready now

[alamb]

Looks like a great start @jimexist -- I do have a question if this will generate the correct answer with multiple partitions.

Also, I suggest an end-to-end integration test using your great harness, but I suspect you plan to do so in a subsequent PR :) 👍

[alamb]

do you need to restrict the window to the partition bounds? If the input array had 10 rows in 2 partitions, wouldn't this code produce 2 output partitions of 10 rows each (rather than 2 output partitions of 5 rows each)?

[jimexist]

@alamb good catch, this is fixed and add with integration tests.

[alamb]

Thanks @jimexist -- I ran out of time today but will check this out tomorrow

[alamb]

Thanks @jimexist !