Skip to content

optimize window function's find_ranges_in_range - #595

Merged
alamb merged 1 commit into
apache:masterfrom
jimexist:window-partition-optimize-ranges
Jun 23, 2021
Merged

optimize window function's find_ranges_in_range#595
alamb merged 1 commit into
apache:masterfrom
jimexist:window-partition-optimize-ranges

Conversation

@jimexist

@jimexistjimexist commented Jun 22, 2021

Copy link
Copy Markdown
Member

Which issue does this PR close?

Optimize window function's find_ranges_in_range

Based on #564 so review that first
Closes #.

Rationale for this change

benchmark:

Benchmarking window empty over, aggregate functions: Collecting 100 samples in estimated 7.0674 s (200 iterati window empty over, aggregate functions
time: [33.115 ms 33.588 ms 34.129 ms]
change: [-17.056% -15.622% -14.056%] (p = 0.00 < 0.05)
Performance has improved.
Found 5 outliers among 100 measurements (5.00%)
4 (4.00%) high mild
1 (1.00%) high severe
Benchmarking window empty over, built-in functions: Collecting 100 samples in estimated 5.9518 s (200 iteratio window empty over, built-in functions
time: [27.812 ms 28.113 ms 28.429 ms]
change: [-23.131% -21.112% -19.100%] (p = 0.00 < 0.05)
Performance has improved.
Found 3 outliers among 100 measurements (3.00%)
3 (3.00%) high mild
Benchmarking window order by, aggregate functions: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 20.8s, or reduce sample count to 20.
Benchmarking window order by, aggregate functions: Collecting 100 samples in estimated 20.778 s (100 iteration window order by, aggregate functions
time: [191.15 ms 196.85 ms 202.61 ms]
change: [-13.162% -10.553% -7.7583%] (p = 0.00 < 0.05)
Performance has improved.
Benchmarking window order by, built-in functions: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 17.9s, or reduce sample count to 20.
Benchmarking window order by, built-in functions: Collecting 100 samples in estimated 17.919 s (100 iterations window order by, built-in functions
time: [192.66 ms 199.27 ms 206.93 ms]
change: [-8.1837% -5.1872% -1.3218%] (p = 0.00 < 0.05)
Performance has improved.
Found 1 outliers among 100 measurements (1.00%)
1 (1.00%) high severe
Benchmarking window partition by, u64_wide, aggregate functions: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 27.8s, or reduce sample count to 10.
Benchmarking window partition by, u64_wide, aggregate functions: Collecting 100 samples in estimated 27.777 s window partition by, u64_wide, aggregate functions
time: [261.49 ms 265.26 ms 268.94 ms]
change: [-18.841% -17.347% -15.891%] (p = 0.00 < 0.05)
Performance has improved.
Benchmarking window partition by, u64_narrow, aggregate functions: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 14.1s, or reduce sample count to 30.
Benchmarking window partition by, u64_narrow, aggregate functions: Collecting 100 samples in estimated 14.061 window partition by, u64_narrow, aggregate functions
time: [133.94 ms 135.53 ms 137.16 ms]
change: [-19.379% -17.908% -16.493%] (p = 0.00 < 0.05)
Performance has improved.
Found 2 outliers among 100 measurements (2.00%)
2 (2.00%) high mild
Benchmarking window partition by, u64_wide, built-in functions: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 22.2s, or reduce sample count to 20.
Benchmarking window partition by, u64_wide, built-in functions: Collecting 100 samples in estimated 22.179 s ( window partition by, u64_wide, built-in functions
time: [213.65 ms 215.64 ms 217.65 ms]
change: [-20.451% -19.390% -18.336%] (p = 0.00 < 0.05)
Performance has improved.
Found 2 outliers among 100 measurements (2.00%)
1 (1.00%) low mild
1 (1.00%) high mild
Benchmarking window partition by, u64_narrow, built-in functions: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 11.8s, or reduce sample count to 40.
Benchmarking window partition by, u64_narrow, built-in functions: Collecting 100 samples in estimated 11.849 s window partition by, u64_narrow, built-in functions
time: [120.97 ms 122.49 ms 124.03 ms]
change: [-22.048% -20.534% -19.035%] (p = 0.00 < 0.05)
Performance has improved.
Benchmarking window partition and order by, u64_wide, aggregate functions: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 45.8s, or reduce sample count to 10.
Benchmarking window partition and order by, u64_wide, aggregate functions: Collecting 100 samples in estimated window partition and order by, u64_wide, aggregate functions
time: [434.30 ms 437.73 ms 441.20 ms]
change: [-6.4053% -5.1483% -3.8173%] (p = 0.00 < 0.05)
Performance has improved.
Benchmarking window partition and order by, u64_narrow, aggregate functions: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 64.5s, or reduce sample count to 10.
Benchmarking window partition and order by, u64_narrow, aggregate functions: Collecting 100 samples in estimat window partition and order by, u64_narrow, aggregate functions
time: [607.27 ms 610.83 ms 614.46 ms]
change: [-5.2088% -4.0570% -2.8806%] (p = 0.00 < 0.05)
Performance has improved.
Benchmarking window partition and order by, u64_wide, built-in functions: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 39.4s, or reduce sample count to 10.
Benchmarking window partition and order by, u64_wide, built-in functions: Collecting 100 samples in estimated window partition and order by, u64_wide, built-in functions
time: [387.44 ms 390.74 ms 394.56 ms]
change: [-0.4466% +0.8918% +2.3908%] (p = 0.21 > 0.05)
No change in performance detected.
Found 3 outliers among 100 measurements (3.00%)
2 (2.00%) high mild
1 (1.00%) high severe
Benchmarking window partition and order by, u64_narrow, built-in functions: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 38.4s, or reduce sample count to 10.
Benchmarking window partition and order by, u64_narrow, built-in functions: Collecting 100 samples in estimate window partition and order by, u64_narrow, built-in functions
time: [384.46 ms 389.51 ms 395.09 ms]
change: [+2.4390% +4.1822% +5.9735%] (p = 0.00 < 0.05)
Performance has regressed.
Found 6 outliers among 100 measurements (6.00%)
2 (2.00%) high mild
4 (4.00%) high severe

What changes are included in this PR?

Are there any user-facing changes?

@jimexistjimexist changed the title Window partition optimize rangesoptimize window function's find_ranges_in_rangeJun 22, 2021
@jimexist
jimexistforce-pushed the window-partition-optimize-ranges branch from 827a331 to 4711836CompareJune 22, 2021 10:11
@jimexist
jimexistforce-pushed the window-partition-optimize-ranges branch from 4711836 to b2ed86fCompareJune 23, 2021 00:30
@alamb

Copy link
Copy Markdown
Contributor

Makes sense to me

@alamb
alamb merged commit 35a0f30 into apache:masterJun 23, 2021
@jimexist
jimexist deleted the window-partition-optimize-ranges branch June 23, 2021 13:29
@houqphouqp added the performance Make DataFusion faster label Jul 31, 2021
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

performanceMake DataFusion faster

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@jimexist@alamb@houqp