Skip to content

[improvement](balance) partition rebalance chose disk by rr - #36826

Merged
dataroaring merged 3 commits into
apache:masterfrom
yujun777:fix-partition-rebalance-prefer-chose-first-disk
Jun 26, 2024
Merged

[improvement](balance) partition rebalance chose disk by rr#36826
dataroaring merged 3 commits into
apache:masterfrom
yujun777:fix-partition-rebalance-prefer-chose-first-disk

Conversation

@yujun777

@yujun777yujun777 commented Jun 25, 2024•
edited
Loading

Copy link
Copy Markdown
Contributor

Partition rebalance will call function takeAnAvailBalanceSlotFrom to chose disk. But this function prefer to choose the first disk. This PR make this function chose disk by RR.

@doris-robot

doris-robot commented Jun 25, 2024

Copy link
Copy Markdown

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@yujun777

yujun777 commented Jun 25, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

@doris-robot

doris-robot commented Jun 25, 2024

Copy link
Copy Markdown
TPC-H: Total hot run time: 39804 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 321a00f452d4c4494f5fe27f0b98cd7d668e9047, data reload: false
------ Round 1 ----------------------------------
q1	17694	4308	4264	4264
q2	2020	188	184	184
q3	10549	1069	1099	1069
q4	10194	853	794	794
q5	7474	2669	2616	2616
q6	220	134	134	134
q7	953	599	618	599
q8	9229	2074	2089	2074
q9	8767	6461	6443	6443
q10	8960	3735	3739	3735
q11	459	249	265	249
q12	461	229	224	224
q13	17767	2976	2980	2976
q14	261	223	226	223
q15	499	478	483	478
q16	518	368	377	368
q17	961	676	681	676
q18	7907	7572	7374	7374
q19	7135	1479	1468	1468
q20	687	323	319	319
q21	4803	3198	3881	3198
q22	392	346	339	339
Total cold run time: 117910 ms
Total hot run time: 39804 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4371	4211	4231	4211
q2	365	267	268	267
q3	2969	2873	2874	2873
q4	1996	1653	1775	1653
q5	5624	5496	5555	5496
q6	228	129	127	127
q7	2205	1878	1922	1878
q8	3280	3408	3390	3390
q9	8727	8788	8747	8747
q10	4196	3702	3849	3702
q11	600	506	494	494
q12	823	632	640	632
q13	16189	3151	3185	3151
q14	312	280	262	262
q15	533	494	478	478
q16	512	430	419	419
q17	1805	1519	1512	1512
q18	8169	7995	7841	7841
q19	1851	1646	1762	1646
q20	2114	1906	1865	1865
q21	6955	4828	4936	4828
q22	664	582	566	566
Total cold run time: 74488 ms
Total hot run time: 56038 ms

@yujun777yujun777 changed the title [improvement](balance) fix partition rebalance prefer chose first disk[improvement](balance) partition rebalance chose disk by rrJun 25, 2024
@doris-robot

doris-robot commented Jun 25, 2024

Copy link
Copy Markdown
TPC-DS: Total hot run time: 173875 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 321a00f452d4c4494f5fe27f0b98cd7d668e9047, data reload: false
query1	896	383	374	374
query2	6405	2387	2436	2387
query3	6628	207	211	207
query4	18931	17393	17272	17272
query5	3644	463	481	463
query6	236	155	175	155
query7	4576	296	282	282
query8	321	316	298	298
query9	8694	2382	2375	2375
query10	573	300	276	276
query11	10420	9974	10001	9974
query12	111	85	78	78
query13	1638	361	362	361
query14	9355	7093	6963	6963
query15	229	182	185	182
query16	7247	276	269	269
query17	1398	560	529	529
query18	1716	292	280	280
query19	199	148	157	148
query20	88	92	90	90
query21	210	121	128	121
query22	4255	4110	3994	3994
query23	33760	33711	33489	33489
query24	11015	2873	2918	2873
query25	636	405	428	405
query26	733	161	170	161
query27	2349	328	338	328
query28	5780	2178	2152	2152
query29	891	663	642	642
query30	272	164	149	149
query31	959	770	757	757
query32	100	60	59	59
query33	677	320	313	313
query34	871	487	484	484
query35	765	668	642	642
query36	1124	992	988	988
query37	144	73	74	73
query38	2919	2864	2841	2841
query39	888	852	829	829
query40	224	135	133	133
query41	79	59	53	53
query42	108	98	103	98
query43	584	554	540	540
query44	1091	734	727	727
query45	193	173	165	165
query46	1068	752	727	727
query47	1872	1750	1800	1750
query48	389	318	306	306
query49	859	417	418	417
query50	766	404	394	394
query51	6830	6763	6773	6763
query52	102	98	91	91
query53	368	285	287	285
query54	890	446	455	446
query55	74	76	74	74
query56	299	285	292	285
query57	1121	1049	1066	1049
query58	263	247	259	247
query59	3495	3215	3092	3092
query60	303	280	285	280
query61	109	108	109	108
query62	724	440	437	437
query63	318	289	286	286
query64	8440	2255	1776	1776
query65	3228	3105	3104	3104
query66	829	322	327	322
query67	15653	14934	14846	14846
query68	8551	555	558	555
query69	719	502	428	428
query70	1419	1127	1144	1127
query71	479	268	272	268
query72	8645	5644	5374	5374
query73	2188	329	322	322
query74	5802	5544	5518	5518
query75	4882	2664	2662	2662
query76	4562	987	936	936
query77	791	298	312	298
query78	10516	9794	9708	9708
query79	8810	532	523	523
query80	1029	462	463	462
query81	558	213	216	213
query82	747	106	107	106
query83	349	167	169	167
query84	276	85	82	82
query85	1265	283	278	278
query86	404	321	293	293
query87	3317	3059	3081	3059
query88	4434	2362	2365	2362
query89	530	398	388	388
query90	1931	192	190	190
query91	171	105	100	100
query92	64	48	47	47
query93	6483	514	509	509
query94	1213	191	188	188
query95	405	317	316	316
query96	612	268	263	263
query97	3195	3073	3028	3028
query98	218	199	191	191
query99	1162	856	850	850
Total cold run time: 287659 ms
Total hot run time: 173875 ms

@doris-robot

doris-robot commented Jun 25, 2024

Copy link
Copy Markdown
ClickBench: Total hot run time: 31.05 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 321a00f452d4c4494f5fe27f0b98cd7d668e9047, data reload: false
query1	0.03	0.04	0.04
query2	0.07	0.04	0.04
query3	0.22	0.05	0.04
query4	1.70	0.07	0.07
query5	0.49	0.49	0.48
query6	1.16	0.73	0.73
query7	0.01	0.01	0.02
query8	0.04	0.04	0.04
query9	0.54	0.48	0.50
query10	0.54	0.54	0.54
query11	0.15	0.12	0.11
query12	0.14	0.12	0.13
query13	0.61	0.60	0.58
query14	0.75	0.78	0.81
query15	0.84	0.82	0.81
query16	0.37	0.36	0.34
query17	1.03	1.02	0.94
query18	0.22	0.24	0.26
query19	1.94	1.75	1.79
query20	0.01	0.01	0.01
query21	15.63	0.66	0.65
query22	4.12	7.02	2.41
query23	18.24	1.38	1.29
query24	2.05	0.22	0.23
query25	0.16	0.08	0.09
query26	0.26	0.18	0.17
query27	0.08	0.08	0.08
query28	13.26	1.02	1.00
query29	12.59	3.35	3.34
query30	0.27	0.06	0.05
query31	2.86	0.38	0.38
query32	3.29	0.47	0.49
query33	2.84	2.90	2.96
query34	17.24	4.39	4.35
query35	4.55	4.47	4.43
query36	0.66	0.48	0.48
query37	0.18	0.15	0.16
query38	0.17	0.14	0.14
query39	0.04	0.03	0.03
query40	0.17	0.14	0.14
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.06	0.04	0.04
Total cold run time: 109.73 s
Total hot run time: 31.05 s

deardeng
deardeng approved these changes Jun 25, 2024
View reviewed changes

@deardengdeardeng left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions

github-actionsBot commented Jun 25, 2024

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

dataroaring
dataroaring previously approved these changes Jun 26, 2024
View reviewed changes

@dataroaringdataroaring left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label Jun 26, 2024
@github-actions

github-actionsBot commented Jun 26, 2024

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@yujun777

yujun777 commented Jun 26, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

@github-actionsgithub-actionsBot removed the approved Indicates a PR has been approved by one committer. label Jun 26, 2024
@yujun777

yujun777 commented Jun 26, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

@doris-robot

doris-robot commented Jun 26, 2024

Copy link
Copy Markdown
TPC-H: Total hot run time: 39501 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 47ff38ee67e72f2c91ba5b47a4cc9cc2827155f7, data reload: false
------ Round 1 ----------------------------------
q1	17622	4357	4268	4268
q2	2016	206	178	178
q3	10454	1151	1084	1084
q4	10185	737	795	737
q5	7455	2661	2619	2619
q6	212	129	133	129
q7	949	597	597	597
q8	9248	2055	2045	2045
q9	8798	6507	6461	6461
q10	8981	3693	3748	3693
q11	461	242	228	228
q12	483	227	226	226
q13	18639	2951	2983	2951
q14	259	233	213	213
q15	519	489	472	472
q16	491	374	372	372
q17	955	654	719	654
q18	7959	7423	7369	7369
q19	5871	1390	1456	1390
q20	657	321	330	321
q21	4910	3166	3938	3166
q22	414	328	337	328
Total cold run time: 117538 ms
Total hot run time: 39501 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4371	4248	4288	4248
q2	375	265	272	265
q3	2946	2888	2907	2888
q4	1952	1663	1649	1649
q5	5613	5484	5484	5484
q6	216	127	131	127
q7	2168	1848	1857	1848
q8	3268	3407	3421	3407
q9	8676	8657	8853	8657
q10	4078	3874	3667	3667
q11	582	500	496	496
q12	830	667	645	645
q13	16161	3152	3162	3152
q14	321	265	284	265
q15	518	490	474	474
q16	484	444	424	424
q17	1786	1561	1502	1502
q18	8054	7931	8221	7931
q19	1793	1669	1655	1655
q20	2154	1929	1948	1929
q21	5252	5005	4857	4857
q22	648	560	556	556
Total cold run time: 72246 ms
Total hot run time: 56126 ms

@doris-robot

doris-robot commented Jun 26, 2024

Copy link
Copy Markdown
TPC-DS: Total hot run time: 174956 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 47ff38ee67e72f2c91ba5b47a4cc9cc2827155f7, data reload: false
query1	892	385	378	378
query2	6430	2489	2301	2301
query3	6629	195	204	195
query4	19120	17462	17382	17382
query5	3599	471	488	471
query6	243	171	147	147
query7	4571	294	289	289
query8	295	291	296	291
query9	8606	2393	2384	2384
query10	556	302	267	267
query11	10612	10167	10140	10140
query12	110	87	79	79
query13	1633	380	370	370
query14	10314	7750	7372	7372
query15	253	183	194	183
query16	8071	275	271	271
query17	1881	551	532	532
query18	2105	280	283	280
query19	200	163	153	153
query20	91	81	83	81
query21	214	131	130	130
query22	4307	4115	4105	4105
query23	33893	33968	33760	33760
query24	10995	2852	2883	2852
query25	595	408	420	408
query26	706	165	159	159
query27	2269	327	331	327
query28	6204	2179	2164	2164
query29	905	668	648	648
query30	264	160	156	156
query31	977	773	749	749
query32	100	55	58	55
query33	831	312	353	312
query34	963	477	480	477
query35	779	677	644	644
query36	1118	974	984	974
query37	148	74	72	72
query38	2988	2929	2860	2860
query39	923	849	877	849
query40	209	124	130	124
query41	52	51	50	50
query42	111	96	104	96
query43	616	555	539	539
query44	1207	730	726	726
query45	209	162	164	162
query46	1074	769	716	716
query47	1871	1795	1801	1795
query48	375	300	286	286
query49	846	398	409	398
query50	768	392	391	391
query51	6907	6780	6745	6745
query52	108	96	92	92
query53	359	289	285	285
query54	863	437	447	437
query55	77	73	74	73
query56	284	266	254	254
query57	1143	1080	1054	1054
query58	253	245	249	245
query59	3516	3176	3144	3144
query60	292	284	277	277
query61	92	90	89	89
query62	603	454	438	438
query63	318	288	292	288
query64	8490	2262	1735	1735
query65	3157	3109	3106	3106
query66	728	333	329	329
query67	15617	15143	15067	15067
query68	4629	545	535	535
query69	612	421	376	376
query70	1144	1126	1125	1125
query71	457	269	267	267
query72	8455	5523	5311	5311
query73	754	323	324	323
query74	5921	5598	5449	5449
query75	4548	2647	2654	2647
query76	3081	957	913	913
query77	710	300	299	299
query78	10410	9850	9836	9836
query79	2923	520	507	507
query80	1307	479	467	467
query81	548	231	221	221
query82	927	109	103	103
query83	360	182	167	167
query84	266	87	90	87
query85	2013	271	317	271
query86	473	319	302	302
query87	3386	3125	3152	3125
query88	4242	2360	2350	2350
query89	487	380	392	380
query90	1821	191	187	187
query91	127	102	98	98
query92	57	51	49	49
query93	5145	529	517	517
query94	1165	187	186	186
query95	402	315	307	307
query96	609	267	268	267
query97	3249	3027	3073	3027
query98	225	203	200	200
query99	1211	851	849	849
Total cold run time: 277199 ms
Total hot run time: 174956 ms

@doris-robot

doris-robot commented Jun 26, 2024

Copy link
Copy Markdown
ClickBench: Total hot run time: 30.73 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 47ff38ee67e72f2c91ba5b47a4cc9cc2827155f7, data reload: false
query1	0.04	0.03	0.02
query2	0.08	0.04	0.03
query3	0.22	0.04	0.05
query4	1.67	0.07	0.07
query5	0.50	0.47	0.47
query6	1.12	0.72	0.72
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.53	0.50	0.50
query10	0.55	0.54	0.54
query11	0.16	0.11	0.11
query12	0.15	0.12	0.12
query13	0.60	0.58	0.57
query14	0.75	0.77	0.80
query15	0.83	0.80	0.81
query16	0.36	0.36	0.37
query17	1.02	0.96	1.02
query18	0.19	0.28	0.23
query19	1.82	1.71	1.70
query20	0.01	0.01	0.01
query21	15.43	0.66	0.66
query22	4.27	7.27	2.15
query23	18.32	1.43	1.24
query24	2.07	0.22	0.23
query25	0.15	0.09	0.08
query26	0.27	0.18	0.18
query27	0.08	0.08	0.08
query28	13.24	1.03	1.00
query29	12.62	3.32	3.29
query30	0.26	0.05	0.06
query31	2.86	0.39	0.39
query32	3.29	0.47	0.47
query33	2.90	2.92	2.93
query34	17.08	4.46	4.41
query35	4.53	4.46	4.49
query36	0.66	0.46	0.46
query37	0.17	0.16	0.16
query38	0.14	0.15	0.14
query39	0.04	0.03	0.03
query40	0.17	0.14	0.13
query41	0.09	0.05	0.06
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.41 s
Total hot run time: 30.73 s

dataroaring
dataroaring approved these changes Jun 26, 2024
View reviewed changes

@dataroaringdataroaring left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions

github-actionsBot commented Jun 26, 2024

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label Jun 26, 2024
@dataroaring
dataroaring merged commit b2747c5 into apache:masterJun 26, 2024
yujun777 added a commit to yujun777/doris that referenced this pull request Jun 27, 2024
@yujun777
[improvement](balance) partition rebalance chose disk by rr (apache#3…
15f9c7f
…6826)
Partition rebalance will call function takeAnAvailBalanceSlotFrom to
chose disk. But this function prefer to choose the first disk. This PR
make this function chose disk by RR.
@yujun777yujun777 mentioned this pull request Jun 27, 2024
Merged
yujun777 added a commit to yujun777/doris that referenced this pull request Jun 27, 2024
@yujun777
[improvement](balance) partition rebalance chose disk by rr (apache#3…
930072d
…6826)
Partition rebalance will call function takeAnAvailBalanceSlotFrom to
chose disk. But this function prefer to choose the first disk. This PR
make this function chose disk by RR.
@yujun777yujun777 mentioned this pull request Jun 27, 2024
Merged
dataroaring pushed a commit that referenced this pull request Jun 27, 2024
@yujun777
[improvement](balance) partition rebalance chose disk by rr#36826(#…
f5c0585
…36901)
cherry pick from #36826
dataroaring pushed a commit that referenced this pull request Jun 27, 2024
@yujun777
[improvement](balance) partition rebalance chose disk by rr#36826(#…
89fc55d
…36900)
cherry pick from #36826
dataroaring pushed a commit that referenced this pull request Jun 28, 2024
@yujun777@dataroaring
[improvement](balance) partition rebalance chose disk by rr (#36826)
2589dd5
Partition rebalance will call function takeAnAvailBalanceSlotFrom to
chose disk. But this function prefer to choose the first disk. This PR
make this function chose disk by RR.
@xiaokangxiaokang mentioned this pull request Jul 14, 2024
Closed
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@yujun777weixingyu12
[improvement](balance) partition rebalance chose disk by rrapache#36826
a408597
 (apache#36901)
cherry pick from apache#36826
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
@yujun777@HappenLee
[improvement](balance) partition rebalance chose disk by rr (apache#3…
fa9974b
…6826)
Partition rebalance will call function takeAnAvailBalanceSlotFrom to
chose disk. But this function prefer to choose the first disk. This PR
make this function chose disk by RR.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@deardengdeardengdeardeng approved these changes
@dataroaringdataroaringdataroaring approved these changes

Assignees

No one assigned

Labels

approvedIndicates a PR has been approved by one committer.dev/2.0.13-mergeddev/2.1.5-mergeddev/3.0.0-mergedreviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@yujun777@doris-robot@deardeng@dataroaring