[opt](cloud) Optimize balance speed by reducing the complexity of the rebalance algorithm by deardeng · Pull Request #51733 · apache/doris

deardeng · 2025-06-15T14:05:37Z

…ctor the code

What problem does this PR solve?

Arrary List remove，time complexity is too high

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Thearas · 2025-06-15T14:05:42Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

deardeng · 2025-06-15T14:05:59Z

run buildall

doris-robot · 2025-06-15T14:45:19Z

TPC-H: Total hot run time: 34870 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 29ed9b003af281108a3a5640e71314fcba6643e0, data reload: false
------ Round 1 ----------------------------------
q1	17593	5124	4916	4916
q2	1948	284	199	199
q3	10306	1288	754	754
q4	10306	998	532	532
q5	8502	2411	2348	2348
q6	182	164	132	132
q7	927	739	615	615
q8	9321	1322	1164	1164
q9	6942	5133	5138	5133
q10	6851	2319	1918	1918
q11	509	311	288	288
q12	351	352	220	220
q13	17795	3707	3118	3118
q14	260	228	222	222
q15	569	497	480	480
q16	441	444	395	395
q17	655	853	388	388
q18	7584	7224	7160	7160
q19	1383	952	547	547
q20	363	362	234	234
q21	4140	3316	3145	3145
q22	1071	1007	962	962
Total cold run time: 107999 ms
Total hot run time: 34870 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5002	4953	4993	4953
q2	256	321	223	223
q3	2178	2647	2328	2328
q4	1372	1801	1372	1372
q5	4224	4227	4397	4227
q6	214	171	127	127
q7	2003	1930	1754	1754
q8	2615	2564	2517	2517
q9	7323	7297	7240	7240
q10	2971	3176	2855	2855
q11	582	508	505	505
q12	722	784	626	626
q13	3533	3920	3331	3331
q14	279	308	275	275
q15	530	496	507	496
q16	458	501	467	467
q17	1176	1502	1412	1412
q18	7737	7484	7632	7484
q19	835	836	983	836
q20	2003	2052	1954	1954
q21	5068	4466	4559	4466
q22	1132	1068	1017	1017
Total cold run time: 52213 ms
Total hot run time: 50465 ms

doris-robot · 2025-06-15T14:57:43Z

TPC-DS: Total hot run time: 196986 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 29ed9b003af281108a3a5640e71314fcba6643e0, data reload: false
query1	1401	1045	987	987
query2	6151	1953	1959	1953
query3	11072	4593	4545	4545
query4	53159	25598	23725	23725
query5	5025	599	489	489
query6	375	235	217	217
query7	4897	542	323	323
query8	309	251	222	222
query9	5637	2928	2930	2928
query10	470	350	290	290
query11	15109	15196	14827	14827
query12	164	116	115	115
query13	1049	569	448	448
query14	10255	6395	6452	6395
query15	216	202	181	181
query16	7131	679	486	486
query17	1106	760	635	635
query18	1619	428	354	354
query19	215	210	183	183
query20	142	135	126	126
query21	214	131	165	131
query22	4434	4588	4432	4432
query23	34698	33682	33626	33626
query24	6784	2464	2501	2464
query25	483	498	435	435
query26	712	290	168	168
query27	2310	542	381	381
query28	2951	2393	2408	2393
query29	603	588	471	471
query30	292	232	201	201
query31	872	874	792	792
query32	78	63	63	63
query33	470	378	331	331
query34	877	917	590	590
query35	814	872	763	763
query36	1025	1056	969	969
query37	128	113	87	87
query38	4306	4319	4342	4319
query39	1605	1551	1498	1498
query40	223	127	123	123
query41	66	64	59	59
query42	140	117	128	117
query43	578	564	519	519
query44	1472	948	949	948
query45	197	173	165	165
query46	901	1051	682	682
query47	1860	1867	1811	1811
query48	435	469	361	361
query49	684	511	405	405
query50	738	741	440	440
query51	4317	4267	4198	4198
query52	125	115	113	113
query53	262	277	214	214
query54	651	625	567	567
query55	101	102	96	96
query56	317	340	320	320
query57	1228	1226	1174	1174
query58	282	293	279	279
query59	2884	2946	2817	2817
query60	364	373	352	352
query61	129	139	133	133
query62	766	764	689	689
query63	250	204	208	204
query64	1875	1047	716	716
query65	4211	4223	4183	4183
query66	730	409	315	315
query67	16295	15841	15780	15780
query68	7159	962	588	588
query69	560	330	285	285
query70	1274	1194	1195	1194
query71	527	353	329	329
query72	6003	4768	4777	4768
query73	1457	676	398	398
query74	8958	9243	8969	8969
query75	4014	3237	2731	2731
query76	4234	1264	780	780
query77	671	379	325	325
query78	10308	10486	9447	9447
query79	2433	844	635	635
query80	663	556	501	501
query81	485	267	228	228
query82	448	143	98	98
query83	290	275	250	250
query84	293	109	94	94
query85	787	420	330	330
query86	369	333	314	314
query87	4445	4494	4389	4389
query88	3558	2481	2480	2480
query89	413	331	308	308
query90	1802	226	221	221
query91	168	156	125	125
query92	79	65	62	62
query93	1999	1012	645	645
query94	675	428	320	320
query95	395	316	311	311
query96	544	610	302	302
query97	2759	2765	2713	2713
query98	249	221	222	221
query99	1481	1460	1302	1302
Total cold run time: 300739 ms
Total hot run time: 196986 ms

doris-robot · 2025-06-15T15:03:02Z

ClickBench: Total hot run time: 29.14 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 29ed9b003af281108a3a5640e71314fcba6643e0, data reload: false
query1	0.03	0.03	0.03
query2	0.08	0.04	0.04
query3	0.24	0.07	0.07
query4	1.61	0.11	0.10
query5	0.43	0.41	0.42
query6	1.18	0.67	0.68
query7	0.02	0.02	0.02
query8	0.06	0.03	0.04
query9	0.60	0.54	0.53
query10	0.58	0.61	0.58
query11	0.17	0.12	0.11
query12	0.15	0.12	0.12
query13	0.62	0.60	0.60
query14	0.80	0.83	0.86
query15	0.89	0.85	0.89
query16	0.39	0.39	0.39
query17	1.07	1.04	1.07
query18	0.24	0.22	0.22
query19	1.94	1.90	1.88
query20	0.02	0.01	0.02
query21	15.39	0.96	0.57
query22	0.77	1.30	0.78
query23	15.12	1.46	0.69
query24	7.42	1.08	0.48
query25	0.47	0.15	0.09
query26	0.62	0.18	0.15
query27	0.06	0.06	0.05
query28	9.20	0.93	0.48
query29	12.56	4.20	3.46
query30	0.26	0.10	0.07
query31	2.82	0.62	0.41
query32	3.24	0.56	0.48
query33	3.08	3.07	3.19
query34	15.82	5.11	4.50
query35	4.53	4.56	4.51
query36	0.68	0.50	0.49
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.03
query40	0.18	0.14	0.13
query41	0.08	0.02	0.02
query42	0.04	0.03	0.03
query43	0.04	0.03	0.04
Total cold run time: 103.67 s
Total hot run time: 29.14 s

hello-stephen · 2025-06-15T15:50:37Z

FE UT Coverage Report

Increment line coverage 1.75% (3/171) 🎉
Increment coverage report
Complete coverage report

…ctor the code

deardeng · 2025-06-26T11:27:47Z

run buildall

doris-robot · 2025-06-26T11:47:38Z

TPC-H: Total hot run time: 34324 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b8829d12c385e8e4682592756ae08c08cf9a5b52, data reload: false
------ Round 1 ----------------------------------
q1	17563	5245	5131	5131
q2	1954	278	187	187
q3	10503	1339	751	751
q4	10309	1020	527	527
q5	8984	2392	2357	2357
q6	192	163	131	131
q7	902	761	603	603
q8	9327	1252	1167	1167
q9	6770	5125	5104	5104
q10	6897	2395	2001	2001
q11	474	288	273	273
q12	362	354	224	224
q13	17775	3708	3111	3111
q14	243	222	216	216
q15	548	469	483	469
q16	425	421	377	377
q17	586	878	361	361
q18	7519	7212	7097	7097
q19	1215	951	574	574
q20	324	349	217	217
q21	3955	3256	2470	2470
q22	1071	1014	976	976
Total cold run time: 107898 ms
Total hot run time: 34324 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5255	5190	5170	5170
q2	259	321	226	226
q3	2155	2714	2337	2337
q4	1393	1780	1348	1348
q5	4297	4484	4507	4484
q6	211	171	132	132
q7	1979	1876	1814	1814
q8	2581	2534	2474	2474
q9	7167	7166	7146	7146
q10	3131	3271	2822	2822
q11	564	505	480	480
q12	659	753	618	618
q13	3597	3875	3274	3274
q14	284	300	257	257
q15	506	462	466	462
q16	447	513	430	430
q17	1178	1509	1381	1381
q18	7615	7481	7545	7481
q19	818	808	925	808
q20	1967	2053	1897	1897
q21	4939	4496	4307	4307
q22	1052	1046	998	998
Total cold run time: 52054 ms
Total hot run time: 50346 ms

doris-robot · 2025-06-26T11:59:12Z

TPC-DS: Total hot run time: 185139 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b8829d12c385e8e4682592756ae08c08cf9a5b52, data reload: false
query1	1013	387	387	387
query2	6520	1780	1707	1707
query3	6751	213	210	210
query4	26536	23961	23099	23099
query5	4358	546	429	429
query6	297	195	199	195
query7	4630	486	280	280
query8	267	217	217	217
query9	8585	2592	2601	2592
query10	475	320	268	268
query11	15627	15140	14808	14808
query12	163	107	102	102
query13	1650	525	403	403
query14	9062	5798	5763	5763
query15	202	181	185	181
query16	7321	632	482	482
query17	1202	724	589	589
query18	1985	410	307	307
query19	198	192	159	159
query20	127	120	121	120
query21	212	122	108	108
query22	4144	4230	3985	3985
query23	33994	33178	33246	33178
query24	8439	2363	2453	2363
query25	534	455	394	394
query26	1234	263	144	144
query27	2781	508	351	351
query28	4339	2098	2092	2092
query29	755	559	435	435
query30	284	225	190	190
query31	948	846	829	829
query32	72	68	62	62
query33	571	363	315	315
query34	790	855	514	514
query35	789	796	757	757
query36	953	970	911	911
query37	109	106	75	75
query38	4048	4101	4068	4068
query39	1462	1429	1401	1401
query40	209	118	102	102
query41	66	61	52	52
query42	122	106	106	106
query43	500	521	476	476
query44	1349	830	818	818
query45	175	167	160	160
query46	830	1073	627	627
query47	1752	1814	1719	1719
query48	376	417	301	301
query49	731	466	378	378
query50	640	679	428	428
query51	4176	4227	4151	4151
query52	111	111	95	95
query53	220	258	188	188
query54	568	564	496	496
query55	85	79	81	79
query56	288	276	277	276
query57	1170	1189	1141	1141
query58	260	249	244	244
query59	2615	2693	2644	2644
query60	328	311	297	297
query61	124	118	122	118
query62	829	722	692	692
query63	221	187	183	183
query64	4348	1043	695	695
query65	4304	4182	4218	4182
query66	1109	405	319	319
query67	15831	15570	15769	15570
query68	8609	896	533	533
query69	460	319	277	277
query70	1189	1064	1124	1064
query71	448	323	293	293
query72	5477	4815	4560	4560
query73	714	568	353	353
query74	9280	9175	8744	8744
query75	4172	3189	2689	2689
query76	3736	1148	735	735
query77	794	380	282	282
query78	10077	10328	9465	9465
query79	2155	781	581	581
query80	612	516	425	425
query81	465	265	224	224
query82	417	126	96	96
query83	272	248	237	237
query84	293	104	90	90
query85	787	350	305	305
query86	329	274	295	274
query87	4430	4428	4402	4402
query88	2911	2299	2278	2278
query89	377	317	291	291
query90	1936	202	202	202
query91	136	144	113	113
query92	78	55	58	55
query93	1148	962	603	603
query94	662	397	298	298
query95	379	289	279	279
query96	479	570	275	275
query97	2701	2761	2639	2639
query98	217	209	214	209
query99	1481	1388	1261	1261
Total cold run time: 273985 ms
Total hot run time: 185139 ms

doris-robot · 2025-06-26T12:04:29Z

ClickBench: Total hot run time: 29.4 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b8829d12c385e8e4682592756ae08c08cf9a5b52, data reload: false
query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.24	0.08	0.07
query4	1.61	0.11	0.11
query5	0.44	0.44	0.40
query6	1.17	0.66	0.65
query7	0.02	0.02	0.01
query8	0.04	0.04	0.04
query9	0.61	0.53	0.52
query10	0.56	0.57	0.57
query11	0.16	0.12	0.11
query12	0.15	0.12	0.12
query13	0.63	0.61	0.62
query14	0.80	0.82	0.82
query15	0.91	0.85	0.90
query16	0.39	0.39	0.41
query17	1.04	1.07	1.07
query18	0.22	0.21	0.20
query19	1.90	1.85	1.82
query20	0.01	0.01	0.02
query21	15.41	0.88	0.56
query22	0.74	1.25	0.63
query23	14.93	1.40	0.62
query24	6.99	2.20	0.64
query25	0.51	0.25	0.12
query26	0.49	0.16	0.14
query27	0.06	0.05	0.04
query28	9.42	0.85	0.43
query29	12.56	3.94	3.28
query30	0.26	0.09	0.06
query31	2.82	0.58	0.41
query32	3.23	0.56	0.48
query33	3.13	3.10	3.13
query34	16.10	5.45	4.77
query35	4.86	4.84	4.84
query36	0.69	0.50	0.51
query37	0.09	0.06	0.07
query38	0.05	0.04	0.03
query39	0.03	0.03	0.02
query40	0.18	0.16	0.15
query41	0.08	0.03	0.02
query42	0.03	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.72 s
Total hot run time: 29.4 s

gavinchou

LGTM

deardeng · 2025-07-01T06:41:11Z

run buildall

hello-stephen · 2025-07-01T07:56:12Z

FE UT Coverage Report

Increment line coverage 1.75% (3/171) 🎉
Increment coverage report
Complete coverage report

github-actions · 2025-07-04T07:32:33Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-07-04T07:32:35Z

PR approved by anyone and no changes requested.

… rebalance algorithm (#51733) Arrary List remove，time complexity is too high <img width="489" alt="image" src="https://github.com/user-attachments/assets/ffa486f8-c9fb-4ee9-9f9c-408e189525d5" />

…exity of the rebalance algorithm #51733 (#52813) Cherry-picked from #51733 Co-authored-by: deardeng <dengxin@selectdb.com>

…exity of the rebalance algorithm #51733 (#52814) Cherry-picked from #51733 Co-authored-by: deardeng <dengxin@selectdb.com>

deardeng requested review from dataroaring, gavinchou and w41ter as code owners June 15, 2025 14:05

deardeng marked this pull request as draft June 15, 2025 14:06

[Fix](cloud)Reduce the complexity of the rebalance algorithm and refa…
b8829d1
…ctor the code

deardeng force-pushed the fix-cloud-rebalance branch from 29ed9b0 to b8829d1Compare June 26, 2025 11:26

deardeng marked this pull request as ready for review June 26, 2025 11:27

gavinchou added dev/3.0.x dev/3.1.x cloud labels Jun 26, 2025

gavinchou reviewed Jun 26, 2025
View reviewed changes

Comment threadfe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudTabletRebalancer.java

gavinchou reviewed Jun 26, 2025
View reviewed changes

Comment threadfe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudTabletRebalancer.java

gavinchou reviewed Jun 26, 2025
View reviewed changes

add comment
f22d0bc

gavinchou changed the title ~~[Fix](cloud)Reduce the complexity of the rebalance algorithm and refa…~~[opt](cloud) Optimize balance speed by reducing the complexity of the rebalance algorithmJul 2, 2025

gavinchou approved these changes Jul 4, 2025
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Jul 4, 2025

github-actionsBot added the reviewed label Jul 4, 2025

Yukang-Lian approved these changes Jul 4, 2025
View reviewed changes

Hastyshell approved these changes Jul 4, 2025
View reviewed changes

gavinchou self-requested a review July 4, 2025 11:19

gavinchou approved these changes Jul 4, 2025
View reviewed changes

gavinchou merged commit 641fe19 into apache:masterJul 4, 2025

github-actionsBot mentioned this pull request Jul 4, 2025
branch-3.0: [opt](cloud) Optimize balance speed by reducing the complexity of the rebalance algorithm #51733 #52813
Merged

github-actionsBot mentioned this pull request Jul 4, 2025
branch-3.1: [opt](cloud) Optimize balance speed by reducing the complexity of the rebalance algorithm #51733 #52814
Merged

dataroaring pushed a commit that referenced this pull request Jul 8, 2025
branch-3.0: [opt](cloud) Optimize balance speed by reducing the compl…
a265588
…exity of the rebalance algorithm #51733 (#52813) Cherry-picked from #51733 Co-authored-by: deardeng <dengxin@selectdb.com>

dataroaring added dev/3.0.7-merged and removed dev/3.0.x labels Jul 8, 2025

morrySnow pushed a commit that referenced this pull request Jul 8, 2025
branch-3.1: [opt](cloud) Optimize balance speed by reducing the compl…
4c19880
…exity of the rebalance algorithm #51733 (#52814) Cherry-picked from #51733 Co-authored-by: deardeng <dengxin@selectdb.com>

morrySnow added dev/3.1.0-merged and removed dev/3.1.x labels Jul 8, 2025

gavinchou mentioned this pull request Aug 15, 2025
Release Note 3.0.7 #54868
Closed

Uh oh!

Conversation

deardeng commented Jun 15, 2025

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented Jun 15, 2025

Uh oh!

deardeng commented Jun 15, 2025

Uh oh!

doris-robot commented Jun 15, 2025

Uh oh!

doris-robot commented Jun 15, 2025

Uh oh!

doris-robot commented Jun 15, 2025

Uh oh!

hello-stephen commented Jun 15, 2025

FE UT Coverage Report

Uh oh!

deardeng commented Jun 26, 2025

Uh oh!

doris-robot commented Jun 26, 2025

Uh oh!

doris-robot commented Jun 26, 2025

Uh oh!

doris-robot commented Jun 26, 2025

Uh oh!

Uh oh!

Uh oh!

gavinchou left a comment

Choose a reason for hiding this comment

Uh oh!

deardeng commented Jul 1, 2025

Uh oh!

hello-stephen commented Jul 1, 2025

FE UT Coverage Report

Uh oh!

github-actionsBot commented Jul 4, 2025

Uh oh!

github-actionsBot commented Jul 4, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants