Skip to content

[improvement](partition rebalance) improve partition rebalance choose candidate speed #36509 - #36978

Merged
dataroaring merged 2 commits into
apache:branch-2.0from
yujun777:pick_36509_to_upstream_branch-2.0
Jun 28, 2024
Merged

[improvement](partition rebalance) improve partition rebalance choose candidate speed #36509#36978
dataroaring merged 2 commits into
apache:branch-2.0from
yujun777:pick_36509_to_upstream_branch-2.0

Conversation

@yujun777

@yujun777yujun777 commented Jun 28, 2024

Copy link
Copy Markdown
Contributor

cherry pick from #36509

@yujun777

yujun777 commented Jun 28, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

@doris-robot

doris-robot commented Jun 28, 2024

Copy link
Copy Markdown

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@yujun777

yujun777 commented Jun 28, 2024

Copy link
Copy Markdown
ContributorAuthor

run feut

@yujun777

yujun777 commented Jun 28, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

@doris-robot

doris-robot commented Jun 28, 2024

Copy link
Copy Markdown
TPC-H: Total hot run time: 49620 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 395d4fa4be3d4e09132c768c01416c67982c16ce, data reload: false
------ Round 1 ----------------------------------
q1	17697	4397	4359	4359
q2	2065	157	149	149
q3	10451	1889	1929	1889
q4	10330	1254	1353	1254
q5	8412	3841	3851	3841
q6	237	149	126	126
q7	2038	1596	1582	1582
q8	9321	2722	2714	2714
q9	10570	10282	10302	10282
q10	8610	3491	3466	3466
q11	420	252	262	252
q12	472	302	306	302
q13	18359	3960	4033	3960
q14	357	317	329	317
q15	491	457	469	457
q16	676	574	577	574
q17	1120	936	986	936
q18	7354	6757	6979	6757
q19	1771	1673	1592	1592
q20	543	314	297	297
q21	4455	4073	4093	4073
q22	522	441	447	441
Total cold run time: 116271 ms
Total hot run time: 49620 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4277	4301	4260	4260
q2	324	230	229	229
q3	4139	4133	4143	4133
q4	2748	2741	2739	2739
q5	7154	7095	7015	7015
q6	236	123	119	119
q7	3216	2821	2827	2821
q8	4390	4470	4511	4470
q9	16905	16680	16763	16680
q10	4261	4271	4274	4271
q11	730	686	692	686
q12	1019	869	848	848
q13	6949	3740	3756	3740
q14	459	421	422	421
q15	507	470	461	461
q16	728	673	692	673
q17	3837	3811	3871	3811
q18	8828	8743	8783	8743
q19	1723	1685	1654	1654
q20	2383	2151	2074	2074
q21	8499	8509	8502	8502
q22	1054	957	958	957
Total cold run time: 84366 ms
Total hot run time: 79307 ms

@doris-robot

doris-robot commented Jun 28, 2024

Copy link
Copy Markdown
TPC-DS: Total hot run time: 205791 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 395d4fa4be3d4e09132c768c01416c67982c16ce, data reload: false
query1	924	421	382	382
query2	6550	2708	2578	2578
query3	6920	206	205	205
query4	19811	18078	17912	17912
query5	19729	6642	6682	6642
query6	307	225	227	225
query7	4161	316	339	316
query8	462	444	416	416
query9	3150	2704	2659	2659
query10	416	292	306	292
query11	11311	10699	10644	10644
query12	129	77	87	77
query13	5598	713	711	711
query14	18425	14020	13726	13726
query15	365	245	259	245
query16	6450	284	262	262
query17	1517	1492	901	901
query18	2235	412	416	412
query19	209	153	155	153
query20	76	80	82	80
query21	198	106	96	96
query22	5278	5015	4977	4977
query23	32788	32258	32181	32181
query24	7067	6643	6667	6643
query25	561	435	447	435
query26	540	166	166	166
query27	1901	309	301	301
query28	6128	2393	2349	2349
query29	2950	2682	2665	2665
query30	243	169	180	169
query31	925	752	766	752
query32	68	70	63	63
query33	405	278	257	257
query34	924	498	483	483
query35	1123	953	958	953
query36	1481	1120	1501	1120
query37	93	63	63	63
query38	3122	2902	3024	2902
query39	1382	1329	1334	1329
query40	206	97	108	97
query41	46	44	45	44
query42	88	86	87	86
query43	751	672	707	672
query44	1139	728	732	728
query45	254	240	238	238
query46	1337	1021	1067	1021
query47	1996	2032	1721	1721
query48	1074	730	720	720
query49	633	382	394	382
query50	969	712	638	638
query51	4816	4732	4669	4669
query52	104	86	88	86
query53	503	326	332	326
query54	2800	2523	2602	2523
query55	95	87	80	80
query56	240	220	221	220
query57	1210	1111	1086	1086
query58	233	202	224	202
query59	4343	4096	3859	3859
query60	216	201	229	201
query61	97	95	98	95
query62	818	492	473	473
query63	528	348	355	348
query64	2594	1583	1496	1496
query65	3758	3627	3626	3626
query66	779	382	380	380
query67	17775	16757	15465	15465
query68	8417	708	690	690
query69	566	343	372	343
query70	1816	1526	1419	1419
query71	399	328	319	319
query72	6569	3580	3573	3573
query73	777	334	317	317
query74	6180	5861	5853	5853
query75	4674	3783	3803	3783
query76	4662	1292	1370	1292
query77	629	258	257	257
query78	12594	11916	11720	11720
query79	10228	662	673	662
query80	1464	414	417	414
query81	501	242	234	234
query82	1557	101	97	97
query83	169	137	135	135
query84	259	71	72	71
query85	966	318	317	317
query86	343	299	329	299
query87	3226	2992	3058	2992
query88	5171	2332	2344	2332
query89	475	289	309	289
query90	1894	218	200	200
query91	177	143	140	140
query92	64	55	56	55
query93	6168	614	612	612
query94	780	217	218	217
query95	1116	1092	1082	1082
query96	701	338	336	336
query97	6503	6405	6501	6405
query98	202	183	172	172
query99	2912	940	949	940
Total cold run time: 317483 ms
Total hot run time: 205791 ms

@doris-robot

doris-robot commented Jun 28, 2024

Copy link
Copy Markdown
ClickBench: Total hot run time: 30.83 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 395d4fa4be3d4e09132c768c01416c67982c16ce, data reload: false
query1	0.03	0.02	0.02
query2	0.07	0.03	0.02
query3	0.25	0.06	0.05
query4	1.78	0.08	0.07
query5	0.53	0.52	0.51
query6	1.22	0.62	0.62
query7	0.01	0.01	0.01
query8	0.03	0.02	0.02
query9	0.52	0.50	0.49
query10	0.54	0.53	0.54
query11	0.12	0.08	0.09
query12	0.11	0.09	0.08
query13	0.62	0.61	0.62
query14	0.79	0.76	0.78
query15	0.79	0.76	0.76
query16	0.36	0.37	0.38
query17	0.96	1.00	1.01
query18	0.22	0.26	0.25
query19	1.92	1.85	1.85
query20	0.02	0.01	0.01
query21	15.48	0.56	0.56
query22	1.90	1.88	1.85
query23	17.41	1.12	0.80
query24	7.50	0.98	1.31
query25	0.38	0.10	0.06
query26	0.77	0.16	0.16
query27	0.04	0.04	0.04
query28	5.34	0.75	0.76
query29	12.74	2.34	2.22
query30	0.55	0.51	0.49
query31	2.81	0.38	0.38
query32	3.39	0.51	0.50
query33	3.06	3.10	3.05
query34	15.24	4.80	4.80
query35	4.84	4.83	4.82
query36	1.04	1.02	1.01
query37	0.06	0.05	0.04
query38	0.04	0.02	0.02
query39	0.02	0.01	0.02
query40	0.16	0.14	0.14
query41	0.06	0.02	0.01
query42	0.02	0.01	0.02
query43	0.02	0.02	0.02
Total cold run time: 103.76 s
Total hot run time: 30.83 s

@doris-robot

doris-robot commented Jun 28, 2024

Copy link
Copy Markdown

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 395d4fa4be3d4e09132c768c01416c67982c16ce with default session variables
Stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select: 21.5 seconds inserted 10000000 Rows, about 465K ops/s

@dataroaring
dataroaring merged commit 77765b2 into apache:branch-2.0Jun 28, 2024
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@yujun777weixingyu12
[improvement](partition rebalance) improve partition rebalance choose…
1d7df5f
… candidate speed apache#36509 (apache#36978)
cherry pick from apache#36509
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@yujun777@doris-robot@dataroaring