Skip to content

[improvement](query) prefer to chose tablet on alive disk #39467 - #39657

Merged
dataroaring merged 4 commits into
apache:branch-2.0from
yujun777:pick_39467_to_upstream_branch-2.0
Aug 23, 2024
Merged

[improvement](query) prefer to chose tablet on alive disk #39467#39657
dataroaring merged 4 commits into
apache:branch-2.0from
yujun777:pick_39467_to_upstream_branch-2.0

Conversation

@yujun777

@yujun777yujun777 commented Aug 20, 2024

Copy link
Copy Markdown
Contributor

cherry pick from #39467

@yujun777

yujun777 commented Aug 20, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

@doris-robot

doris-robot commented Aug 20, 2024

Copy link
Copy Markdown

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@yujun777

yujun777 commented Aug 20, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

@github-actions

github-actionsBot commented Aug 20, 2024

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions

github-actionsBot commented Aug 20, 2024

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@github-actionsgithub-actionsBot added the area/planner Issues or PRs related to the query planner label Aug 20, 2024
@github-actions

github-actionsBot commented Aug 22, 2024

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@yujun777

yujun777 commented Aug 22, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

@github-actions

github-actionsBot commented Aug 22, 2024

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot

doris-robot commented Aug 22, 2024

Copy link
Copy Markdown
TPC-H: Total hot run time: 50429 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4e23b79650466165f0e920ea5c519baae8d7e742, data reload: false
------ Round 1 ----------------------------------
q1	18327	4512	4372	4372
q2	2076	197	195	195
q3	10177	1987	1950	1950
q4	10129	1267	1342	1267
q5	8563	3970	3960	3960
q6	270	145	139	139
q7	2074	1648	1645	1645
q8	9558	2789	2761	2761
q9	13852	10310	10211	10211
q10	8705	3566	3579	3566
q11	447	279	277	277
q12	508	329	330	329
q13	18391	3983	4043	3983
q14	368	356	348	348
q15	537	498	502	498
q16	718	613	602	602
q17	1161	951	940	940
q18	7238	6850	7007	6850
q19	1737	1662	1612	1612
q20	551	352	353	352
q21	4452	4178	4127	4127
q22	540	457	445	445
Total cold run time: 120379 ms
Total hot run time: 50429 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4371	4357	4347	4347
q2	359	265	262	262
q3	4208	4135	4165	4135
q4	2755	2750	2751	2750
q5	7136	7153	7098	7098
q6	258	137	134	134
q7	3269	2893	2891	2891
q8	4375	4528	4524	4524
q9	16936	16863	16692	16692
q10	4254	4293	4252	4252
q11	799	737	691	691
q12	1089	894	903	894
q13	6466	3747	3766	3747
q14	479	451	442	442
q15	537	496	521	496
q16	760	710	712	710
q17	3868	3928	3790	3790
q18	8678	8708	8823	8708
q19	1784	1750	1709	1709
q20	2428	2131	2143	2131
q21	8501	8527	8486	8486
q22	1040	988	971	971
Total cold run time: 84350 ms
Total hot run time: 79860 ms

@doris-robot

doris-robot commented Aug 22, 2024

Copy link
Copy Markdown

TeamCity be ut coverage result:
Function Coverage: 37.83% (8130/21489)
Line Coverage: 29.55% (66840/226202)
Region Coverage: 29.02% (34458/118733)
Branch Coverage: 24.91% (17727/71178)
Coverage Report: http://coverage.selectdb-in.cc/coverage/4e23b79650466165f0e920ea5c519baae8d7e742_4e23b79650466165f0e920ea5c519baae8d7e742/report/index.html

@doris-robot

doris-robot commented Aug 22, 2024

Copy link
Copy Markdown
TPC-DS: Total hot run time: 209802 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4e23b79650466165f0e920ea5c519baae8d7e742, data reload: false
query1	980	419	412	412
query2	6784	2417	2382	2382
query3	6951	229	226	226
query4	20015	18018	18093	18018
query5	19939	6669	6712	6669
query6	320	253	263	253
query7	4196	330	344	330
query8	468	499	463	463
query9	3177	2731	2684	2684
query10	468	384	369	369
query11	11470	10914	10952	10914
query12	163	118	110	110
query13	5683	715	718	715
query14	17986	14022	13894	13894
query15	387	266	251	251
query16	6512	345	310	310
query17	1624	1471	934	934
query18	2347	482	479	479
query19	241	182	179	179
query20	114	110	118	110
query21	244	151	148	148
query22	5139	5176	5078	5078
query23	32529	32285	32062	32062
query24	7105	6563	6584	6563
query25	572	472	474	472
query26	559	189	190	189
query27	1814	334	330	330
query28	6024	2404	2348	2348
query29	3023	2897	2834	2834
query30	284	211	208	208
query31	993	844	846	844
query32	103	97	91	91
query33	472	344	333	333
query34	906	532	538	532
query35	1191	1001	988	988
query36	1395	1261	1303	1261
query37	119	89	90	89
query38	3162	2934	2998	2934
query39	1492	1438	1433	1433
query40	260	146	148	146
query41	170	158	154	154
query42	114	119	109	109
query43	716	715	622	622
query44	1178	773	777	773
query45	282	269	269	269
query46	1284	1038	1020	1020
query47	1864	2004	1918	1918
query48	1032	737	737	737
query49	842	593	582	582
query50	943	617	725	617
query51	4842	4746	4755	4746
query52	115	104	106	104
query53	505	378	378	378
query54	2689	2556	2542	2542
query55	104	102	105	102
query56	317	317	290	290
query57	1345	1216	1261	1216
query58	329	310	293	293
query59	3717	3389	3725	3389
query60	297	269	280	269
query61	157	159	182	159
query62	903	508	557	508
query63	545	391	403	391
query64	2747	1692	1663	1663
query65	3640	3622	3629	3622
query66	1249	818	830	818
query67	17118	15407	15016	15016
query68	8893	725	699	699
query69	651	411	402	402
query70	1803	1562	1645	1562
query71	472	360	377	360
query72	6530	3570	3542	3542
query73	778	374	376	374
query74	6396	5995	5949	5949
query75	5378	3880	3729	3729
query76	5488	1221	1150	1150
query77	1096	424	422	422
query78	12626	11832	12183	11832
query79	7103	676	686	676
query80	1104	562	553	553
query81	538	277	275	275
query82	1299	130	129	129
query83	251	221	219	219
query84	276	98	102	98
query85	970	407	414	407
query86	369	321	338	321
query87	3259	3072	3120	3072
query88	4993	2559	2565	2559
query89	404	345	361	345
query90	1972	258	243	243
query91	197	170	163	163
query92	90	89	84	84
query93	3689	604	591	591
query94	731	250	253	250
query95	1173	1130	1123	1123
query96	668	350	350	350
query97	6509	6423	6365	6365
query98	230	218	212	212
query99	3002	947	854	854
Total cold run time: 315193 ms
Total hot run time: 209802 ms

@doris-robot

doris-robot commented Aug 22, 2024

Copy link
Copy Markdown
ClickBench: Total hot run time: 31.15 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4e23b79650466165f0e920ea5c519baae8d7e742, data reload: false
query1	0.02	0.02	0.02
query2	0.07	0.03	0.02
query3	0.25	0.05	0.05
query4	1.78	0.06	0.06
query5	0.54	0.52	0.52
query6	1.23	0.62	0.62
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.53	0.48	0.49
query10	0.56	0.54	0.54
query11	0.13	0.09	0.09
query12	0.13	0.09	0.10
query13	0.63	0.62	0.63
query14	0.80	0.82	0.78
query15	0.78	0.76	0.78
query16	0.38	0.38	0.38
query17	1.02	1.04	1.04
query18	0.22	0.27	0.24
query19	1.93	1.83	1.86
query20	0.01	0.01	0.02
query21	15.44	0.55	0.54
query22	2.05	2.42	1.81
query23	17.09	0.94	1.00
query24	6.30	2.25	0.77
query25	0.36	0.05	0.06
query26	0.86	0.17	0.15
query27	0.06	0.05	0.04
query28	6.13	0.77	0.72
query29	12.66	2.24	2.40
query30	0.77	0.70	0.70
query31	2.82	0.39	0.38
query32	3.37	0.51	0.50
query33	3.06	3.08	3.09
query34	15.27	4.77	4.81
query35	4.87	4.86	4.86
query36	1.07	1.03	1.03
query37	0.08	0.06	0.06
query38	0.05	0.03	0.04
query39	0.03	0.03	0.04
query40	0.19	0.16	0.17
query41	0.09	0.04	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.04
Total cold run time: 103.77 s
Total hot run time: 31.15 s

@doris-robot

doris-robot commented Aug 22, 2024

Copy link
Copy Markdown

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 4e23b79650466165f0e920ea5c519baae8d7e742 with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 21.3 seconds inserted 10000000 Rows, about 469K ops/s

@dataroaring
dataroaring merged commit 6f8b949 into apache:branch-2.0Aug 23, 2024
mongo360 pushed a commit to mongo360/doris that referenced this pull request Dec 11, 2024
@yujun777@Baymine
[improvement](query) prefer to chose tablet on alive diskapache#39467(
6c9fa50
apache#39657)
cherry pick from apache#39467
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

area/plannerIssues or PRs related to the query plannerdev/2.0.15-merged

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@yujun777@doris-robot@dataroaring