[Opt](ShortCircuit) add more stats info to trace by eldenmoon · Pull Request #42697 · apache/doris

eldenmoon · 2024-10-29T06:54:14Z

No description provided.

doris-robot · 2024-10-29T06:54:19Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

eldenmoon · 2024-10-29T06:54:42Z

run buildall

eldenmoon · 2024-10-29T06:55:05Z

run buildall

github-actions · 2024-10-29T07:00:10Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-10-29T07:50:49Z

TPC-H: Total hot run time: 41836 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4fdd150d971b76f8271743e2b60a5358bba46cc9, data reload: false
------ Round 1 ----------------------------------
q1	17591	7634	7482	7482
q2	2058	166	180	166
q3	10596	1087	1146	1087
q4	10571	913	903	903
q5	7746	3122	3079	3079
q6	235	152	143	143
q7	1040	617	588	588
q8	9357	1978	2006	1978
q9	6604	6450	6456	6450
q10	7100	2459	2482	2459
q11	457	251	245	245
q12	434	224	216	216
q13	17803	3087	3027	3027
q14	249	216	218	216
q15	578	510	508	508
q16	688	600	595	595
q17	981	505	605	505
q18	7209	6739	6843	6739
q19	1332	1037	1020	1020
q20	468	190	200	190
q21	4034	3302	3230	3230
q22	1125	1017	1010	1010
Total cold run time: 108256 ms
Total hot run time: 41836 ms
----- Round 2, with runtime_filter_mode=off -----
q1	7286	7305	7324	7305
q2	323	233	231	231
q3	3092	2984	2982	2982
q4	2120	1886	1779	1779
q5	5740	5779	6159	5779
q6	226	142	139	139
q7	2326	1816	1767	1767
q8	3376	3510	3484	3484
q9	8985	8925	8927	8925
q10	3596	3592	3589	3589
q11	599	479	505	479
q12	855	624	606	606
q13	8869	3204	3150	3150
q14	301	280	275	275
q15	565	540	515	515
q16	719	656	664	656
q17	1865	1666	1612	1612
q18	8453	7781	7786	7781
q19	1706	1609	1595	1595
q20	2136	1901	1882	1882
q21	5703	5457	5513	5457
q22	1164	1057	1038	1038
Total cold run time: 70005 ms
Total hot run time: 61026 ms

doris-robot · 2024-10-29T07:57:53Z

TeamCity be ut coverage result:
Function Coverage: 37.47% (9719/25939)
Line Coverage: 28.72% (80598/280651)
Region Coverage: 28.15% (41673/148056)
Branch Coverage: 24.71% (21168/85672)
Coverage Report: http://coverage.selectdb-in.cc/coverage/4fdd150d971b76f8271743e2b60a5358bba46cc9_4fdd150d971b76f8271743e2b60a5358bba46cc9/report/index.html

doris-robot · 2024-10-29T07:58:18Z

TeamCity be ut coverage result:
Function Coverage: 37.47% (9720/25939)
Line Coverage: 28.72% (80616/280651)
Region Coverage: 28.15% (41674/148056)
Branch Coverage: 24.71% (21171/85672)
Coverage Report: http://coverage.selectdb-in.cc/coverage/4fdd150d971b76f8271743e2b60a5358bba46cc9_4fdd150d971b76f8271743e2b60a5358bba46cc9/report/index.html

doris-robot · 2024-10-29T08:03:02Z

TPC-DS: Total hot run time: 197101 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4fdd150d971b76f8271743e2b60a5358bba46cc9, data reload: false
query1	1282	972	1003	972
query2	6238	1989	2023	1989
query3	11617	4714	4866	4714
query4	40009	23894	23519	23519
query5	4731	457	437	437
query6	227	187	170	170
query7	3687	298	291	291
query8	291	238	240	238
query9	6571	2684	2664	2664
query10	426	257	244	244
query11	15952	15133	15244	15133
query12	151	111	96	96
query13	891	427	414	414
query14	8714	7189	7564	7189
query15	248	177	187	177
query16	7671	490	498	490
query17	1432	618	584	584
query18	2008	312	306	306
query19	364	159	160	159
query20	121	117	112	112
query21	209	107	108	107
query22	4838	4641	4362	4362
query23	34645	34135	34023	34023
query24	11053	2877	2790	2790
query25	590	409	403	403
query26	1186	168	164	164
query27	2334	285	300	285
query28	7458	2441	2424	2424
query29	700	429	422	422
query30	255	163	164	163
query31	996	804	806	804
query32	65	56	54	54
query33	627	280	269	269
query34	905	522	533	522
query35	1064	929	886	886
query36	1055	965	949	949
query37	142	75	73	73
query38	4412	4332	4205	4205
query39	1461	1432	1431	1431
query40	201	99	99	99
query41	48	45	46	45
query42	109	98	101	98
query43	544	503	484	484
query44	1234	820	814	814
query45	181	163	166	163
query46	1131	716	722	716
query47	1942	1850	1846	1846
query48	442	317	328	317
query49	949	405	406	405
query50	828	408	403	403
query51	7098	6878	7034	6878
query52	100	89	89	89
query53	264	184	183	183
query54	553	402	403	402
query55	78	77	78	77
query56	261	264	233	233
query57	1299	1143	1180	1143
query58	220	204	203	203
query59	3297	2958	3067	2958
query60	264	250	247	247
query61	105	97	104	97
query62	840	675	656	656
query63	214	188	185	185
query64	3448	623	618	618
query65	3239	3611	3246	3246
query66	827	301	335	301
query67	16119	15792	15691	15691
query68	4065	584	586	584
query69	424	259	252	252
query70	1215	1146	1179	1146
query71	338	252	252	252
query72	6262	4054	3996	3996
query73	782	363	363	363
query74	10102	9056	8946	8946
query75	3416	2686	2656	2656
query76	1997	1020	963	963
query77	438	274	278	274
query78	10650	9637	9569	9569
query79	1687	603	626	603
query80	965	429	442	429
query81	563	241	243	241
query82	362	113	111	111
query83	226	137	146	137
query84	225	72	71	71
query85	1393	285	288	285
query86	384	303	301	301
query87	4874	4715	4749	4715
query88	3444	2230	2184	2184
query89	409	298	292	292
query90	1941	188	187	187
query91	137	111	99	99
query92	56	49	48	48
query93	2730	561	560	560
query94	914	301	305	301
query95	359	250	253	250
query96	629	281	296	281
query97	2920	2676	2694	2676
query98	227	206	202	202
query99	1543	1296	1305	1296
Total cold run time: 298138 ms
Total hot run time: 197101 ms

doris-robot · 2024-10-29T08:08:31Z

ClickBench: Total hot run time: 32.8 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4fdd150d971b76f8271743e2b60a5358bba46cc9, data reload: false
query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.23	0.06	0.06
query4	1.65	0.10	0.10
query5	0.41	0.41	0.41
query6	1.16	0.66	0.65
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.56	0.51	0.50
query10	0.57	0.55	0.55
query11	0.14	0.12	0.11
query12	0.13	0.11	0.11
query13	0.62	0.60	0.59
query14	2.79	2.77	2.71
query15	0.89	0.83	0.82
query16	0.39	0.38	0.38
query17	1.10	1.00	1.10
query18	0.20	0.19	0.19
query19	1.86	1.81	1.89
query20	0.02	0.01	0.02
query21	15.36	0.60	0.59
query22	2.59	2.40	2.00
query23	17.05	0.89	0.89
query24	3.22	1.08	0.95
query25	0.33	0.10	0.04
query26	0.43	0.15	0.14
query27	0.04	0.04	0.05
query28	10.51	1.10	1.08
query29	12.58	3.36	3.30
query30	0.25	0.06	0.05
query31	2.88	0.39	0.38
query32	3.28	0.46	0.45
query33	2.95	3.08	3.03
query34	17.11	4.52	4.47
query35	4.57	4.54	4.50
query36	0.66	0.47	0.48
query37	0.09	0.06	0.06
query38	0.04	0.04	0.04
query39	0.03	0.02	0.02
query40	0.16	0.12	0.12
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 107.17 s
Total hot run time: 32.8 s

qidaye

LGTM

github-actions · 2024-11-01T07:51:41Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-11-01T07:51:43Z

PR approved by anyone and no changes requested.

xiaokang · 2024-11-01T11:41:22Z

 specified_rowsets = _tablet->get_rowset_by_ids(nullptr);
 }
 std::vector<std::unique_ptr<SegmentCacheHandle>> segment_caches(specified_rowsets.size());
+ // init segment_cache


Why add this logic?

in order to trace the time of loading segments

xiaokang

LGTM

(#42697)

…he#43327) (apache#42697)

Need need to open segments for all rowsets, only open in `lookup_row_key` stage. Introduced by apache#42697

No need need to open segments for all rowsets, only open in `lookup_row_key` stage. Introduced by #42697

…he#44073) No need need to open segments for all rowsets, only open in `lookup_row_key` stage. Introduced by apache#42697

[Opt](ShortCircuit) add more stats info to trace
4fdd150

eldenmoon added dev/3.0.3-merged dev/2.1.x dev/3.0.x and removed dev/3.0.3-merged labels Oct 29, 2024

qidaye approved these changes Nov 1, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Nov 1, 2024

github-actionsBot added the reviewed label Nov 1, 2024

xiaokang reviewed Nov 1, 2024
View reviewed changes

xiaokang approved these changes Nov 4, 2024
View reviewed changes

xiaokang merged commit 90ae514 into apache:masterNov 4, 2024

eldenmoon deleted the add_pq-stat branch November 4, 2024 06:52

eldenmoon added dev/3.0.x and removed dev/3.0.x labels Nov 4, 2024

github-actionsBot added cherry-pick-conflict-in-3.0 cherry-pick-conflict-in-2.1.x labels Nov 4, 2024

eldenmoon added a commit to eldenmoon/incubator-doris that referenced this pull request Nov 6, 2024
[opt](ShortCircuit) add more stats info to trace (apache#42697)
f5dd935

eldenmoon mentioned this pull request Nov 6, 2024
[opt](ShortCircuit) add more stats info to trace (#42697) #43327
Merged

eldenmoon added a commit to eldenmoon/incubator-doris that referenced this pull request Nov 6, 2024
[opt](ShortCircuit) add more stats info to trace (apache#42697)
014dd39

eldenmoon added a commit to eldenmoon/incubator-doris that referenced this pull request Nov 6, 2024
[opt](ShortCircuit) add more stats info to trace (apache#42697)
6514518

eldenmoon removed dev/2.1.x cherry-pick-conflict-in-2.1.x labels Nov 6, 2024

eldenmoon added a commit that referenced this pull request Nov 6, 2024
[opt](ShortCircuit) add more stats info to trace (#42697) (#43327)
e0dfae1
(#42697)

eldenmoon added dev/3.0.3-merged and removed dev/3.0.x cherry-pick-conflict-in-3.0 labels Nov 6, 2024

eldenmoon added a commit to eldenmoon/incubator-doris that referenced this pull request Nov 12, 2024
[opt](ShortCircuit) add more stats info to trace (apache#42697) (apac…
28729dc
…he#43327) (apache#42697)

eldenmoon added a commit to eldenmoon/incubator-doris that referenced this pull request Nov 16, 2024
[Fix](ShortCircuit) fix open segments leading to bad performace
08db852
Need need to open segments for all rowsets, only open in `lookup_row_key` stage. Introduced by apache#42697

eldenmoon added a commit to eldenmoon/incubator-doris that referenced this pull request Nov 16, 2024
[Fix](ShortCircuit) fix open segments leading to bad performace
d11fe39
Need need to open segments for all rowsets, only open in `lookup_row_key` stage. Introduced by apache#42697

eldenmoon mentioned this pull request Nov 16, 2024
[Fix](ShortCircuit) fix open segments leading to bad performace #44073
Merged
16 tasks

eldenmoon added a commit that referenced this pull request Nov 16, 2024
[Fix](ShortCircuit) fix open segments leading to bad performace (#44073)
893e05a
No need need to open segments for all rowsets, only open in `lookup_row_key` stage. Introduced by #42697

github-actionsBot pushed a commit that referenced this pull request Nov 16, 2024
[Fix](ShortCircuit) fix open segments leading to bad performace (#44073)
e6306c7
No need need to open segments for all rowsets, only open in `lookup_row_key` stage. Introduced by #42697

airborne12 mentioned this pull request Dec 20, 2024
[opt](profile) add index page profile for io #45675
Merged
16 tasks

HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
[opt](ShortCircuit) add more stats info to trace (apache#42697)
1c86a76

Uh oh!

Conversation

eldenmoon commented Oct 29, 2024

Uh oh!

doris-robot commented Oct 29, 2024

Uh oh!

eldenmoon commented Oct 29, 2024

Uh oh!

eldenmoon commented Oct 29, 2024

Uh oh!

github-actionsBot commented Oct 29, 2024

Uh oh!

doris-robot commented Oct 29, 2024

Uh oh!

doris-robot commented Oct 29, 2024

Uh oh!

doris-robot commented Oct 29, 2024

Uh oh!

doris-robot commented Oct 29, 2024

Uh oh!

doris-robot commented Oct 29, 2024

Uh oh!

qidaye left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Nov 1, 2024

Uh oh!

github-actionsBot commented Nov 1, 2024

Uh oh!

Uh oh!

xiaokangNov 1, 2024

Choose a reason for hiding this comment

Uh oh!

eldenmoonNov 4, 2024

Choose a reason for hiding this comment

Uh oh!

xiaokang left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants