[opt](inverted index) Add NumInvertedIndexRemoteIOTotal statistics in profile by zzzxl1993 · Pull Request #44863 · apache/doris

zzzxl1993 · 2024-12-02T08:40:58Z

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #43542

Problem Summary:

Release note

Add NumInvertedIndexRemoteIOTotal to count the number of remote IO operations in the inverted index

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

doris-robot · 2024-12-02T08:41:05Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

zzzxl1993 · 2024-12-03T11:45:39Z

run buildall

zzzxl1993 · 2024-12-03T11:50:52Z

run buildall

doris-robot · 2024-12-03T13:33:02Z

TPC-H: Total hot run time: 40274 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fce7745ec65b1a7234720bccf6040ce5c6c8269d, data reload: false
------ Round 1 ----------------------------------
q1	17717	7514	7323	7323
q2	2049	175	172	172
q3	10643	1109	1193	1109
q4	10284	764	797	764
q5	7618	2821	2775	2775
q6	244	147	147	147
q7	1021	634	593	593
q8	9237	1905	1945	1905
q9	6714	6504	6518	6504
q10	7049	2320	2305	2305
q11	458	259	269	259
q12	421	220	224	220
q13	17787	3035	3042	3035
q14	244	214	217	214
q15	576	539	520	520
q16	654	586	595	586
q17	984	570	577	570
q18	7229	6642	6722	6642
q19	1325	989	1066	989
q20	462	183	178	178
q21	4040	3151	3215	3151
q22	387	319	313	313
Total cold run time: 107143 ms
Total hot run time: 40274 ms
----- Round 2, with runtime_filter_mode=off -----
q1	7278	7246	7226	7226
q2	332	231	229	229
q3	2947	2850	2933	2850
q4	2075	1817	1842	1817
q5	5720	5703	5665	5665
q6	228	136	141	136
q7	2246	1794	1817	1794
q8	3475	3536	3526	3526
q9	8961	9063	8985	8985
q10	3624	3594	3556	3556
q11	595	508	509	508
q12	831	621	624	621
q13	10667	3263	3245	3245
q14	313	279	264	264
q15	585	527	542	527
q16	692	645	660	645
q17	1855	1650	1609	1609
q18	8328	7795	7588	7588
q19	1719	1466	1607	1466
q20	2130	1848	1889	1848
q21	5653	5574	5405	5405
q22	637	592	590	590
Total cold run time: 70891 ms
Total hot run time: 60100 ms

doris-robot · 2024-12-03T13:45:47Z

TPC-DS: Total hot run time: 197730 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fce7745ec65b1a7234720bccf6040ce5c6c8269d, data reload: false
query1	1259	989	996	989
query2	6257	2114	2077	2077
query3	10968	4524	4559	4524
query4	67763	28733	23757	23757
query5	5073	466	451	451
query6	425	198	180	180
query7	5641	307	301	301
query8	336	243	260	243
query9	8971	2655	2645	2645
query10	453	242	243	242
query11	17490	15366	15786	15366
query12	157	111	103	103
query13	1585	395	403	395
query14	10046	7013	7507	7013
query15	214	180	188	180
query16	7615	477	484	477
query17	1063	583	611	583
query18	2027	316	307	307
query19	203	162	153	153
query20	120	115	116	115
query21	221	106	116	106
query22	4745	4539	4338	4338
query23	35652	35308	34491	34491
query24	5485	2626	2564	2564
query25	499	384	391	384
query26	638	165	150	150
query27	1846	284	282	282
query28	4645	2491	2490	2490
query29	695	436	422	422
query30	211	151	147	147
query31	1047	851	867	851
query32	82	59	56	56
query33	403	294	294	294
query34	914	510	520	510
query35	919	794	747	747
query36	1056	983	956	956
query37	122	77	77	77
query38	4572	4475	4498	4475
query39	1504	1481	1488	1481
query40	202	98	98	98
query41	47	41	42	41
query42	115	97	99	97
query43	551	516	504	504
query44	1196	834	829	829
query45	201	176	178	176
query46	1197	722	714	714
query47	2012	1919	1928	1919
query48	418	311	339	311
query49	728	389	379	379
query50	845	407	414	407
query51	7426	7298	7120	7120
query52	98	84	84	84
query53	255	179	179	179
query54	508	398	385	385
query55	74	79	78	78
query56	252	243	240	240
query57	1230	1120	1116	1116
query58	215	204	219	204
query59	3258	3177	2909	2909
query60	266	255	243	243
query61	109	106	103	103
query62	776	683	690	683
query63	220	185	188	185
query64	1377	678	664	664
query65	3306	3192	3247	3192
query66	729	305	313	305
query67	16088	15756	15570	15570
query68	4197	571	558	558
query69	484	251	245	245
query70	1172	1176	1123	1123
query71	330	255	248	248
query72	6158	4013	4043	4013
query73	772	351	352	351
query74	10210	9083	9094	9083
query75	3399	2711	2657	2657
query76	1878	1112	1203	1112
query77	487	255	264	255
query78	10429	9532	9487	9487
query79	2335	600	610	600
query80	1282	434	430	430
query81	521	229	228	228
query82	1103	119	121	119
query83	251	147	160	147
query84	277	75	70	70
query85	1045	296	296	296
query86	432	316	303	303
query87	4811	4599	4498	4498
query88	4121	2129	2117	2117
query89	418	301	289	289
query90	1993	187	178	178
query91	141	105	102	102
query92	69	50	48	48
query93	2835	581	540	540
query94	892	311	296	296
query95	347	256	252	252
query96	622	275	271	271
query97	2860	2708	2659	2659
query98	222	193	200	193
query99	1647	1316	1305	1305
Total cold run time: 324361 ms
Total hot run time: 197730 ms

doris-robot · 2024-12-03T13:51:16Z

ClickBench: Total hot run time: 32.2 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit fce7745ec65b1a7234720bccf6040ce5c6c8269d, data reload: false
query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.23	0.07	0.07
query4	1.62	0.10	0.11
query5	0.43	0.42	0.42
query6	1.16	0.65	0.66
query7	0.02	0.02	0.02
query8	0.04	0.03	0.04
query9	0.58	0.53	0.51
query10	0.56	0.56	0.56
query11	0.14	0.11	0.11
query12	0.13	0.11	0.12
query13	0.62	0.61	0.60
query14	2.69	2.73	2.75
query15	0.89	0.84	0.83
query16	0.39	0.38	0.41
query17	1.06	1.04	1.06
query18	0.23	0.21	0.21
query19	1.84	1.86	1.91
query20	0.01	0.02	0.01
query21	15.36	0.56	0.59
query22	2.57	2.60	1.92
query23	17.11	0.88	0.73
query24	2.99	1.35	0.46
query25	0.16	0.11	0.11
query26	0.59	0.14	0.15
query27	0.05	0.05	0.04
query28	10.93	1.11	1.08
query29	12.54	3.27	3.31
query30	0.25	0.06	0.06
query31	2.87	0.37	0.38
query32	3.30	0.46	0.45
query33	2.98	3.02	3.13
query34	17.00	4.53	4.44
query35	4.59	4.47	4.48
query36	0.67	0.50	0.48
query37	0.08	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.03
query40	0.17	0.13	0.12
query41	0.08	0.02	0.03
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 107.17 s
Total hot run time: 32.2 s

doris-robot · 2024-12-03T13:52:01Z

TeamCity be ut coverage result:
Function Coverage: 38.51% (10009/25988)
Line Coverage: 29.52% (83842/284053)
Region Coverage: 28.63% (43114/150566)
Branch Coverage: 25.24% (21917/86824)
Coverage Report: http://coverage.selectdb-in.cc/coverage/fce7745ec65b1a7234720bccf6040ce5c6c8269d_fce7745ec65b1a7234720bccf6040ce5c6c8269d/report/index.html

qidaye

LGTM

… profile

zzzxl1993 · 2024-12-09T04:14:37Z

run buildall

doris-robot · 2024-12-09T05:53:17Z

TPC-H: Total hot run time: 40352 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7d4f781762a32c6851901dd8daef97bb51792404, data reload: false
------ Round 1 ----------------------------------
q1	17629	7589	7356	7356
q2	2060	177	172	172
q3	10906	1138	1193	1138
q4	10438	816	688	688
q5	7618	2782	2699	2699
q6	251	148	147	147
q7	1025	614	599	599
q8	9232	1860	1962	1860
q9	6677	6559	6565	6559
q10	7062	2300	2323	2300
q11	470	271	261	261
q12	421	222	220	220
q13	17784	3013	3030	3013
q14	256	222	232	222
q15	573	540	531	531
q16	648	579	600	579
q17	1007	540	520	520
q18	7377	6894	6773	6773
q19	1359	1047	980	980
q20	446	181	182	181
q21	4050	3245	3234	3234
q22	374	322	320	320
Total cold run time: 107663 ms
Total hot run time: 40352 ms
----- Round 2, with runtime_filter_mode=off -----
q1	7298	7264	7277	7264
q2	326	230	225	225
q3	2950	2937	2921	2921
q4	2085	1876	1846	1846
q5	5619	5681	5720	5681
q6	231	139	142	139
q7	2246	1824	1783	1783
q8	3403	3520	3537	3520
q9	8921	9060	9099	9060
q10	3614	3582	3552	3552
q11	622	509	498	498
q12	796	632	629	629
q13	9920	3229	3214	3214
q14	315	277	272	272
q15	588	527	534	527
q16	667	629	643	629
q17	1856	1637	1604	1604
q18	8271	7813	7568	7568
q19	1682	1618	1683	1618
q20	2122	1904	1854	1854
q21	5748	5435	5563	5435
q22	635	582	552	552
Total cold run time: 69915 ms
Total hot run time: 60391 ms

doris-robot · 2024-12-09T06:01:49Z

TeamCity be ut coverage result:
Function Coverage: 38.48% (10007/26009)
Line Coverage: 29.49% (83899/284529)
Region Coverage: 28.59% (43126/150819)
Branch Coverage: 25.19% (21923/87030)
Coverage Report: http://coverage.selectdb-in.cc/coverage/7d4f781762a32c6851901dd8daef97bb51792404_7d4f781762a32c6851901dd8daef97bb51792404/report/index.html

doris-robot · 2024-12-09T06:05:34Z

TPC-DS: Total hot run time: 197168 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7d4f781762a32c6851901dd8daef97bb51792404, data reload: false
query1	1265	976	947	947
query2	6235	2059	1966	1966
query3	11080	4594	4764	4594
query4	32948	23558	23475	23475
query5	3537	473	462	462
query6	281	191	181	181
query7	4005	321	305	305
query8	297	242	235	235
query9	9413	2652	2642	2642
query10	473	260	259	259
query11	17837	15149	15347	15149
query12	161	101	104	101
query13	1585	413	380	380
query14	10100	7473	6815	6815
query15	277	192	202	192
query16	8082	484	503	484
query17	1724	579	598	579
query18	2251	317	307	307
query19	367	152	154	152
query20	120	113	109	109
query21	213	105	134	105
query22	4794	4580	4314	4314
query23	35064	34474	34430	34430
query24	10508	2489	2449	2449
query25	612	379	400	379
query26	1365	151	144	144
query27	2449	301	278	278
query28	7224	2464	2455	2455
query29	828	408	405	405
query30	241	148	153	148
query31	1045	871	877	871
query32	99	57	94	57
query33	781	299	293	293
query34	1201	505	526	505
query35	895	783	777	777
query36	1108	964	945	945
query37	134	71	76	71
query38	4542	4444	4303	4303
query39	1535	1467	1484	1467
query40	206	96	96	96
query41	46	44	44	44
query42	110	101	98	98
query43	527	496	475	475
query44	1277	839	839	839
query45	185	166	180	166
query46	1179	725	734	725
query47	2025	1959	1999	1959
query48	404	315	315	315
query49	900	425	423	423
query50	832	382	394	382
query51	7323	7345	7293	7293
query52	107	96	91	91
query53	265	196	180	180
query54	1193	406	405	405
query55	84	82	79	79
query56	259	235	252	235
query57	1339	1129	1104	1104
query58	238	211	217	211
query59	3252	3071	2979	2979
query60	278	248	253	248
query61	107	112	120	112
query62	871	674	681	674
query63	217	197	191	191
query64	3953	663	645	645
query65	3319	3181	3188	3181
query66	834	300	303	300
query67	16344	15686	15716	15686
query68	4448	580	560	560
query69	424	260	254	254
query70	1213	1090	1128	1090
query71	353	247	259	247
query72	6543	4199	4059	4059
query73	756	354	364	354
query74	10415	9116	9176	9116
query75	3411	2682	2668	2668
query76	2555	1073	1090	1073
query77	360	270	281	270
query78	10368	9559	9620	9559
query79	2274	616	609	609
query80	1302	424	429	424
query81	537	248	225	225
query82	815	115	116	115
query83	241	142	143	142
query84	236	71	75	71
query85	1376	300	294	294
query86	436	307	299	299
query87	4636	4552	4586	4552
query88	3635	2196	2140	2140
query89	425	299	302	299
query90	2065	187	183	183
query91	145	108	104	104
query92	66	48	52	48
query93	2247	543	552	543
query94	1002	292	307	292
query95	352	250	245	245
query96	635	282	277	277
query97	2854	2694	2698	2694
query98	226	196	195	195
query99	1558	1323	1301	1301
Total cold run time: 302259 ms
Total hot run time: 197168 ms

doris-robot · 2024-12-09T06:11:02Z

ClickBench: Total hot run time: 32.82 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7d4f781762a32c6851901dd8daef97bb51792404, data reload: false
query1	0.04	0.03	0.04
query2	0.07	0.03	0.04
query3	0.23	0.08	0.07
query4	1.62	0.10	0.11
query5	0.42	0.40	0.41
query6	1.16	0.65	0.65
query7	0.02	0.02	0.01
query8	0.04	0.04	0.03
query9	0.57	0.52	0.49
query10	0.57	0.55	0.57
query11	0.14	0.11	0.10
query12	0.14	0.12	0.11
query13	0.61	0.60	0.60
query14	2.71	2.87	2.79
query15	0.91	0.84	0.83
query16	0.38	0.39	0.39
query17	1.02	1.07	1.06
query18	0.22	0.21	0.21
query19	1.86	1.88	2.02
query20	0.01	0.01	0.02
query21	15.35	0.58	0.57
query22	2.53	1.93	1.73
query23	17.11	0.95	0.78
query24	2.51	1.05	1.29
query25	0.32	0.08	0.04
query26	0.43	0.15	0.14
query27	0.04	0.04	0.04
query28	10.66	1.10	1.07
query29	12.54	3.34	3.34
query30	0.25	0.07	0.07
query31	2.85	0.38	0.37
query32	3.29	0.46	0.46
query33	2.96	3.01	3.02
query34	16.83	4.52	4.60
query35	4.52	4.52	4.56
query36	0.66	0.47	0.48
query37	0.09	0.07	0.05
query38	0.04	0.03	0.03
query39	0.03	0.02	0.03
query40	0.17	0.13	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.07 s
Total hot run time: 32.82 s

csun5285

LGTM

github-actions · 2024-12-12T04:15:13Z

PR approved by anyone and no changes requested.

github-actions · 2024-12-12T06:22:52Z

PR approved by at least one committer and no changes requested.

… profile (apache#44863) Add NumInvertedIndexRemoteIOTotal to count the number of remote IO operations in the inverted index Related PR: apache#43542

… profile #44863 (#45619) pick #44863

… profile (apache#44863) Add NumInvertedIndexRemoteIOTotal to count the number of remote IO operations in the inverted index Related PR: apache#43542

zzzxl1993 requested review from dataroaring, gavinchou and platoneko as code owners December 2, 2024 08:40

zzzxl1993 force-pushed the 202412021604 branch 3 times, most recently from 92d1b3c to 82865deCompare December 3, 2024 11:43

zzzxl1993 changed the title ~~[fix](inverted index) Fix Inaccurate Display of Inverted Index Profile Remote I/O~~[opt](inverted index) Add NumInvertedIndexRemoteIOTotal statistics in profileDec 3, 2024

zzzxl1993 force-pushed the 202412021604 branch from 82865de to fce7745Compare December 3, 2024 11:49

qidaye previously approved these changes Dec 6, 2024
View reviewed changes

xiaokang reviewed Dec 9, 2024
View reviewed changes

Comment threadbe/src/io/cache/cached_remote_file_reader.cpp

[opt](inverted index) Add NumInvertedIndexRemoteIOTotal statistics in…
7d4f781
… profile

zzzxl1993 dismissed qidaye’s stale review via 7d4f781December 9, 2024 04:14

zzzxl1993 force-pushed the 202412021604 branch from fce7745 to 7d4f781Compare December 9, 2024 04:14

csun5285 approved these changes Dec 12, 2024
View reviewed changes

github-actionsBot added the reviewed label Dec 12, 2024

qidaye approved these changes Dec 12, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Dec 12, 2024

gavinchou reviewed Dec 13, 2024
View reviewed changes

Comment threadbe/src/io/io_common.h

gavinchou added the dev/3.0.x label Dec 13, 2024

gavinchou approved these changes Dec 13, 2024
View reviewed changes

gavinchou merged commit 559a2f9 into apache:masterDec 13, 2024

github-actionsBot added the dev/3.0.x-conflict label Dec 13, 2024

zzzxl1993 mentioned this pull request Dec 19, 2024
[opt](inverted index) Add NumInvertedIndexRemoteIOTotal statistics in profile #44863 #45619
Merged

qidaye pushed a commit that referenced this pull request Dec 19, 2024
[opt](inverted index) Add NumInvertedIndexRemoteIOTotal statistics in…
e807b37
… profile #44863 (#45619) pick #44863

qidaye added dev/3.0.4-merged and removed dev/3.0.x dev/3.0.x-conflict labels Dec 19, 2024

gavinchou mentioned this pull request Feb 18, 2025
Release Notes 3.0.4 #48013
Closed

Uh oh!

Conversation

zzzxl1993 commented Dec 2, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

doris-robot commented Dec 2, 2024

Uh oh!

zzzxl1993 commented Dec 3, 2024

Uh oh!

zzzxl1993 commented Dec 3, 2024

Uh oh!

doris-robot commented Dec 3, 2024

Uh oh!

doris-robot commented Dec 3, 2024

Uh oh!

doris-robot commented Dec 3, 2024

Uh oh!

doris-robot commented Dec 3, 2024

Uh oh!

qidaye left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

zzzxl1993 commented Dec 9, 2024

Uh oh!

doris-robot commented Dec 9, 2024

Uh oh!

doris-robot commented Dec 9, 2024

Uh oh!

doris-robot commented Dec 9, 2024

Uh oh!

doris-robot commented Dec 9, 2024

Uh oh!

csun5285 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Dec 12, 2024

Uh oh!

github-actionsBot commented Dec 12, 2024

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

zzzxl1993 commented Dec 2, 2024•
edited
Loading