[Enhancement](inverted index) improve expr evaluate_inverted_index performace and remove useless code by airborne12 · Pull Request #40600 · apache/doris

airborne12 · 2024-09-10T07:33:57Z

Proposed changes

Pre-allocate vectors based on an estimated or known size in vexpr evaluated_inverted_index

doris-robot · 2024-09-10T07:34:01Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

airborne12 · 2024-09-10T07:34:15Z

run buildall

doris-robot · 2024-09-10T08:27:49Z

TeamCity be ut coverage result:
Function Coverage: 36.83% (9393/25505)
Line Coverage: 28.22% (77473/274494)
Region Coverage: 27.63% (40003/144776)
Branch Coverage: 24.26% (20351/83874)
Coverage Report: http://coverage.selectdb-in.cc/coverage/0498bc65aa4cde29033f85cc8a351547a5f0e5a4_0498bc65aa4cde29033f85cc8a351547a5f0e5a4/report/index.html

eldenmoon

LGTM

github-actions · 2024-09-10T09:48:00Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-09-10T09:48:02Z

PR approved by anyone and no changes requested.

doris-robot · 2024-09-10T12:53:19Z

TPC-H: Total hot run time: 38521 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0498bc65aa4cde29033f85cc8a351547a5f0e5a4, data reload: false
------ Round 1 ----------------------------------
q1	17602	4622	4435	4435
q2	2022	193	184	184
q3	11841	996	1109	996
q4	10509	712	705	705
q5	7770	2925	2845	2845
q6	233	140	139	139
q7	975	621	634	621
q8	9320	2128	2120	2120
q9	7276	6623	6653	6623
q10	7014	2256	2217	2217
q11	442	245	244	244
q12	398	227	224	224
q13	18742	3113	3104	3104
q14	281	243	249	243
q15	543	505	483	483
q16	529	423	420	420
q17	1011	688	679	679
q18	7463	6826	6987	6826
q19	1389	1092	1046	1046
q20	685	341	324	324
q21	4407	3041	3106	3041
q22	1112	1009	1002	1002
Total cold run time: 111564 ms
Total hot run time: 38521 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4393	4360	4343	4343
q2	391	272	273	272
q3	2867	2641	2735	2641
q4	1968	1662	1726	1662
q5	5748	5713	5844	5713
q6	234	134	135	134
q7	2270	1828	1824	1824
q8	3339	3509	3565	3509
q9	8904	8875	8845	8845
q10	3707	3402	3390	3390
q11	612	517	527	517
q12	865	685	683	683
q13	13060	3362	3330	3330
q14	325	290	286	286
q15	534	475	512	475
q16	571	530	516	516
q17	1890	1608	1548	1548
q18	8157	7888	7912	7888
q19	1775	1579	1654	1579
q20	2170	1913	1953	1913
q21	5785	5414	5379	5379
q22	1138	1055	1018	1018
Total cold run time: 70703 ms
Total hot run time: 57465 ms

doris-robot · 2024-09-10T13:05:33Z

TPC-DS: Total hot run time: 197888 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0498bc65aa4cde29033f85cc8a351547a5f0e5a4, data reload: false
query1	1262	895	873	873
query2	6301	1978	1917	1917
query3	10622	3873	4127	3873
query4	59617	27944	23527	23527
query5	4990	497	495	495
query6	389	167	175	167
query7	5632	294	289	289
query8	306	214	213	213
query9	7561	2509	2493	2493
query10	430	264	267	264
query11	17428	15095	15382	15095
query12	145	103	103	103
query13	1463	409	386	386
query14	10346	7451	7196	7196
query15	201	186	170	170
query16	6450	460	475	460
query17	1127	572	572	572
query18	1465	294	290	290
query19	214	149	139	139
query20	122	111	107	107
query21	218	105	105	105
query22	4643	4418	4910	4418
query23	34276	33760	33616	33616
query24	6018	2901	2788	2788
query25	491	395	382	382
query26	608	149	152	149
query27	1599	272	279	272
query28	3671	2055	2036	2036
query29	631	402	403	402
query30	233	149	158	149
query31	946	769	785	769
query32	73	53	54	53
query33	388	283	293	283
query34	876	469	463	463
query35	837	738	715	715
query36	1069	935	903	903
query37	145	87	81	81
query38	4026	3870	3914	3870
query39	1447	1427	1417	1417
query40	214	116	119	116
query41	48	49	47	47
query42	116	95	97	95
query43	500	490	482	482
query44	1071	766	747	747
query45	204	169	173	169
query46	1123	728	730	728
query47	1877	1795	1813	1795
query48	368	295	303	295
query49	760	436	442	436
query50	826	407	407	407
query51	6978	6932	6871	6871
query52	101	85	85	85
query53	250	175	178	175
query54	555	446	451	446
query55	74	73	74	73
query56	281	252	252	252
query57	1241	1039	1090	1039
query58	213	239	236	236
query59	3131	2989	2915	2915
query60	290	270	302	270
query61	99	101	95	95
query62	755	656	657	656
query63	219	183	184	183
query64	1344	680	681	680
query65	3261	3172	3180	3172
query66	628	339	338	338
query67	15952	15434	15396	15396
query68	1600	848	861	848
query69	418	331	325	325
query70	1174	1167	1170	1167
query71	345	339	344	339
query72	4581	3544	3505	3505
query73	591	570	578	570
query74	9046	8979	8939	8939
query75	3005	2981	2969	2969
query76	926	843	823	823
query77	544	399	408	399
query78	9282	9364	9198	9198
query79	903	890	846	846
query80	803	786	784	784
query81	449	264	263	263
query82	267	264	258	258
query83	193	187	187	187
query84	202	107	106	106
query85	610	453	500	453
query86	301	314	296	296
query87	4263	4382	4284	4284
query88	4194	4079	4060	4060
query89	373	354	359	354
query90	1294	308	306	306
query91	121	120	127	120
query92	78	76	78	76
query93	909	899	897	897
query94	477	391	360	360
query95	411	409	403	403
query96	464	473	467	467
query97	3078	3086	3113	3086
query98	229	239	230	230
query99	1333	1271	1283	1271
Total cold run time: 293209 ms
Total hot run time: 197888 ms

doris-robot · 2024-09-10T13:11:04Z

ClickBench: Total hot run time: 32.25 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0498bc65aa4cde29033f85cc8a351547a5f0e5a4, data reload: false
query1	0.05	0.05	0.04
query2	0.08	0.04	0.03
query3	0.23	0.05	0.05
query4	1.66	0.08	0.08
query5	0.52	0.50	0.52
query6	1.13	0.74	0.73
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.53	0.49	0.49
query10	0.54	0.57	0.53
query11	0.16	0.12	0.12
query12	0.15	0.13	0.12
query13	0.60	0.59	0.59
query14	1.40	1.41	1.38
query15	0.85	0.86	0.84
query16	0.39	0.38	0.39
query17	1.07	1.06	1.01
query18	0.21	0.20	0.20
query19	1.99	1.81	1.80
query20	0.01	0.02	0.01
query21	15.39	0.66	0.65
query22	4.31	6.23	2.56
query23	18.31	1.33	1.28
query24	2.19	0.23	0.21
query25	0.14	0.08	0.08
query26	0.27	0.17	0.18
query27	0.07	0.08	0.07
query28	13.22	1.04	1.01
query29	12.59	3.41	3.41
query30	0.24	0.05	0.05
query31	2.86	0.39	0.39
query32	3.27	0.48	0.48
query33	3.04	3.08	3.02
query34	17.03	4.42	4.44
query35	4.43	4.50	4.44
query36	0.66	0.47	0.48
query37	0.18	0.16	0.15
query38	0.16	0.15	0.15
query39	0.04	0.03	0.04
query40	0.15	0.12	0.12
query41	0.09	0.06	0.05
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 110.39 s
Total hot run time: 32.25 s

…rformace and remove useless code

airborne12 · 2024-09-11T01:06:22Z

run buildall

doris-robot · 2024-09-11T01:40:13Z

TeamCity be ut coverage result:
Function Coverage: 36.84% (9396/25507)
Line Coverage: 28.24% (77515/274509)
Region Coverage: 27.63% (40009/144784)
Branch Coverage: 24.25% (20344/83880)
Coverage Report: http://coverage.selectdb-in.cc/coverage/77a507ce26a77e5205ec5cf78899f724d7ac57ab_77a507ce26a77e5205ec5cf78899f724d7ac57ab/report/index.html

doris-robot · 2024-09-11T01:43:43Z

TPC-H: Total hot run time: 37957 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 77a507ce26a77e5205ec5cf78899f724d7ac57ab, data reload: false
------ Round 1 ----------------------------------
q1	17635	4847	4322	4322
q2	2021	195	188	188
q3	11823	926	1120	926
q4	10505	688	822	688
q5	7750	2790	2786	2786
q6	228	138	137	137
q7	964	609	590	590
q8	9536	2089	2077	2077
q9	7549	6528	6543	6528
q10	7011	2225	2251	2225
q11	444	252	244	244
q12	397	225	226	225
q13	18124	3080	3093	3080
q14	292	247	251	247
q15	539	484	482	482
q16	527	436	433	433
q17	986	748	667	667
q18	7360	6870	6872	6870
q19	1387	1053	919	919
q20	682	334	341	334
q21	3963	3023	2960	2960
q22	1105	1046	1029	1029
Total cold run time: 110828 ms
Total hot run time: 37957 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4359	4281	4310	4281
q2	367	272	274	272
q3	2893	2610	2624	2610
q4	1910	1639	1659	1639
q5	5704	5718	5768	5718
q6	231	141	135	135
q7	2249	1845	1829	1829
q8	3260	3411	3482	3411
q9	8842	8819	8844	8819
q10	3628	3379	3383	3379
q11	602	512	518	512
q12	828	677	656	656
q13	14027	3225	3291	3225
q14	327	286	299	286
q15	540	508	480	480
q16	538	506	493	493
q17	1832	1554	1533	1533
q18	8144	7808	7977	7808
q19	1753	1665	1487	1487
q20	2176	1918	1936	1918
q21	5746	5487	5472	5472
q22	1099	1026	1065	1026
Total cold run time: 71055 ms
Total hot run time: 56989 ms

doris-robot · 2024-09-11T01:55:54Z

TPC-DS: Total hot run time: 196574 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 77a507ce26a77e5205ec5cf78899f724d7ac57ab, data reload: false
query1	1266	932	869	869
query2	6366	1957	1871	1871
query3	10625	4014	3939	3939
query4	60025	26259	23175	23175
query5	4979	495	493	493
query6	393	163	170	163
query7	5644	302	289	289
query8	327	225	215	215
query9	7692	2510	2500	2500
query10	408	296	263	263
query11	17363	15108	15398	15108
query12	154	106	100	100
query13	1439	398	416	398
query14	10726	6773	7057	6773
query15	210	181	174	174
query16	6795	454	504	454
query17	1091	573	587	573
query18	1462	323	304	304
query19	212	151	154	151
query20	129	115	114	114
query21	201	109	104	104
query22	4728	4419	4847	4419
query23	34356	33347	33430	33347
query24	6035	2827	2848	2827
query25	493	366	388	366
query26	611	165	150	150
query27	1572	272	277	272
query28	3877	2024	1999	1999
query29	663	404	409	404
query30	240	152	149	149
query31	923	790	730	730
query32	68	51	53	51
query33	424	295	282	282
query34	884	464	474	464
query35	828	723	703	703
query36	1039	943	926	926
query37	147	85	91	85
query38	3953	3886	3860	3860
query39	1468	1531	1406	1406
query40	196	115	111	111
query41	46	46	45	45
query42	116	92	94	92
query43	508	479	467	467
query44	1105	743	737	737
query45	197	167	166	166
query46	1078	775	737	737
query47	1909	1823	1844	1823
query48	380	291	297	291
query49	764	461	431	431
query50	811	418	411	411
query51	6919	6940	6897	6897
query52	103	82	85	82
query53	249	178	171	171
query54	560	461	459	459
query55	78	73	72	72
query56	263	282	258	258
query57	1220	1079	1115	1079
query58	227	230	225	225
query59	3022	2842	2882	2842
query60	288	265	269	265
query61	101	99	99	99
query62	789	642	659	642
query63	217	182	179	179
query64	1368	678	660	660
query65	3247	3165	3157	3157
query66	664	332	358	332
query67	15743	15652	15390	15390
query68	1762	841	868	841
query69	421	328	324	324
query70	1161	1164	1180	1164
query71	345	338	337	337
query72	4602	3703	3487	3487
query73	581	580	576	576
query74	9054	8842	8898	8842
query75	2973	2904	2967	2904
query76	1000	849	835	835
query77	507	399	398	398
query78	9337	9229	9252	9229
query79	892	876	849	849
query80	783	866	787	787
query81	448	254	273	254
query82	259	260	265	260
query83	189	189	185	185
query84	190	107	103	103
query85	624	394	379	379
query86	312	315	308	308
query87	4397	4416	4255	4255
query88	4279	4141	4128	4128
query89	368	356	366	356
query90	1137	317	306	306
query91	121	120	122	120
query92	78	72	75	72
query93	899	889	890	889
query94	441	366	354	354
query95	415	407	413	407
query96	472	477	468	468
query97	3118	3062	3110	3062
query98	234	234	232	232
query99	1324	1273	1276	1273
Total cold run time: 294707 ms
Total hot run time: 196574 ms

doris-robot · 2024-09-11T02:01:25Z

ClickBench: Total hot run time: 32.01 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 77a507ce26a77e5205ec5cf78899f724d7ac57ab, data reload: false
query1	0.05	0.04	0.04
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.68	0.07	0.07
query5	0.52	0.50	0.50
query6	1.12	0.73	0.75
query7	0.02	0.01	0.01
query8	0.05	0.05	0.05
query9	0.54	0.48	0.50
query10	0.54	0.57	0.55
query11	0.15	0.12	0.11
query12	0.15	0.13	0.12
query13	0.60	0.59	0.59
query14	1.37	1.42	1.46
query15	0.82	0.82	0.81
query16	0.37	0.39	0.37
query17	1.00	1.03	1.00
query18	0.21	0.20	0.19
query19	1.86	1.80	1.83
query20	0.01	0.01	0.02
query21	15.39	0.67	0.65
query22	3.91	6.96	2.17
query23	18.33	1.32	1.31
query24	2.15	0.22	0.22
query25	0.14	0.09	0.07
query26	0.27	0.17	0.18
query27	0.07	0.08	0.07
query28	13.22	1.03	1.01
query29	12.62	3.56	3.58
query30	0.25	0.05	0.05
query31	2.87	0.41	0.38
query32	3.25	0.49	0.47
query33	3.00	2.98	3.02
query34	17.12	4.42	4.43
query35	4.44	4.45	4.51
query36	0.67	0.48	0.48
query37	0.18	0.16	0.16
query38	0.15	0.14	0.15
query39	0.04	0.04	0.04
query40	0.18	0.13	0.13
query41	0.10	0.05	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.81 s
Total hot run time: 32.01 s

zzzxl1993

LGTM

…rformace and remove useless code (apache#40600) ## Proposed changes Pre-allocate vectors based on an estimated or known size in vexpr evaluated_inverted_index

…rformace and remove useless code #40600 (#41122) cherry pick from #40600

…rformace and remove useless code (apache#40600) ## Proposed changes Pre-allocate vectors based on an estimated or known size in vexpr evaluated_inverted_index

…rformace and remove useless code #40600 (#41442) cherry pick from #40600

…rformace and remove useless code (apache#40600) ## Proposed changes Pre-allocate vectors based on an estimated or known size in vexpr evaluated_inverted_index

eldenmoon approved these changes Sep 10, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Sep 10, 2024

github-actionsBot added the reviewed label Sep 10, 2024

[Enhancement](inverted index) improve expr evaluate_inverted_index pe…
77a507c
…rformace and remove useless code

airborne12 force-pushed the enhance branch from 0498bc6 to 77a507cCompare September 11, 2024 01:06

zzzxl1993 approved these changes Sep 11, 2024
View reviewed changes

airborne12 merged commit faee56f into apache:masterSep 11, 2024

airborne12 deleted the enhance branch September 11, 2024 06:13

airborne12 mentioned this pull request Sep 23, 2024
[Enhancement](inverted index) improve expr evaluate_inverted_index performace and remove useless code #40600 #41122
Merged

airborne12 added a commit that referenced this pull request Sep 23, 2024
[Enhancement](inverted index) improve expr evaluate_inverted_index pe…
fbd5e76
…rformace and remove useless code #40600 (#41122) cherry pick from #40600

airborne12 added the dev/3.0.2-merged label Sep 23, 2024

airborne12 mentioned this pull request Sep 29, 2024
[Enhancement](inverted index) improve expr evaluate_inverted_index performace and remove useless code #40600 #41442
Merged

airborne12 added a commit that referenced this pull request Sep 29, 2024
[Enhancement](inverted index) improve expr evaluate_inverted_index pe…
b3bb4f4
…rformace and remove useless code #40600 (#41442) cherry pick from #40600

airborne12 added the dev/2.1.7-merged label Sep 29, 2024

Uh oh!

Conversation

airborne12 commented Sep 10, 2024

Proposed changes

Uh oh!

doris-robot commented Sep 10, 2024

Uh oh!

airborne12 commented Sep 10, 2024

Uh oh!

doris-robot commented Sep 10, 2024

Uh oh!

eldenmoon left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Sep 10, 2024

Uh oh!

github-actionsBot commented Sep 10, 2024

Uh oh!

doris-robot commented Sep 10, 2024

Uh oh!

doris-robot commented Sep 10, 2024

Uh oh!

doris-robot commented Sep 10, 2024

Uh oh!

airborne12 commented Sep 11, 2024

Uh oh!

doris-robot commented Sep 11, 2024

Uh oh!

doris-robot commented Sep 11, 2024

Uh oh!

doris-robot commented Sep 11, 2024

Uh oh!

doris-robot commented Sep 11, 2024

Uh oh!

zzzxl1993 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants