Skip to content

branch-3.0: [feature](index)Support light index add for inverted index without parser and ngram bf index #52894 - #52988

Merged
dataroaring merged 3 commits into
apache:branch-3.0from
airborne12:pick_52894_to_origin_branch-3.0
Jul 15, 2025
Merged

branch-3.0: [feature](index)Support light index add for inverted index without parser and ngram bf index #52894#52988
dataroaring merged 3 commits into
apache:branch-3.0from
airborne12:pick_52894_to_origin_branch-3.0

Conversation

@airborne12

@airborne12airborne12 commented Jul 9, 2025

Copy link
Copy Markdown
Member

cherry pick from #52894

@airborne12@qidaye
branch-3.1: [feature](index)Support light index add for inverted inde…
f513c2f
…x without parser and ngram bf index apache#48461apache#52251 (apache#52894)
cherry pick from apache#48461 and apache#52251
---------
Co-authored-by: qiye <jianliang5669@gmail.com>
@airborne12

airborne12 commented Jul 9, 2025

Copy link
Copy Markdown
MemberAuthor

run buildall

@hello-stephen

hello-stephen commented Jul 9, 2025

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@airborne12

airborne12 commented Jul 9, 2025

Copy link
Copy Markdown
MemberAuthor

run buildall

@airborne12airborne12 changed the title branch-3.0: branch-3.1: [feature](index)Support light index add for inverted index without parser and ngram bf index #52894branch-3.0: [feature](index)Support light index add for inverted index without parser and ngram bf index #52894Jul 9, 2025
@doris-robot

doris-robot commented Jul 9, 2025

Copy link
Copy Markdown
TPC-H: Total hot run time: 39607 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4fca5f36b6b4e1dd35fd7dd4c8d72aa66bf53f0c, data reload: false
------ Round 1 ----------------------------------
q1	17603	6907	6621	6621
q2	2094	171	159	159
q3	10651	1125	1204	1125
q4	10412	724	707	707
q5	7706	2804	2749	2749
q6	215	132	131	131
q7	984	622	613	613
q8	9361	1956	2006	1956
q9	6703	6354	6331	6331
q10	6997	2190	2262	2190
q11	464	260	249	249
q12	410	217	213	213
q13	17802	2985	3001	2985
q14	245	212	209	209
q15	502	463	464	463
q16	478	382	373	373
q17	967	547	595	547
q18	7250	6655	6569	6569
q19	1385	1120	1118	1118
q20	471	195	200	195
q21	4063	3181	3112	3112
q22	1093	992	1013	992
Total cold run time: 107856 ms
Total hot run time: 39607 ms
----- Round 2, with runtime_filter_mode=off -----
q1	6607	6552	6629	6552
q2	326	231	224	224
q3	2887	2910	2904	2904
q4	2040	1793	1893	1793
q5	5702	5735	5729	5729
q6	214	130	130	130
q7	2230	1815	1872	1815
q8	3345	3528	3542	3528
q9	8762	8872	8869	8869
q10	3585	3507	3496	3496
q11	593	499	495	495
q12	816	607	605	605
q13	8485	3126	3136	3126
q14	300	294	270	270
q15	515	462	490	462
q16	500	470	466	466
q17	1849	1665	1638	1638
q18	8389	7843	7828	7828
q19	1642	1480	1548	1480
q20	2052	1865	1863	1863
q21	5198	4900	4914	4900
q22	1130	1031	1030	1030
Total cold run time: 67167 ms
Total hot run time: 59203 ms

@doris-robot

doris-robot commented Jul 9, 2025

Copy link
Copy Markdown
TPC-DS: Total hot run time: 196699 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4fca5f36b6b4e1dd35fd7dd4c8d72aa66bf53f0c, data reload: false
query1	1272	931	879	879
query2	6250	1924	2004	1924
query3	10909	4391	4588	4391
query4	33136	23545	23431	23431
query5	3692	462	453	453
query6	262	178	196	178
query7	3992	313	328	313
query8	288	247	219	219
query9	9301	2587	2579	2579
query10	462	255	261	255
query11	17856	15202	15224	15202
query12	167	106	102	102
query13	1558	441	434	434
query14	9537	6771	7094	6771
query15	249	189	195	189
query16	8073	469	512	469
query17	1669	615	584	584
query18	2154	346	300	300
query19	275	162	158	158
query20	119	113	117	113
query21	209	105	108	105
query22	4697	4505	4342	4342
query23	35184	34331	34415	34331
query24	11156	2818	2846	2818
query25	644	425	411	411
query26	1282	163	165	163
query27	2473	356	348	348
query28	7358	2191	2151	2151
query29	857	443	450	443
query30	269	153	159	153
query31	1063	839	820	820
query32	93	52	58	52
query33	756	294	294	294
query34	955	525	513	513
query35	849	730	741	730
query36	1100	964	939	939
query37	139	70	70	70
query38	4092	4023	3932	3932
query39	1513	1471	1639	1471
query40	212	103	101	101
query41	50	51	48	48
query42	117	104	100	100
query43	533	499	471	471
query44	1325	823	826	823
query45	197	168	172	168
query46	1166	741	740	740
query47	2042	1914	1943	1914
query48	490	392	419	392
query49	969	432	412	412
query50	846	417	418	417
query51	7385	7111	7248	7111
query52	103	90	94	90
query53	268	187	187	187
query54	1284	484	493	484
query55	82	81	82	81
query56	275	270	264	264
query57	1347	1212	1214	1212
query58	239	220	220	220
query59	3289	3102	3064	3064
query60	295	254	261	254
query61	109	111	110	110
query62	866	690	697	690
query63	229	188	187	187
query64	4171	674	637	637
query65	3388	3296	3324	3296
query66	771	285	289	285
query67	15872	15566	15503	15503
query68	4485	572	584	572
query69	450	260	274	260
query70	1165	1098	1071	1071
query71	329	248	253	248
query72	6295	4021	4127	4021
query73	746	354	348	348
query74	10134	9249	9042	9042
query75	3362	2602	2665	2602
query76	2655	1026	1105	1026
query77	384	275	275	275
query78	10628	9527	9650	9527
query79	1844	601	617	601
query80	1219	427	455	427
query81	541	219	219	219
query82	921	86	84	84
query83	220	138	145	138
query84	238	79	81	79
query85	1370	298	291	291
query86	445	302	297	297
query87	4450	4233	4222	4222
query88	3604	2390	2371	2371
query89	413	294	293	293
query90	1944	180	183	180
query91	178	146	149	146
query92	69	50	47	47
query93	2133	555	561	555
query94	842	303	299	299
query95	358	259	257	257
query96	621	277	278	277
query97	3270	3116	3126	3116
query98	212	199	197	197
query99	1513	1330	1320	1320
Total cold run time: 300868 ms
Total hot run time: 196699 ms

@doris-robot

doris-robot commented Jul 9, 2025

Copy link
Copy Markdown
ClickBench: Total hot run time: 29.24 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4fca5f36b6b4e1dd35fd7dd4c8d72aa66bf53f0c, data reload: false
query1	0.04	0.03	0.03
query2	0.07	0.03	0.02
query3	0.23	0.07	0.07
query4	1.62	0.10	0.10
query5	0.50	0.49	0.49
query6	1.12	0.72	0.73
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.56	0.50	0.50
query10	0.54	0.54	0.56
query11	0.14	0.10	0.11
query12	0.14	0.11	0.11
query13	0.60	0.59	0.59
query14	0.76	0.79	0.81
query15	0.84	0.81	0.82
query16	0.37	0.38	0.38
query17	1.07	0.95	1.02
query18	0.23	0.23	0.22
query19	1.86	1.88	1.80
query20	0.01	0.01	0.01
query21	15.42	0.59	0.58
query22	2.50	1.82	1.04
query23	16.95	0.90	0.80
query24	3.43	0.87	0.86
query25	0.20	0.15	0.14
query26	0.36	0.13	0.13
query27	0.03	0.05	0.04
query28	10.35	0.53	0.46
query29	12.61	3.20	3.20
query30	0.25	0.06	0.06
query31	2.88	0.38	0.38
query32	3.24	0.47	0.45
query33	2.96	3.02	3.00
query34	17.26	4.50	4.55
query35	4.54	4.61	4.56
query36	0.62	0.47	0.47
query37	0.09	0.06	0.06
query38	0.05	0.03	0.03
query39	0.03	0.02	0.02
query40	0.15	0.12	0.12
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 104.83 s
Total hot run time: 29.24 s

@airborne12

airborne12 commented Jul 15, 2025

Copy link
Copy Markdown
MemberAuthor

run buildall

@doris-robot

doris-robot commented Jul 15, 2025

Copy link
Copy Markdown
TPC-H: Total hot run time: 40069 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fd7e095acfea5f1e7461be6677d18f7d20a823dd, data reload: false
------ Round 1 ----------------------------------
q1	17631	6759	6646	6646
q2	2076	203	185	185
q3	10481	1110	1134	1110
q4	10211	762	719	719
q5	7743	2865	2795	2795
q6	212	133	133	133
q7	973	635	610	610
q8	9349	1994	2031	1994
q9	6693	6370	6419	6370
q10	6975	2258	2330	2258
q11	454	265	265	265
q12	397	214	219	214
q13	17791	2990	3002	2990
q14	243	210	209	209
q15	511	489	473	473
q16	455	381	375	375
q17	967	654	629	629
q18	7454	6573	6818	6573
q19	1406	1063	1029	1029
q20	480	203	207	203
q21	3946	3313	3301	3301
q22	1119	988	1004	988
Total cold run time: 107567 ms
Total hot run time: 40069 ms
----- Round 2, with runtime_filter_mode=off -----
q1	6604	6578	6583	6578
q2	326	231	237	231
q3	2883	2905	2952	2905
q4	2033	1825	1900	1825
q5	5686	5715	5731	5715
q6	215	131	132	131
q7	2246	1810	1856	1810
q8	3362	3501	3537	3501
q9	8815	8807	8904	8807
q10	3572	3567	3516	3516
q11	607	501	492	492
q12	834	579	602	579
q13	10171	3194	3146	3146
q14	300	267	281	267
q15	506	463	462	462
q16	487	439	452	439
q17	1855	1609	1632	1609
q18	8222	7789	7674	7674
q19	1721	1619	1561	1561
q20	2088	1874	1841	1841
q21	5314	5080	5174	5080
q22	1135	1070	1078	1070
Total cold run time: 68982 ms
Total hot run time: 59239 ms

@doris-robot

doris-robot commented Jul 15, 2025

Copy link
Copy Markdown
TPC-DS: Total hot run time: 197005 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fd7e095acfea5f1e7461be6677d18f7d20a823dd, data reload: false
query1	1274	924	893	893
query2	6277	1859	1865	1859
query3	10929	4472	4575	4472
query4	33070	23857	23418	23418
query5	4084	448	439	439
query6	280	180	189	180
query7	3988	314	321	314
query8	290	234	218	218
query9	9481	2572	2553	2553
query10	473	253	247	247
query11	17845	15139	15160	15139
query12	152	102	101	101
query13	1543	420	403	403
query14	8442	6977	7550	6977
query15	231	178	197	178
query16	7843	504	498	498
query17	1638	636	611	611
query18	2068	320	317	317
query19	214	179	168	168
query20	124	118	112	112
query21	203	105	104	104
query22	4783	4461	4644	4461
query23	35149	33817	33745	33745
query24	11928	2872	2959	2872
query25	733	427	425	425
query26	1850	175	169	169
query27	3056	359	364	359
query28	7568	2172	2201	2172
query29	1113	476	465	465
query30	261	166	166	166
query31	1086	806	827	806
query32	93	58	58	58
query33	773	309	308	308
query34	1076	506	519	506
query35	850	766	723	723
query36	1118	918	995	918
query37	269	66	69	66
query38	4084	3906	3995	3906
query39	1608	1471	1478	1471
query40	253	105	98	98
query41	52	48	48	48
query42	114	102	103	102
query43	509	465	483	465
query44	1268	826	828	826
query45	193	176	170	170
query46	1170	741	737	737
query47	2047	1910	1949	1910
query48	489	386	386	386
query49	1230	409	393	393
query50	846	419	429	419
query51	7399	7427	7318	7318
query52	99	92	89	89
query53	253	183	191	183
query54	1196	487	503	487
query55	78	81	80	80
query56	265	274	245	245
query57	1316	1208	1222	1208
query58	229	210	215	210
query59	3227	3086	3072	3072
query60	281	257	263	257
query61	132	117	112	112
query62	884	690	722	690
query63	220	187	190	187
query64	5067	691	640	640
query65	3414	3293	3289	3289
query66	1412	304	303	303
query67	16171	15697	15763	15697
query68	4849	584	567	567
query69	439	261	264	261
query70	1147	1142	1137	1137
query71	325	258	254	254
query72	6245	4093	4093	4093
query73	765	352	366	352
query74	10542	8971	9077	8971
query75	3300	2667	2649	2649
query76	2500	1098	1075	1075
query77	383	277	271	271
query78	10563	9696	9612	9612
query79	1887	582	606	582
query80	1176	432	410	410
query81	543	221	213	213
query82	931	90	92	90
query83	243	148	148	148
query84	247	80	87	80
query85	1374	307	299	299
query86	433	298	268	268
query87	4363	4234	4240	4234
query88	3841	2395	2366	2366
query89	421	289	295	289
query90	2060	180	183	180
query91	179	149	151	149
query92	62	50	53	50
query93	2568	566	560	560
query94	956	283	293	283
query95	355	263	260	260
query96	614	273	284	273
query97	3285	3160	3123	3123
query98	233	208	194	194
query99	1486	1302	1294	1294
Total cold run time: 305976 ms
Total hot run time: 197005 ms

@doris-robot

doris-robot commented Jul 15, 2025

Copy link
Copy Markdown
ClickBench: Total hot run time: 29.74 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit fd7e095acfea5f1e7461be6677d18f7d20a823dd, data reload: false
query1	0.03	0.03	0.03
query2	0.06	0.03	0.03
query3	0.23	0.07	0.06
query4	1.63	0.10	0.11
query5	0.52	0.50	0.50
query6	1.13	0.74	0.72
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.56	0.50	0.48
query10	0.55	0.54	0.56
query11	0.14	0.09	0.10
query12	0.14	0.11	0.11
query13	0.61	0.60	0.61
query14	0.76	0.79	0.80
query15	0.84	0.81	0.82
query16	0.39	0.39	0.39
query17	1.04	1.05	1.02
query18	0.23	0.22	0.21
query19	1.83	1.79	1.81
query20	0.01	0.01	0.01
query21	15.45	0.60	0.60
query22	2.57	1.59	2.26
query23	16.93	0.91	0.89
query24	3.34	0.71	0.95
query25	0.27	0.15	0.13
query26	0.37	0.14	0.13
query27	0.04	0.04	0.06
query28	10.61	0.53	0.46
query29	12.59	3.26	3.25
query30	0.24	0.06	0.07
query31	2.84	0.39	0.36
query32	3.25	0.46	0.46
query33	3.00	2.97	2.98
query34	16.82	4.54	4.45
query35	4.58	4.59	4.54
query36	0.68	0.49	0.48
query37	0.09	0.06	0.06
query38	0.04	0.03	0.03
query39	0.03	0.02	0.02
query40	0.16	0.13	0.12
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 104.81 s
Total hot run time: 29.74 s

dataroaring
dataroaring approved these changes Jul 15, 2025
View reviewed changes

@dataroaringdataroaring left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring
dataroaring merged commit 29a008c into apache:branch-3.0Jul 15, 2025
@airborne12
airborne12 deleted the pick_52894_to_origin_branch-3.0 branch July 15, 2025 08:49
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaringdataroaringdataroaring approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@airborne12@hello-stephen@doris-robot@dataroaring