[fix](inverted index) implement phrase-level BM25 scoring with phrase frequency by zzzxl1993 · Pull Request #60331 · apache/doris

zzzxl1993 · 2026-01-29T03:35:09Z

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2026-01-29T03:35:13Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

zzzxl1993 · 2026-01-29T03:35:31Z

run buildall

doris-robot · 2026-01-29T04:02:36Z

TPC-H: Total hot run time: 31749 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 929c34acb8242e728695bf53425784ea5d6d0f90, data reload: false
------ Round 1 ----------------------------------
q1	17627	5311	5081	5081
q2	1986	309	198	198
q3	10245	1400	746	746
q4	10186	817	314	314
q5	7494	2190	1849	1849
q6	197	181	148	148
q7	869	730	601	601
q8	9261	1447	1122	1122
q9	5183	4857	4879	4857
q10	6803	1951	1584	1584
q11	500	284	285	284
q12	351	376	224	224
q13	17828	4079	3230	3230
q14	233	231	209	209
q15	924	841	808	808
q16	706	673	624	624
q17	644	804	451	451
q18	6751	6579	6300	6300
q19	1242	1006	627	627
q20	384	339	230	230
q21	2606	2072	1985	1985
q22	349	315	277	277
Total cold run time: 102369 ms
Total hot run time: 31749 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5319	5346	5322	5322
q2	253	328	254	254
q3	2179	2676	2254	2254
q4	1335	1748	1275	1275
q5	4336	4250	4254	4250
q6	207	179	141	141
q7	2194	2187	1847	1847
q8	2587	2471	2463	2463
q9	7678	7461	7543	7461
q10	2937	3006	2574	2574
q11	543	482	447	447
q12	667	755	625	625
q13	3918	4795	3421	3421
q14	306	315	301	301
q15	902	822	812	812
q16	709	753	686	686
q17	1202	1324	1392	1324
q18	8245	8128	7907	7907
q19	864	828	853	828
q20	2103	2194	2049	2049
q21	4608	4232	4162	4162
q22	589	546	494	494
Total cold run time: 53681 ms
Total hot run time: 50897 ms

doris-robot · 2026-01-29T04:19:14Z

ClickBench: Total hot run time: 28.79 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 929c34acb8242e728695bf53425784ea5d6d0f90, data reload: false
query1	0.05	0.04	0.04
query2	0.10	0.05	0.04
query3	0.25	0.08	0.09
query4	1.61	0.11	0.11
query5	0.27	0.25	0.26
query6	1.16	0.68	0.68
query7	0.03	0.03	0.02
query8	0.05	0.04	0.04
query9	0.56	0.50	0.50
query10	0.56	0.55	0.55
query11	0.14	0.09	0.10
query12	0.15	0.11	0.11
query13	0.63	0.61	0.62
query14	1.08	1.08	1.05
query15	0.87	0.86	0.87
query16	0.39	0.38	0.39
query17	1.10	1.13	1.09
query18	0.22	0.21	0.21
query19	1.99	1.98	2.01
query20	0.02	0.01	0.02
query21	15.40	0.28	0.14
query22	5.23	0.06	0.04
query23	16.06	0.27	0.10
query24	1.23	0.82	0.92
query25	0.12	0.06	0.07
query26	0.14	0.13	0.13
query27	0.09	0.06	0.04
query28	5.00	1.16	0.96
query29	12.57	3.88	3.13
query30	0.29	0.13	0.12
query31	2.80	0.64	0.41
query32	3.23	0.59	0.50
query33	3.22	3.29	3.31
query34	16.25	5.41	4.72
query35	4.80	4.80	4.77
query36	0.65	0.50	0.50
query37	0.11	0.06	0.06
query38	0.07	0.04	0.03
query39	0.04	0.02	0.03
query40	0.19	0.16	0.16
query41	0.09	0.03	0.03
query42	0.04	0.03	0.03
query43	0.05	0.04	0.03
Total cold run time: 98.9 s
Total hot run time: 28.79 s

hello-stephen · 2026-01-29T05:05:29Z

BE UT Coverage Report

Increment line coverage 67.69% (44/65) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	52.73% (19267/36536)
Line Coverage	36.11% (178989/495668)
Region Coverage	32.56% (138837/426350)
Branch Coverage	33.49% (60053/179323)

hello-stephen · 2026-01-29T06:56:42Z

BE Regression && UT Coverage Report

Increment line coverage 90.77% (59/65) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	73.25% (26233/35814)
Line Coverage	56.33% (278525/494435)
Region Coverage	54.25% (233652/430668)
Branch Coverage	55.81% (100473/180039)

hello-stephen · 2026-02-04T08:54:10Z

BE Regression && UT Coverage Report

Increment line coverage 90.77% (59/65) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	73.23% (26227/35814)
Line Coverage	56.31% (278409/494435)
Region Coverage	54.24% (233588/430668)
Branch Coverage	55.79% (100436/180039)

zzzxl1993 · 2026-02-25T02:54:39Z

run buildall

… frequency

doris-robot · 2026-02-25T03:40:03Z

TPC-H: Total hot run time: 28707 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1e81f01b376b04c5c6e7f4ddadfa85dc0d40693d, data reload: false
------ Round 1 ----------------------------------
============================================
q1	17617	4541	4271	4271
q2	q3	10647	814	521	521
q4	4680	366	251	251
q5	7548	1213	1032	1032
q6	173	176	146	146
q7	777	870	687	687
q8	9296	1461	1335	1335
q9	4927	5081	4693	4693
q10	6878	1884	1636	1636
q11	498	256	242	242
q12	694	564	480	480
q13	17803	4187	3452	3452
q14	230	226	206	206
q15	942	806	795	795
q16	758	723	669	669
q17	708	886	441	441
q18	5938	5364	5214	5214
q19	1101	984	608	608
q20	520	498	386	386
q21	4376	1838	1397	1397
q22	343	286	245	245
Total cold run time: 96454 ms
Total hot run time: 28707 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4420	4338	4338	4338
q2	q3	1765	2176	1742	1742
q4	858	1162	777	777
q5	4011	4353	4332	4332
q6	181	176	144	144
q7	1717	1623	1478	1478
q8	2461	2672	2519	2519
q9	7574	7471	7384	7384
q10	2727	2907	2380	2380
q11	503	429	411	411
q12	488	582	444	444
q13	3932	4437	3655	3655
q14	286	311	277	277
q15	886	802	815	802
q16	704	802	735	735
q17	1194	1554	1359	1359
q18	7248	6776	6581	6581
q19	897	899	888	888
q20	2181	2188	2021	2021
q21	4072	3491	3389	3389
q22	468	467	404	404
Total cold run time: 48573 ms
Total hot run time: 46060 ms

doris-robot · 2026-02-25T03:51:25Z

TPC-DS: Total hot run time: 184213 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1e81f01b376b04c5c6e7f4ddadfa85dc0d40693d, data reload: false
query5	4946	620	499	499
query6	328	215	221	215
query7	4217	461	268	268
query8	333	242	236	236
query9	8698	2778	2748	2748
query10	557	422	350	350
query11	16997	17525	17219	17219
query12	202	132	136	132
query13	1304	462	346	346
query14	7328	3329	3110	3110
query14_1	2895	2915	2939	2915
query15	235	207	180	180
query16	1047	496	458	458
query17	1088	741	638	638
query18	3027	477	348	348
query19	214	198	188	188
query20	147	154	130	130
query21	214	136	127	127
query22	5331	5064	4914	4914
query23	17216	16824	16606	16606
query23_1	16610	16660	16658	16658
query24	7167	1600	1218	1218
query24_1	1216	1231	1221	1221
query25	542	466	398	398
query26	1229	252	151	151
query27	2786	470	289	289
query28	4526	1883	1871	1871
query29	781	551	481	481
query30	322	233	206	206
query31	894	719	649	649
query32	79	73	71	71
query33	513	333	279	279
query34	918	903	573	573
query35	663	668	596	596
query36	1107	1146	977	977
query37	129	95	85	85
query38	2972	2897	2890	2890
query39	874	861	851	851
query39_1	868	813	821	813
query40	224	152	140	140
query41	62	63	59	59
query42	104	101	105	101
query43	375	383	374	374
query44 query45	204	191	180	180
query46	883	980	610	610
query47	2139	2132	2048	2048
query48	311	324	251	251
query49	646	472	398	398
query50	676	287	225	225
query51	4141	4075	4046	4046
query52	117	116	101	101
query53	294	338	298	298
query54	319	285	274	274
query55	94	85	80	80
query56	328	356	339	339
query57	1369	1339	1278	1278
query58	302	290	293	290
query59	2647	2688	2582	2582
query60	356	356	346	346
query61	171	170	170	170
query62	633	606	543	543
query63	327	282	277	277
query64	4963	1345	1091	1091
query65 query66	1398	454	372	372
query67	16392	16473	16340	16340
query68 query69	438	317	291	291
query70	980	979	943	943
query71	343	320	307	307
query72	2904	2618	2355	2355
query73	544	539	324	324
query74	10029	9904	9790	9790
query75	2842	2738	2469	2469
query76	2319	1036	676	676
query77	359	390	316	316
query78	11272	11408	10663	10663
query79	1361	792	616	616
query80	1355	604	543	543
query81	559	293	254	254
query82	1005	145	117	117
query83	348	261	256	256
query84	247	118	91	91
query85	892	471	422	422
query86	411	304	263	263
query87	3089	3111	3004	3004
query88	3587	2677	2660	2660
query89	430	365	349	349
query90	1941	172	172	172
query91	161	159	135	135
query92	77	78	69	69
query93	996	815	498	498
query94	638	309	314	309
query95	573	359	306	306
query96	646	518	224	224
query97	2494	2516	2383	2383
query98	232	221	218	218
query99	994	1001	898	898
Total cold run time: 255570 ms
Total hot run time: 184213 ms

zzzxl1993 · 2026-02-25T09:51:50Z

run buildall

doris-robot · 2026-02-25T10:28:35Z

TPC-H: Total hot run time: 28765 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2aa0761e59ad1f2943c8abf4d78616dd450776da, data reload: false
------ Round 1 ----------------------------------
============================================
q1	17651	4549	4292	4292
q2	q3	10645	783	527	527
q4	4677	354	247	247
q5	7555	1200	1003	1003
q6	172	176	145	145
q7	800	840	670	670
q8	9299	1468	1330	1330
q9	4857	4748	4716	4716
q10	6806	1865	1648	1648
q11	464	250	247	247
q12	690	559	466	466
q13	17778	4189	3407	3407
q14	225	232	215	215
q15	955	797	785	785
q16	746	713	674	674
q17	708	835	447	447
q18	5900	5480	5169	5169
q19	1130	978	600	600
q20	530	494	390	390
q21	5015	2218	1503	1503
q22	367	329	284	284
Total cold run time: 96970 ms
Total hot run time: 28765 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4703	4614	4597	4597
q2	q3	1823	2201	1808	1808
q4	883	1291	764	764
q5	4045	4370	4334	4334
q6	186	170	140	140
q7	1797	1661	1556	1556
q8	2524	2695	2576	2576
q9	7408	7340	7349	7340
q10	2677	2880	2409	2409
q11	495	425	402	402
q12	492	586	448	448
q13	4138	4481	3614	3614
q14	274	303	285	285
q15	871	813	812	812
q16	733	791	720	720
q17	1217	1624	1308	1308
q18	7062	6894	6641	6641
q19	899	835	863	835
q20	2124	2178	2091	2091
q21	3953	3463	3354	3354
q22	494	425	376	376
Total cold run time: 48798 ms
Total hot run time: 46410 ms

doris-robot · 2026-02-25T10:39:56Z

TPC-DS: Total hot run time: 183744 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2aa0761e59ad1f2943c8abf4d78616dd450776da, data reload: false
query5	4772	625	514	514
query6	335	225	214	214
query7	4217	501	292	292
query8	345	252	253	252
query9	8774	2778	2753	2753
query10	540	376	331	331
query11	17016	17454	17150	17150
query12	204	126	124	124
query13	1480	481	365	365
query14	6889	3323	3196	3196
query14_1	2925	2890	2868	2868
query15	238	214	186	186
query16	987	524	463	463
query17	1064	745	598	598
query18	4140	457	351	351
query19	209	212	188	188
query20	137	129	129	129
query21	221	141	119	119
query22	5522	5048	4926	4926
query23	17313	16819	16724	16724
query23_1	16803	16707	16695	16695
query24	7295	1624	1224	1224
query24_1	1235	1253	1231	1231
query25	577	479	408	408
query26	1245	259	153	153
query27	2760	469	297	297
query28	4499	1888	1894	1888
query29	820	577	481	481
query30	309	250	212	212
query31	886	748	641	641
query32	86	75	73	73
query33	530	350	296	296
query34	918	905	560	560
query35	644	665	594	594
query36	1075	1142	944	944
query37	137	98	83	83
query38	2968	2890	2875	2875
query39	896	882	837	837
query39_1	837	832	818	818
query40	246	155	140	140
query41	67	68	65	65
query42	108	104	103	103
query43	376	393	364	364
query44 query45	199	195	188	188
query46	888	1009	605	605
query47	2139	2141	2029	2029
query48	327	320	239	239
query49	640	479	400	400
query50	688	279	218	218
query51	4086	3998	4032	3998
query52	105	107	95	95
query53	292	345	284	284
query54	310	279	316	279
query55	90	78	81	78
query56	300	289	316	289
query57	1380	1319	1295	1295
query58	280	278	270	270
query59	2535	2661	2521	2521
query60	338	336	322	322
query61	147	145	144	144
query62	645	603	540	540
query63	311	273	276	273
query64	4849	1250	986	986
query65 query66	1401	462	349	349
query67	16352	16345	16328	16328
query68 query69	393	302	275	275
query70	1004	902	975	902
query71	330	304	286	286
query72	2825	2637	2380	2380
query73	544	556	313	313
query74	9995	9914	9717	9717
query75	2827	2747	2478	2478
query76	2292	1042	651	651
query77	346	370	305	305
query78	11184	11416	10696	10696
query79	1238	797	603	603
query80	1363	618	508	508
query81	561	284	244	244
query82	979	151	114	114
query83	338	258	236	236
query84	253	114	90	90
query85	857	464	423	423
query86	420	305	306	305
query87	3152	3087	3005	3005
query88	3558	2657	2637	2637
query89	431	377	334	334
query90	2023	172	172	172
query91	162	163	128	128
query92	73	72	73	72
query93	990	813	503	503
query94	633	314	303	303
query95	586	391	319	319
query96	637	516	229	229
query97	2486	2503	2388	2388
query98	235	220	219	219
query99	1008	992	920	920
Total cold run time: 256142 ms
Total hot run time: 183744 ms

doris-robot · 2026-02-25T12:08:11Z

BE UT Coverage Report

Increment line coverage 68.75% (44/64) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	52.52% (19592/37301)
Line Coverage	36.13% (182639/505533)
Region Coverage	32.47% (141768/436549)
Branch Coverage	33.42% (61436/183816)

hello-stephen · 2026-02-25T13:32:14Z

BE Regression && UT Coverage Report

Increment line coverage 90.62% (58/64) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	73.16% (26728/36534)
Line Coverage	56.45% (284530/504040)
Region Coverage	53.87% (237413/440725)
Branch Coverage	55.59% (102513/184410)

hoshinojyunn

LGTM

github-actions · 2026-02-26T02:45:38Z

PR approved by anyone and no changes requested.

eldenmoon

LGTM

… frequency (#60331)

… with phrase frequency #60331 (#61243) Cherry-picked from #60331 Co-authored-by: zzzxl <yangsiyu@selectdb.com>

zzzxl1993 force-pushed the 202501291132 branch from 929c34a to 1e81f01Compare February 25, 2026 02:54

[fix](inverted index) implement phrase-level BM25 scoring with phrase…
1b293f6
… frequency

zzzxl1993 force-pushed the 202501291132 branch from 1e81f01 to 1b293f6Compare February 25, 2026 03:12

fix
2aa0761

hoshinojyunn approved these changes Feb 26, 2026
View reviewed changes

github-actionsBot added the reviewed label Feb 26, 2026

eldenmoon approved these changes Mar 2, 2026
View reviewed changes

zzzxl1993 merged commit 0eaac8c into apache:masterMar 2, 2026
29 of 30 checks passed

airborne12 added the dev/4.0.x label Mar 12, 2026

github-actionsBot pushed a commit that referenced this pull request Mar 12, 2026
[fix](inverted index) implement phrase-level BM25 scoring with phrase…
c1e2650
… frequency (#60331)

github-actionsBot mentioned this pull request Mar 12, 2026
branch-4.0: [fix](inverted index) implement phrase-level BM25 scoring with phrase frequency #60331 #61243
Merged

yiguolei pushed a commit that referenced this pull request Mar 12, 2026
branch-4.0: [fix](inverted index) implement phrase-level BM25 scoring…
8d0ba18
… with phrase frequency #60331 (#61243) Cherry-picked from #60331 Co-authored-by: zzzxl <yangsiyu@selectdb.com>

yiguolei added dev/4.0.5-merged and removed dev/4.0.x labels Mar 12, 2026

yiguolei added the dev/4.1.0-merged label Mar 12, 2026

yiguolei mentioned this pull request Mar 27, 2026
4.0.5 Release Notes #61817
Closed

Uh oh!

Conversation

zzzxl1993 commented Jan 29, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Jan 29, 2026

Uh oh!

zzzxl1993 commented Jan 29, 2026

Uh oh!

doris-robot commented Jan 29, 2026

Uh oh!

doris-robot commented Jan 29, 2026

Uh oh!

hello-stephen commented Jan 29, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Jan 29, 2026

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Feb 4, 2026

BE Regression && UT Coverage Report

Uh oh!

zzzxl1993 commented Feb 25, 2026

Uh oh!

doris-robot commented Feb 25, 2026

Uh oh!

doris-robot commented Feb 25, 2026

Uh oh!

zzzxl1993 commented Feb 25, 2026

Uh oh!

doris-robot commented Feb 25, 2026

Uh oh!

doris-robot commented Feb 25, 2026

Uh oh!

doris-robot commented Feb 25, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Feb 25, 2026

BE Regression && UT Coverage Report

Uh oh!

hoshinojyunn left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Feb 26, 2026

Uh oh!

eldenmoon left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

zzzxl1993 commented Jan 29, 2026•
edited
Loading