Skip to content

[fix](be) Isolate SNII cache across analyzer generations - #66876

Open
airborne12 wants to merge 1 commit into
apache:masterfrom
airborne12:fix-snii-analyzer-generation-cache
Open

[fix](be) Isolate SNII cache across analyzer generations#66876
airborne12 wants to merge 1 commit into
apache:masterfrom
airborne12:fix-snii-analyzer-generation-cache

Conversation

@airborne12

@airborne12airborne12 commented Aug 18, 2026•
edited
Loading

Copy link
Copy Markdown
Member

What problem does this PR solve?

Issue Number: None

Related PR: #66052

Problem Summary:

SNII keyed its result cache and query single-flight by the raw query before analysis. Plain custom-analyzer segments do not persist an immutable analyzer identity, so two live generations under the same analyzer name could tokenize identical raw bytes differently while reusing the first generation bitmap. Gate pre-analysis sharing with the existing analyzer-context capability. Providers without a segment-fixed identity now analyze independently, while raw pattern queries and contexts with complete immutable CommonGrams identity retain sharing. Existing admission and partial-failure tests now preload the same raw query and prove that providers without complete immutable identity bypass the stale entry.

Release note

Prevent SNII queries from reusing cached results produced by a different custom analyzer generation.

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

    Tests run:

    • InvertedIndexReaderAnalysisPurposeTest.*
    • SniiIndexReaderCountFallback.*
    • 37/37 focused ASAN unit tests passed
    • ./build.sh --be -j 192

  • Behavior changed:

    • No.
    • Yes. SNII bypasses pre-analysis result sharing when analyzer semantics are not fixed by immutable segment identity.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen

hello-stephen commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@airborne12

airborne12 commented Aug 18, 2026

Copy link
Copy Markdown
MemberAuthor

run buildall

@hello-stephen

hello-stephen commented Aug 18, 2026

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 17143 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 379a69a6e3fb42fd8ad8b582181a666015cd07a6, data reload: false
------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL ============================================
q1	17622	3008	2989	2989
q2	q3	10886	870	499	499
q4	4675	258	217	217
q5	7652	595	400	400
q6	145	123	99	99
q7	540	508	404	404
q8	9253	906	958	906
q9	3494	2358	2366	2358
q10	6527	842	709	709
q11	445	254	249	249
q12	690	392	345	345
q13	17868	1559	1159	1159
q14	155	149	143	143
q15	q16	444	396	367	367
q17	773	769	726	726
q18	3131	2313	2290	2290
q19	1117	948	832	832
q20	704	566	459	459
q21	4887	1758	1856	1758
q22	331	271	234	234
Total cold run time: 91339 ms
Total hot run time: 17143 ms
----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL ============================================
q1	3338	3303	3269	3269
q2	q3	2242	2247	2134	2134
q4	1188	1161	872	872
q5	2162	2094	2104	2094
q6	169	118	87	87
q7	1003	938	890	890
q8	1578	1394	1383	1383
q9	3091	3106	3076	3076
q10	1949	1878	1687	1687
q11	364	276	258	258
q12	446	418	346	346
q13	1479	1545	1138	1138
q14	179	171	167	167
q15	q16	399	392	365	365
q17	1023	1021	1011	1011
q18	5011	4392	4765	4392
q19	856	859	847	847
q20	983	969	805	805
q21	3501	3339	3206	3206
q22	419	347	330	330
Total cold run time: 31380 ms
Total hot run time: 28357 ms

@hello-stephen

hello-stephen commented Aug 18, 2026

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 80117 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 379a69a6e3fb42fd8ad8b582181a666015cd07a6, data reload: false
query5	4276	415	325	325
query6	419	163	148	148
query7	4871	476	268	268
query8	300	125	118	118
query9	8700	2892	2897	2892
query10	381	243	239	239
query11	5367	1034	905	905
query12	121	72	71	71
query13	1207	457	349	349
query14	6174	2243	2076	2076
query14_1	1972	1977	1961	1961
query15	178	118	109	109
query16	925	375	364	364
query17	804	465	362	362
query18	2371	327	243	243
query19	190	144	120	120
query20	72	69	69	69
query21	221	117	100	100
query22	5221	5221	5150	5150
query23	6697	6210	6162	6162
query23_1	5983	6168	5884	5884
query24	7337	1139	788	788
query24_1	799	811	808	808
query25	448	318	264	264
query26	1259	270	162	162
query27	2718	439	300	300
query28	4635	1504	1505	1504
query29	941	465	368	368
query30	281	179	151	151
query31	845	442	347	347
query32	105	51	49	49
query33	474	221	187	187
query34	1056	819	481	481
query35	422	406	343	343
query36	555	553	537	537
query37	130	84	72	72
query38	1002	833	794	794
query39	493	493	470	470
query39_1	478	484	463	463
query40	231	127	113	113
query41	63	69	57	57
query42	83	83	80	80
query43	242	243	211	211
query44 query45	107	106	100	100
query46	773	817	516	516
query47	760	766	697	697
query48	307	302	253	253
query49	548	237	218	218
query50	804	334	257	257
query51	7937	7827	8000	7827
query52	70	108	62	62
query53	212	218	158	158
query54	245	194	163	163
query55	81	64	57	57
query56	220	227	208	208
query57	660	648	657	648
query58	223	197	189	189
query59	1174	1220	1077	1077
query60	261	220	195	195
query61	118	113	109	109
query62	366	195	169	169
query63	184	154	154	154
query64	2754	706	590	590
query65 query66	1851	305	262	262
query67	10010	10103	9845	9845
query68 query69	372	223	204	204
query70	601	618	598	598
query71	324	258	230	230
query72	2327	1711	1582	1582
query73	660	588	337	337
query74	1835	1220	1153	1153
query75	1229	1154	1008	1008
query76	2320	702	538	538
query77	252	254	207	207
query78	3943	3632	3207	3207
query79	2871	753	602	602
query80	1623	385	351	351
query81	525	196	175	175
query82	797	131	101	101
query83	333	249	227	227
query84 query85	886	435	381	381
query86	485	175	172	172
query87	994	983	888	888
query88	3074	2118	2150	2118
query89	321	234	210	210
query90	2084	163	149	149
query91	167	152	124	124
query92	60	46	44	44
query93	2421	1185	808	808
query94	674	261	229	229
query95	655	361	431	361
query96	834	580	271	271
query97	1064	1083	1015	1015
query98	188	133	135	133
query99	451	354	320	320
Total cold run time: 174512 ms
Total hot run time: 80117 ms

@airborne12
[fix](be) Isolate SNII cache across analyzer generations
0c7cd3b
### What problem does this PR solve?
Issue Number: None
Related PR: apache#66052
Problem Summary: SNII keyed its result cache and query single-flight by the raw query before analysis. Plain custom-analyzer segments do not persist an immutable analyzer identity, so two live generations under the same analyzer name could tokenize identical raw bytes differently while reusing the first generation bitmap. Gate pre-analysis sharing with the existing analyzer-context capability. Providers without a segment-fixed identity now analyze independently, while raw pattern queries and contexts with complete immutable CommonGrams identity retain sharing. Update the existing admission and partial-failure tests to reproduce the isolation contract by preloading the same raw query and proving providers without complete immutable identity bypass that stale entry.
### Release note
Prevent SNII queries from reusing cached results produced by a different custom analyzer generation.
### Check List (For Author)
- Test: Unit Test
- InvertedIndexReaderAnalysisPurposeTest.*
- SniiIndexReaderCountFallback.*
- 37/37 focused ASAN unit tests passed
- ./build.sh --be -j 192
- Behavior changed: Yes. SNII bypasses pre-analysis result sharing when analyzer semantics are not fixed by immutable segment identity.
- Does this need documentation: No
@airborne12
airborne12force-pushed the fix-snii-analyzer-generation-cache branch from 379a69a to 0c7cd3bCompareAugust 18, 2026 07:33
@airborne12

airborne12 commented Aug 18, 2026

Copy link
Copy Markdown
MemberAuthor

The failed BE UTs were stale expectations exposed by this PR, not a production regression. They asserted that SNII could consult a raw-query cache entry even when the request provider had no complete immutable analyzer identity, which is exactly the cross-generation reuse this PR removes.

The revised test preloads the identical raw query, then verifies that incomplete/no-identity providers bypass that entry, enter the segment, and invoke their own analyzer. Red: the three reported tests failed locally with the old expectations. Green: InvertedIndexReaderAnalysisPurposeTest.*:SniiIndexReaderCountFallback.* passes 37/37 under ASAN, and ./build.sh --be -j 192 completes successfully. Fixed in 0c7cd3b4c7e.

@airborne12

airborne12 commented Aug 18, 2026

Copy link
Copy Markdown
MemberAuthor

run buildall

@hello-stephen

hello-stephen commented Aug 18, 2026

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 17001 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0c7cd3b4c7e700bc9ad9d6a89517de75b5a6ee6a, data reload: false
------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL ============================================
q1	17643	3137	3139	3137
q2	q3	10872	851	488	488
q4	4681	253	205	205
q5	7671	560	397	397
q6	139	117	98	98
q7	525	499	387	387
q8	9250	940	859	859
q9	3445	2380	2382	2380
q10	6521	849	712	712
q11	442	254	242	242
q12	681	403	332	332
q13	17856	1536	1182	1182
q14	168	149	137	137
q15	q16	481	392	370	370
q17	760	796	740	740
q18	3108	2242	2242	2242
q19	1108	856	732	732
q20	720	538	461	461
q21	5253	1670	1844	1670
q22	319	272	230	230
Total cold run time: 91643 ms
Total hot run time: 17001 ms
----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL ============================================
q1	3507	3447	3432	3432
q2	q3	2173	2330	2154	2154
q4	1203	1168	892	892
q5	2194	2129	2125	2125
q6	166	119	90	90
q7	1039	891	885	885
q8	1606	1458	1451	1451
q9	3166	3122	3124	3122
q10	1835	1813	1607	1607
q11	366	273	262	262
q12	446	442	341	341
q13	1501	1548	1180	1180
q14	169	183	162	162
q15	q16	393	395	357	357
q17	1054	1049	1042	1042
q18	4977	4392	4722	4392
q19	852	796	820	796
q20	960	921	796	796
q21	3537	3267	3295	3267
q22	400	348	312	312
Total cold run time: 31544 ms
Total hot run time: 28665 ms

@hello-stephen

hello-stephen commented Aug 18, 2026

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 80629 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0c7cd3b4c7e700bc9ad9d6a89517de75b5a6ee6a, data reload: false
query5	4276	423	346	346
query6	416	165	155	155
query7	4845	462	274	274
query8	300	139	118	118
query9	8676	2912	2881	2881
query10	373	259	214	214
query11	5377	1048	925	925
query12	119	71	72	71
query13	1190	443	321	321
query14	5953	2206	2105	2105
query14_1	2011	1978	1982	1978
query15	176	122	111	111
query16	940	386	382	382
query17	782	439	343	343
query18	2321	323	227	227
query19	169	139	107	107
query20	71	67	69	67
query21	212	115	99	99
query22	5359	5312	5206	5206
query23	6661	6248	5938	5938
query23_1	6231	6020	6131	6020
query24	7330	1064	742	742
query24_1	755	775	782	775
query25	414	291	236	236
query26	1245	267	163	163
query27	2714	426	265	265
query28	4656	1486	1488	1486
query29	907	420	351	351
query30	286	180	153	153
query31	862	436	351	351
query32	96	49	51	49
query33	459	231	184	184
query34	987	840	505	505
query35	419	396	340	340
query36	565	539	510	510
query37	127	81	73	73
query38	998	836	814	814
query39	496	482	495	482
query39_1	449	472	468	468
query40	226	129	116	116
query41	57	56	56	56
query42	80	79	79	79
query43	245	253	217	217
query44 query45	109	112	100	100
query46	826	822	524	524
query47	767	755	735	735
query48	336	317	242	242
query49	549	256	195	195
query50	851	341	258	258
query51	8131	7987	8055	7987
query52	79	75	68	68
query53	211	217	168	168
query54	237	187	192	187
query55	75	66	55	55
query56	251	225	230	225
query57	701	750	651	651
query58	239	203	200	200
query59	1262	1229	1127	1127
query60	260	204	201	201
query61	128	122	112	112
query62	392	207	181	181
query63	191	155	167	155
query64	2748	743	578	578
query65 query66	1950	313	238	238
query67	10146	9867	9882	9867
query68 query69	367	228	200	200
query70	646	635	614	614
query71	292	272	244	244
query72	2404	1809	1667	1667
query73	672	587	359	359
query74	1868	1242	1188	1188
query75	1259	1179	1048	1048
query76	2467	721	538	538
query77	264	256	208	208
query78	3950	3586	3229	3229
query79	3494	726	595	595
query80	1649	427	381	381
query81	541	206	181	181
query82	698	135	112	112
query83	307	249	230	230
query84 query85	876	451	395	395
query86	505	164	164	164
query87	1002	981	904	904
query88	4423	2145	2127	2127
query89	298	223	206	206
query90	2120	151	146	146
query91	156	141	124	124
query92	69	46	43	43
query93	2990	1211	752	752
query94	662	256	222	222
query95	655	433	338	338
query96	853	558	269	269
query97	1034	1057	1027	1027
query98	179	133	132	132
query99	470	337	302	302
Total cold run time: 177856 ms
Total hot run time: 80629 ms

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@eldenmooneldenmoonAwaiting requested review from eldenmooneldenmoon is a code owner
@csun5285csun5285Awaiting requested review from csun5285csun5285 is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@airborne12@hello-stephen