[fix](cloud) preserve rowset inverted index format in recycler by hoshinojyunn · Pull Request #66793 · apache/doris

hoshinojyunn · 2026-08-14T17:01:45Z

What problem does this PR solve?

Issue Number: None

Related PR: #66475

Problem Summary:

This PR separately submits the Meta Service changes from #66475. When the shared schema format differs from the physical format used by an existing rowset, recycler and checker must use the rowset format to select the correct inverted-index files.

The change persists the physical inverted-index format in metadata and makes recycler/checker prefer the rowset value, with the existing schema and schema-KV fallback retained for old metadata.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
  - ./run-cloud-ut.sh --run --filter=recycler_test:RecyclerTest.delete_tmp_rowset_data_with_rowset_idx_v3
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes. Recycler and checker select inverted-index files from the persisted rowset format.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2026-08-14T17:01:52Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

gavinchou · 2026-08-14T17:03:40Z

run buildall

github-actions · 2026-08-14T17:04:24Z

PR approved by at least one committer and no changes requested.

github-actions · 2026-08-14T17:04:26Z

PR approved by anyone and no changes requested.

hello-stephen · 2026-08-14T17:39:02Z

Cloud UT Coverage Report

Increment line coverage 100.00% (21/21) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	78.23% (2001/2558)
Line Coverage	65.35% (36543/55923)
Region Coverage	65.54% (18591/28364)
Branch Coverage	55.20% (10125/18342)

hello-stephen · 2026-08-14T17:49:40Z

TPC-H: Total hot run time: 17830 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 60dc7a2d5b9e98fdec363f8679c3b461588bcaa9, data reload: false
------ Round 1 ----------------------------------
============================================
q1	17571	3133	3173	3133
q2	1908	234	154	154
q3	10454	909	497	497
q4	4667	251	201	201
q5	7680	571	384	384
q6	135	118	93	93
q7	538	508	388	388
q8	9299	897	943	897
q9	3466	2411	2381	2381
q10	6525	869	711	711
q11	447	259	232	232
q12	691	398	335	335
q13	17893	1906	1574	1574
q14	160	154	150	150
q15	q16	444	400	368	368
q17	784	771	749	749
q18	3124	2294	2272	2272
q19	1116	941	851	851
q20	680	511	430	430
q21	5259	1787	1953	1787
q22	329	275	243	243
Total cold run time: 93170 ms
Total hot run time: 17830 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3543	3444	3458	3444
q2	225	208	160	160
q3	2270	2647	2182	2182
q4	1208	1174	902	902
q5	2216	2141	2119	2119
q6	186	123	88	88
q7	1076	948	851	851
q8	1642	1442	1437	1437
q9	3145	3124	3118	3118
q10	1882	1805	1640	1640
q11	362	272	254	254
q12	467	450	349	349
q13	1798	1873	1562	1562
q14	181	179	173	173
q15	q16	402	397	369	369
q17	1072	1042	1056	1042
q18	4979	4422	4810	4422
q19	912	864	858	858
q20	952	940	899	899
q21	3819	3221	3408	3221
q22	401	352	337	337
Total cold run time: 32738 ms
Total hot run time: 29427 ms

hello-stephen · 2026-08-14T17:56:19Z

TPC-DS: Total hot run time: 86088 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 60dc7a2d5b9e98fdec363f8679c3b461588bcaa9, data reload: false
query5	4227	404	334	334
query6	392	169	154	154
query7	4903	458	250	250
query8	291	120	124	120
query9	8683	2942	2941	2941
query10	437	251	217	217
query11	5373	1049	929	929
query12	115	72	68	68
query13	1186	450	312	312
query14	5960	2049	1892	1892
query14_1	1817	1807	1817	1807
query15	173	124	117	117
query16	927	390	364	364
query17	796	488	385	385
query18	2343	331	239	239
query19	175	141	116	116
query20	70	73	70	70
query21	211	134	100	100
query22	5623	5498	5297	5297
query23	7302	6739	6511	6511
query23_1	6684	6506	6508	6506
query24	7258	1096	762	762
query24_1	803	822	782	782
query25	419	285	233	233
query26	1261	261	171	171
query27	2724	440	288	288
query28	4633	1504	1505	1504
query29	965	438	342	342
query30	276	180	158	158
query31	968	679	614	614
query32	126	48	47	47
query33	460	205	171	171
query34	986	806	494	494
query35	408	408	349	349
query36	546	547	490	490
query37	127	85	74	74
query38	1022	857	837	837
query39	533	543	514	514
query39_1	529	531	515	515
query40	219	123	110	110
query41	53	53	50	50
query42	77	77	72	72
query43	256	251	232	232
query44	1034	582	578	578
query45	107	103	97	97
query46	771	865	517	517
query47	953	997	963	963
query48	315	308	230	230
query49	540	243	195	195
query50	836	337	266	266
query51	8470	8359	8273	8273
query52	69	75	63	63
query53	210	211	169	169
query54	241	200	194	194
query55	76	61	56	56
query56	277	247	231	231
query57	670	656	618	618
query58	233	226	210	210
query59	1129	1139	989	989
query60	260	215	193	193
query61	109	109	111	109
query62	368	201	182	182
query63	184	156	149	149
query64	2627	658	558	558
query65	1575	1595	1519	1519
query66	1789	302	228	228
query67	9944	9855	9675	9675
query68	3051	1247	785	785
query69	340	231	202	202
query70	642	577	631	577
query71	287	267	234	234
query72	2337	1900	1725	1725
query73	651	605	358	358
query74	2017	1255	1152	1152
query75	1253	1161	1020	1020
query76	2387	775	558	558
query77	261	311	198	198
query78	5146	4903	4523	4523
query79	1239	902	598	598
query80	1203	382	337	337
query81	480	195	167	167
query82	619	137	113	113
query83	312	258	238	238
query84	302	123	102	102
query85	836	442	383	383
query86	381	168	171	168
query87	1036	977	912	912
query88	2810	2144	2164	2144
query89	315	231	211	211
query90	1846	146	145	145
query91	153	139	121	121
query92	48	46	43	43
query93	1355	1199	791	791
query94	621	286	235	235
query95	615	441	357	357
query96	779	584	272	272
query97	1117	1066	1066	1066
query98	146	137	143	137
query99	414	346	309	309
Total cold run time: 179510 ms
Total hot run time: 86088 ms

hello-stephen · 2026-08-14T18:00:42Z

ClickBench: Total hot run time: 14.85 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 60dc7a2d5b9e98fdec363f8679c3b461588bcaa9, data reload: false
query1	0.00	0.01	0.01
query2	0.08	0.04	0.04
query3	0.25	0.12	0.11
query4	1.60	0.11	0.10
query5	0.16	0.16	0.16
query6	1.22	0.71	0.71
query7	0.03	0.00	0.01
query8	0.05	0.03	0.03
query9	0.30	0.21	0.21
query10	0.36	0.35	0.35
query11	0.16	0.12	0.12
query12	0.14	0.12	0.12
query13	0.31	0.30	0.30
query14	0.47	0.46	0.46
query15	0.36	0.35	0.36
query16	0.23	0.23	0.24
query17	0.69	0.75	0.73
query18	0.19	0.16	0.18
query19	1.18	1.24	1.14
query20	0.02	0.01	0.01
query21	15.43	0.15	0.12
query22	5.05	0.04	0.04
query23	16.16	0.25	0.10
query24	2.68	0.33	0.28
query25	0.12	0.04	0.03
query26	0.78	0.16	0.13
query27	0.04	0.03	0.02
query28	3.69	0.55	0.28
query29	12.46	3.22	2.61
query30	0.25	0.11	0.13
query31	2.75	0.36	0.18
query32	3.52	0.34	0.22
query33	1.46	1.56	1.40
query34	15.35	2.21	1.78
query35	1.73	1.70	1.76
query36	0.47	0.30	0.28
query37	0.06	0.04	0.04
query38	0.04	0.03	0.03
query39	0.04	0.03	0.03
query40	0.11	0.08	0.07
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 90.14 s
Total hot run time: 14.85 s

### What problem does this PR solve? Issue Number: None Related PR: apache#66475 Problem Summary: This change separately submits the Meta Service portion of apache#66475. During a partition-level inverted-index format rollout, a rowset can use a format that differs from its shared schema metadata. The recycler and checker previously selected index paths from shared schema metadata or its cache, which could target V1 paths for a rowset whose files use V3. Persist the actual inverted-index storage format in rowset and tablet metadata. Make the recycler and checker prefer the rowset format, while retaining schema and schema-KV fallback for metadata written before the field existed. Add a Cloud unit test where the shared schema is V1 and the rowset is V3, verifying that recycler deletes the V3 index file. ### Release note None ### Check List (For Author) - Test: Unit Test - `./run-cloud-ut.sh --run --filter=recycler_test:RecyclerTest.delete_tmp_rowset_data_with_rowset_idx_v3` - Behavior changed: Yes. Meta Service recycler and checker use each rowset\047s persisted inverted-index format when selecting index files. - Does this need documentation: No

hoshinojyunn · 2026-08-15T02:40:30Z

run buildall

hello-stephen · 2026-08-15T03:54:57Z

Cloud UT Coverage Report

Increment line coverage 100.00% (21/21) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	78.23% (2001/2558)
Line Coverage	65.30% (36518/55923)
Region Coverage	65.47% (18569/28364)
Branch Coverage	55.10% (10107/18342)

hello-stephen · 2026-08-15T05:30:00Z

FE UT Coverage Report

Increment line coverage `` 🎉
Increment coverage report
Complete coverage report

airborne12

LGTM

github-actions · 2026-08-15T06:02:24Z

PR approved by at least one committer and no changes requested.

hello-stephen · 2026-08-15T06:55:03Z

BE Regression && UT Coverage Report

Increment line coverage 100.00% (24/24) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	74.72% (32023/42857)
Line Coverage	59.16% (355068/600190)
Region Coverage	55.40% (295700/533791)
Branch Coverage	56.26% (133203/236763)

hello-stephen · 2026-08-15T07:08:02Z

FE Regression Coverage Report

Increment line coverage 100% (0/0) 🎉
Increment coverage report
Complete coverage report

airborne12

LGTM

hello-stephen · 2026-08-15T15:29:28Z

TPC-H: Total hot run time: 17787 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fbd7a28fa82bc91d35da407f69807c1d6e4e0eed, data reload: false
------ Round 1 ----------------------------------
============================================
q1	17572	3199	3161	3161
q2	1933	239	151	151
q3	10435	918	517	517
q4	4679	254	204	204
q5	7670	576	395	395
q6	144	117	94	94
q7	539	508	386	386
q8	9238	934	915	915
q9	3456	2392	2395	2392
q10	6554	850	727	727
q11	444	261	234	234
q12	678	395	333	333
q13	17881	1878	1565	1565
q14	169	155	145	145
q15	q16	433	398	365	365
q17	835	793	758	758
q18	3113	2331	2240	2240
q19	1107	938	796	796
q20	675	526	459	459
q21	5372	1715	1925	1715
q22	342	272	235	235
Total cold run time: 93269 ms
Total hot run time: 17787 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3544	3472	3467	3467
q2	209	226	160	160
q3	2215	2385	2214	2214
q4	1211	1180	905	905
q5	2196	2150	2117	2117
q6	182	125	89	89
q7	1061	900	844	844
q8	1635	1422	1422	1422
q9	3151	3118	3097	3097
q10	1877	1820	1650	1650
q11	358	273	247	247
q12	462	447	352	352
q13	1855	1858	1567	1567
q14	166	171	160	160
q15	q16	398	402	353	353
q17	1061	1051	1052	1051
q18	4987	4467	4785	4467
q19	858	853	890	853
q20	980	933	831	831
q21	3935	3225	3431	3225
q22	393	355	337	337
Total cold run time: 32734 ms
Total hot run time: 29408 ms

hello-stephen · 2026-08-15T15:36:07Z

TPC-DS: Total hot run time: 86850 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fbd7a28fa82bc91d35da407f69807c1d6e4e0eed, data reload: false
query5	4249	427	359	359
query6	406	168	144	144
query7	4839	461	262	262
query8	306	125	116	116
query9	8673	2934	2971	2934
query10	384	254	212	212
query11	5385	1063	940	940
query12	126	73	77	73
query13	1198	461	327	327
query14	5958	2106	1913	1913
query14_1	1847	1820	1817	1817
query15	170	123	132	123
query16	921	415	392	392
query17	1046	481	375	375
query18	2343	320	239	239
query19	173	144	118	118
query20	86	70	70	70
query21	208	117	109	109
query22	5409	5411	5257	5257
query23	7341	6821	6688	6688
query23_1	6785	6776	6720	6720
query24	7256	1098	779	779
query24_1	781	787	795	787
query25	416	288	235	235
query26	1261	271	164	164
query27	2715	455	287	287
query28	4510	1503	1530	1503
query29	921	446	338	338
query30	270	169	150	150
query31	948	665	611	611
query32	104	50	48	48
query33	455	207	170	170
query34	996	847	477	477
query35	411	439	342	342
query36	558	563	513	513
query37	122	81	72	72
query38	998	864	817	817
query39	518	540	545	540
query39_1	529	504	533	504
query40	219	126	115	115
query41	52	53	52	52
query42	76	72	73	72
query43	251	251	222	222
query44	1032	595	583	583
query45	114	99	99	99
query46	809	869	560	560
query47	973	997	957	957
query48	308	310	239	239
query49	538	247	199	199
query50	817	327	261	261
query51	8384	8427	8475	8427
query52	75	71	60	60
query53	217	210	164	164
query54	249	189	176	176
query55	75	56	56	56
query56	250	235	218	218
query57	686	636	617	617
query58	234	214	213	213
query59	1141	1121	1058	1058
query60	297	225	204	204
query61	129	132	131	131
query62	348	216	181	181
query63	185	160	157	157
query64	2693	776	574	574
query65	1557	1571	1524	1524
query66	1779	302	229	229
query67	10111	9797	9772	9772
query68	2887	1282	774	774
query69	327	221	194	194
query70	683	622	613	613
query71	300	249	233	233
query72	2326	1761	1672	1672
query73	669	584	341	341
query74	1844	1247	1131	1131
query75	1229	1154	1000	1000
query76	2291	746	571	571
query77	254	264	220	220
query78	5195	4848	4533	4533
query79	1277	855	593	593
query80	1198	379	347	347
query81	486	199	171	171
query82	617	136	108	108
query83	314	259	238	238
query84	251	127	99	99
query85	847	447	378	378
query86	394	178	167	167
query87	1035	978	888	888
query88	2836	2151	2139	2139
query89	309	233	206	206
query90	1875	140	144	140
query91	152	143	129	129
query92	49	46	44	44
query93	1410	1137	787	787
query94	627	246	230	230
query95	619	458	351	351
query96	850	572	281	281
query97	1063	1050	1031	1031
query98	152	137	136	136
query99	418	353	312	312
Total cold run time: 179444 ms
Total hot run time: 86850 ms

hello-stephen · 2026-08-15T15:40:29Z

ClickBench: Total hot run time: 14.78 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit fbd7a28fa82bc91d35da407f69807c1d6e4e0eed, data reload: false
query1	0.00	0.01	0.01
query2	0.08	0.04	0.04
query3	0.24	0.11	0.11
query4	1.60	0.09	0.09
query5	0.17	0.16	0.15
query6	1.30	0.66	0.67
query7	0.04	0.01	0.01
query8	0.04	0.03	0.03
query9	0.29	0.22	0.22
query10	0.35	0.35	0.34
query11	0.16	0.11	0.11
query12	0.15	0.12	0.12
query13	0.31	0.31	0.30
query14	0.47	0.45	0.47
query15	0.37	0.34	0.36
query16	0.22	0.22	0.24
query17	0.69	0.69	0.72
query18	0.19	0.17	0.18
query19	1.20	1.21	1.16
query20	0.01	0.01	0.01
query21	15.47	0.15	0.11
query22	5.09	0.04	0.04
query23	16.18	0.25	0.10
query24	2.97	0.32	0.26
query25	0.10	0.05	0.04
query26	0.77	0.16	0.13
query27	0.03	0.04	0.03
query28	3.63	0.55	0.27
query29	12.43	3.17	2.59
query30	0.26	0.12	0.12
query31	2.77	0.37	0.18
query32	3.52	0.32	0.24
query33	1.40	1.43	1.43
query34	15.33	2.24	1.79
query35	1.77	1.75	1.72
query36	0.46	0.29	0.28
query37	0.06	0.04	0.03
query38	0.04	0.03	0.02
query39	0.03	0.02	0.02
query40	0.11	0.08	0.08
query41	0.07	0.02	0.03
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 90.43 s
Total hot run time: 14.78 s

hoshinojyunn requested review from gavinchou, liaoxin01, luwei16 and yiguolei as code owners August 14, 2026 17:01

gavinchou previously approved these changes Aug 14, 2026
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Aug 14, 2026

github-actionsBot added the reviewed label Aug 14, 2026

gavinchou added dev/4.0.x dev/4.1.x labels Aug 14, 2026

hoshinojyunn dismissed gavinchou’s stale review via fbd7a28August 15, 2026 02:38

hoshinojyunn force-pushed the fix/ms branch from 60dc7a2 to fbd7a28Compare August 15, 2026 02:38

github-actionsBot removed the approved Indicates a PR has been approved by one committer. label Aug 15, 2026

airborne12 approved these changes Aug 15, 2026
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Aug 15, 2026

airborne12 approved these changes Aug 15, 2026
View reviewed changes

yiguolei approved these changes Aug 17, 2026
View reviewed changes

gavinchou approved these changes Aug 18, 2026
View reviewed changes

gavinchou merged commit a303308 into apache:masterAug 18, 2026
35 of 36 checks passed

github-actionsBot added the dev/4.1.x-conflict label Aug 18, 2026

Uh oh!

Conversation

hoshinojyunn commented Aug 14, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Aug 14, 2026

Uh oh!

gavinchou commented Aug 14, 2026

Uh oh!

github-actionsBot commented Aug 14, 2026

Uh oh!

github-actionsBot commented Aug 14, 2026

Uh oh!

hello-stephen commented Aug 14, 2026

Cloud UT Coverage Report

Uh oh!

hello-stephen commented Aug 14, 2026

Uh oh!

hello-stephen commented Aug 14, 2026

Uh oh!

hello-stephen commented Aug 14, 2026

Uh oh!

hoshinojyunn commented Aug 15, 2026

Uh oh!

hello-stephen commented Aug 15, 2026

Cloud UT Coverage Report

Uh oh!

hello-stephen commented Aug 15, 2026

FE UT Coverage Report

Uh oh!

airborne12 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Aug 15, 2026

Uh oh!

hello-stephen commented Aug 15, 2026

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Aug 15, 2026

FE Regression Coverage Report

Uh oh!

airborne12 left a comment

Choose a reason for hiding this comment

Uh oh!

hello-stephen commented Aug 15, 2026

Uh oh!

hello-stephen commented Aug 15, 2026

Uh oh!

hello-stephen commented Aug 15, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

hoshinojyunn commented Aug 14, 2026•
edited
Loading