[fix](meta cache) Invalidate external row count cache after metadata changes by foxtail463 · Pull Request #64160 · apache/doris

foxtail463 · 2026-06-05T12:48:03Z

Problem Summary

External table row counts are cached separately from metadata caches. Some metadata mutation and replay paths did not invalidate row counts, allowing stale values to surviverefreshes,
DML/DDL operations, cold-cache replay, and table renames until cache expiration, which could affect statistics estimation.

Solution

Add scoped invalidation and concurrent-load fencing to ExternalRowCountCache, and integrate it with unified metadata invalidation paths. Cover external metadata mutations and replay,
including cold deterministic identities and both sides of a rename. Add real-cache tests to verify eviction and reload behavior.

hello-stephen · 2026-06-05T12:48:08Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

foxtail463 · 2026-06-05T13:38:10Z

run buildall

hello-stephen · 2026-06-05T14:46:15Z

TPC-H: Total hot run time: 29357 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a3e55294493715e30bdd5c23fbffbc33690b33e3, data reload: false
------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL ============================================
q1	17613	4156	4046	4046
q2	q3	10765	1385	829	829
q4	4681	505	359	359
q5	7562	887	598	598
q6	191	178	141	141
q7	777	856	638	638
q8	9345	1612	1581	1581
q9	5951	4547	4571	4547
q10	6770	1802	1523	1523
q11	438	277	244	244
q12	629	426	299	299
q13	18100	3410	2789	2789
q14	266	259	250	250
q15	q16	836	783	725	725
q17	980	1024	920	920
q18	6846	5713	5555	5555
q19	1173	1273	1154	1154
q20	506	414	264	264
q21	6054	2768	2574	2574
q22	467	372	321	321
Total cold run time: 99950 ms
Total hot run time: 29357 ms
----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL ============================================
q1	5367	5021	4889	4889
q2	q3	4946	5270	4708	4708
q4	2155	2291	1420	1420
q5	4917	4747	4785	4747
q6	242	179	132	132
q7	1898	1722	1575	1575
q8	2531	2213	2196	2196
q9	8043	7957	7466	7466
q10	4777	4685	4216	4216
q11	576	402	380	380
q12	734	756	535	535
q13	2976	3422	2786	2786
q14	281	290	252	252
q15	q16	687	716	616	616
q17	1341	1297	1287	1287
q18	7725	6782	6916	6782
q19	1103	1115	1118	1115
q20	2226	2218	1974	1974
q21	5381	4718	4562	4562
q22	549	470	427	427
Total cold run time: 58455 ms
Total hot run time: 52065 ms

hello-stephen · 2026-06-05T14:57:09Z

TPC-DS: Total hot run time: 169748 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a3e55294493715e30bdd5c23fbffbc33690b33e3, data reload: false
query5	4332	637	488	488
query6	468	201	188	188
query7	4869	577	291	291
query8	367	218	201	201
query9	8779	4063	4068	4063
query10	480	318	256	256
query11	5950	2346	2187	2187
query12	162	106	103	103
query13	1275	609	421	421
query14	6379	5455	5113	5113
query14_1	4467	4448	4473	4448
query15	206	199	175	175
query16	1049	464	471	464
query17	1133	716	610	610
query18	2727	487	365	365
query19	213	200	162	162
query20	116	108	113	108
query21	224	147	124	124
query22	13664	13604	13523	13523
query23	17373	16425	16221	16221
query23_1	16259	16340	16379	16340
query24	7502	1802	1321	1321
query24_1	1352	1292	1321	1292
query25	580	472	410	410
query26	1312	318	161	161
query27	2624	575	344	344
query28	4414	2025	2006	2006
query29	1095	634	504	504
query30	319	242	207	207
query31	1131	1082	962	962
query32	100	65	60	60
query33	520	367	267	267
query34	1168	1136	688	688
query35	743	785	674	674
query36	1396	1431	1192	1192
query37	165	109	90	90
query38	3214	3169	3038	3038
query39	937	923	895	895
query39_1	895	853	864	853
query40	226	125	101	101
query41	64	62	63	62
query42	98	95	94	94
query43	338	330	290	290
query44 query45	204	189	179	179
query46	1108	1202	743	743
query47	2315	2364	2252	2252
query48	396	425	304	304
query49	632	470	367	367
query50	945	355	258	258
query51	4342	4295	4189	4189
query52	88	90	79	79
query53	241	272	193	193
query54	278	217	206	206
query55	80	76	72	72
query56	247	231	221	221
query57	1418	1392	1297	1297
query58	249	217	215	215
query59	1588	1662	1421	1421
query60	296	249	240	240
query61	165	160	161	160
query62	690	660	598	598
query63	228	191	190	190
query64	2548	778	656	656
query65 query66	1712	460	346	346
query67	29842	29089	29554	29089
query68 query69	429	327	270	270
query70	962	955	977	955
query71	337	232	216	216
query72	2967	2840	2406	2406
query73	820	768	424	424
query74	5127	4950	4773	4773
query75	2655	2580	2230	2230
query76	2314	1194	786	786
query77	362	382	284	284
query78	12383	12503	11917	11917
query79	1410	1104	744	744
query80	583	485	389	389
query81	456	287	239	239
query82	586	167	124	124
query83	353	273	257	257
query84	260	140	114	114
query85	863	557	444	444
query86	361	299	285	285
query87	3453	3409	3211	3211
query88	3635	2780	2743	2743
query89	431	372	324	324
query90	1969	191	186	186
query91	179	171	136	136
query92	65	60	57	57
query93	1481	1517	933	933
query94	547	368	299	299
query95	687	387	446	387
query96	1060	791	349	349
query97	2676	2699	2549	2549
query98	206	208	206	206
query99	1155	1191	1047	1047
Total cold run time: 251425 ms
Total hot run time: 169748 ms

hello-stephen · 2026-06-05T15:09:01Z

FE UT Coverage Report

Increment line coverage 48.98% (24/49) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-06-05T16:12:29Z

FE Regression Coverage Report

Increment line coverage 6.91% (17/246) 🎉
Increment coverage report
Complete coverage report

morrySnow · 2026-06-08T09:09:21Z

 cacheRegistry.resetForTest(caches);
 }

+ void replaceRowCountCacheForTest(ExternalRowCountCache rowCountCache) {


change name to setRowCountCache, and add comment to explain this function only used for test

morrySnow · 2026-06-08T09:59:13Z

/review

github-actions · 2026-06-08T10:58:55Z

PR approved by at least one committer and no changes requested.

github-actions · 2026-06-08T10:58:57Z

PR approved by anyone and no changes requested.

github-actions

Review result: no additional blocking issues found in this pass. I did not duplicate the existing inline thread about renaming/commenting replaceRowCountCacheForTest.

Critical checkpoint conclusions:

Goal/test: The PR invalidates ExternalRowCountCache on external metadata changes. The main refresh, partition add/drop, Hive insert, Iceberg action, drop, truncate, db, and catalog invalidation paths were traced; added unit tests cover table invalidation and manager wiring, though they do not exhaustively cover every metadata operation.
Scope: The change is small and focused, mostly replacing table-cache invalidation with table-level invalidation that also clears row counts, plus targeted partition-row-count invalidation.
Concurrency: The new cache invalidation uses Caffeine concurrent cache APIs; no new locks or lock-order changes were introduced. I did not find heavy work added under FE catalog/database/table locks.
Lifecycle/static initialization: No new static/global lifecycle dependency was introduced. The test-only cache replacement is already covered by an existing review thread.
Configuration/compatibility: No new config, persistence format, thrift protocol, storage format, or mixed-version compatibility issue was introduced.
Parallel paths: I checked refresh table/db/catalog, refresh partitions, Hive partition add/drop, Hive insert replay, Iceberg actions, metadata drop/truncate hooks, and engine-specific invalidation callers. The relevant content-changing paths appear covered; engine-specific property-cache removals do not appear to be content changes requiring row-count invalidation.
Conditional checks/error handling: No new suspicious defensive checks or silently ignored Status-like results were introduced in the FE Java paths.
Observability: Existing logging remains sufficient for this cache invalidation change; no new metric appears necessary.
Transaction/persistence/data writes: Hive insert writes the existing refresh edit log after local invalidation, and replay invalidates row count on partition/full refresh. I did not find a master-failover replay gap from this patch.
Performance: Catalog/db invalidation removes matching cache keys by scanning the row-count cache map. Given these operations are metadata refresh/removal paths rather than query hot paths and the cache capacity is bounded, this is acceptable.
Tests: New unit tests validate table invalidation reload behavior and ExternalMetaCacheMgr table invalidation wiring. I did not run tests in this review runner.

User focus points: No additional user-provided review focus was specified, so no extra focused issue was found.

foxtail463 · 2026-06-09T07:32:20Z

run buildall

hello-stephen · 2026-06-09T07:57:56Z

TPC-H: Total hot run time: 28934 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3934f5a2fdad47083bbbe88421b7f084c5bbddca, data reload: false
------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL ============================================
q1	17598	3989	4047	3989
q2	q3	10739	1420	785	785
q4	4683	477	338	338
q5	7614	881	579	579
q6	182	168	132	132
q7	791	830	638	638
q8	9353	1649	1702	1649
q9	5846	4539	4485	4485
q10	6741	1813	1535	1535
q11	439	273	257	257
q12	626	427	285	285
q13	18200	3381	2774	2774
q14	264	258	233	233
q15	q16	823	769	699	699
q17	944	896	959	896
q18	6803	5796	5684	5684
q19	1310	1306	1092	1092
q20	532	403	273	273
q21	5920	2667	2308	2308
q22	438	357	303	303
Total cold run time: 99846 ms
Total hot run time: 28934 ms
----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL ============================================
q1	4350	4491	4246	4246
q2	q3	4485	4919	4342	4342
q4	2049	2156	1389	1389
q5	4392	4275	4294	4275
q6	223	173	127	127
q7	1732	1622	1678	1622
q8	2923	2268	2129	2129
q9	8206	8309	8035	8035
q10	4841	4796	4264	4264
q11	565	413	402	402
q12	748	816	550	550
q13	3233	3676	2953	2953
q14	293	293	276	276
q15	q16	719	719	639	639
q17	1340	1344	1313	1313
q18	8052	7092	7200	7092
q19	1145	1174	1154	1154
q20	2230	2233	1915	1915
q21	5233	4511	4380	4380
q22	506	440	396	396
Total cold run time: 57265 ms
Total hot run time: 51499 ms

hello-stephen · 2026-06-09T08:08:46Z

TPC-DS: Total hot run time: 168174 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3934f5a2fdad47083bbbe88421b7f084c5bbddca, data reload: false
query5	4343	622	477	477
query6	443	205	181	181
query7	4832	577	309	309
query8	368	209	196	196
query9	8817	4034	3983	3983
query10	478	306	253	253
query11	5868	2318	2171	2171
query12	165	100	99	99
query13	1242	589	436	436
query14	6393	5399	5058	5058
query14_1	4405	4394	4410	4394
query15	208	197	182	182
query16	1000	455	447	447
query17	1146	718	597	597
query18	2494	479	354	354
query19	211	185	148	148
query20	122	107	109	107
query21	227	152	118	118
query22	13636	13477	13408	13408
query23	17461	16420	16117	16117
query23_1	16352	16236	16220	16220
query24	7505	1778	1286	1286
query24_1	1304	1280	1329	1280
query25	573	471	406	406
query26	1305	321	163	163
query27	2669	554	349	349
query28	4468	2010	2009	2009
query29	1114	624	518	518
query30	317	238	195	195
query31	1137	1078	974	974
query32	111	68	62	62
query33	537	330	263	263
query34	1195	1124	644	644
query35	762	775	684	684
query36	1420	1435	1248	1248
query37	148	99	86	86
query38	3219	3114	3036	3036
query39	942	895	884	884
query39_1	864	876	886	876
query40	218	118	99	99
query41	77	61	61	61
query42	93	91	92	91
query43	324	334	274	274
query44 query45	197	186	175	175
query46	1071	1189	696	696
query47	2366	2351	2266	2266
query48	396	420	282	282
query49	620	476	353	353
query50	1050	332	263	263
query51	4298	4292	4274	4274
query52	86	88	75	75
query53	229	273	186	186
query54	263	211	193	193
query55	77	74	70	70
query56	242	217	223	217
query57	1412	1404	1311	1311
query58	253	225	204	204
query59	1561	1627	1480	1480
query60	282	250	227	227
query61	157	161	157	157
query62	703	646	579	579
query63	230	191	181	181
query64	2546	770	632	632
query65 query66	1809	449	339	339
query67	29637	29545	29539	29539
query68 query69	413	296	246	246
query70	952	945	929	929
query71	311	225	194	194
query72	3098	2672	2385	2385
query73	838	720	437	437
query74	5143	4969	4773	4773
query75	2650	2565	2228	2228
query76	2335	1169	765	765
query77	344	372	276	276
query78	12388	12401	11824	11824
query79	1432	1035	773	773
query80	579	465	384	384
query81	447	285	253	253
query82	583	156	118	118
query83	349	289	247	247
query84 query85	869	519	442	442
query86	363	295	274	274
query87	3518	3292	3238	3238
query88	3630	2733	2702	2702
query89	402	384	334	334
query90	1985	173	169	169
query91	179	162	138	138
query92	66	60	57	57
query93	1426	1394	861	861
query94	542	354	312	312
query95	678	460	343	343
query96	1027	773	324	324
query97	2714	2696	2554	2554
query98	209	206	212	206
query99	1145	1171	1017	1017
Total cold run time: 250845 ms
Total hot run time: 168174 ms

hello-stephen · 2026-06-09T10:46:52Z

FE Regression Coverage Report

Increment line coverage 14.29% (17/119) 🎉
Increment coverage report
Complete coverage report

foxtail463 · 2026-06-10T06:47:52Z

run nonConcurrent

hello-stephen · 2026-06-10T09:20:34Z

FE Regression Coverage Report

Increment line coverage 8.46% (17/201) 🎉
Increment coverage report
Complete coverage report

morrySnow · 2026-06-17T02:38:40Z

/review

github-actions

Automated review completed for PR 64160. I did not find blocking issues in the changed code.

Checkpoint conclusions:

Goal/test: The PR targets stale external row-count cache after external metadata changes. The changed paths now invalidate row-count cache for catalog/db/table invalidation, HMS partition add/drop/refresh/replay, Hive insert commit, and Iceberg table-changing actions. Added FE unit coverage verifies table invalidation and reload behavior.
Scope/focus: The changes are small and focused on unifying metadata cache invalidation with row-count cache invalidation.
Concurrency/locking: No new locks or shared mutable structures outside the existing Caffeine cache paths. Catalog/db invalidation scans the row-count cache via concurrent cache views; table invalidation uses the existing synchronous invalidation API.
Lifecycle/static/config/compatibility: No new static lifecycle, config item, storage format, protocol, or FE/BE variable compatibility concern found.
Parallel paths: I checked refresh table/db/catalog, partition refresh replay, HMS add/drop partition events, Hive insert commit, and Iceberg snapshot/manifest actions. The changed data-mutating paths are covered; engine-specific cache config removals are not data-change paths.
Transaction/persistence: Hive insert and refresh replay paths propagate row-count invalidation consistently with the existing edit-log refresh model.
Testing/results: New tests are FE unit tests only; no regression .out files are involved. I did not run tests in this review runner.
Observability/performance: No new hot-path logging. The only broad scan is bounded row-count cache invalidation on catalog/db invalidation, not a query hot path.
Security: The PR does not touch authentication, authorization, network boundary, or tenancy-sensitive behavior.

User focus response: .code-review.KuD14R/review_focus.txt contained no additional focus points; no extra focus-specific issue was found.

foxtail463 · 2026-08-17T07:12:44Z

run buildall

foxtail463 · 2026-08-17T07:12:58Z

/review

foxtail463 · 2026-08-17T08:59:37Z

run performance

…changes

foxtail463 · 2026-08-17T09:12:17Z

run buildall

foxtail463 · 2026-08-17T12:40:14Z

run buildall

hello-stephen · 2026-08-17T14:48:17Z

TPC-H: Total hot run time: 17193 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4ba3cf68954b0ff90691746713f897c94a1db645, data reload: false
------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL ============================================
q1	17585	2973	2994	2973
q2	q3	10866	877	517	517
q4	4673	263	205	205
q5	7664	581	393	393
q6	144	118	97	97
q7	528	515	404	404
q8	9257	899	937	899
q9	3598	2419	2438	2419
q10	6542	874	768	768
q11	435	259	247	247
q12	688	420	352	352
q13	17842	1571	1180	1180
q14	160	156	140	140
q15	q16	486	407	379	379
q17	771	788	749	749
q18	3234	2320	2270	2270
q19	1116	901	805	805
q20	677	537	452	452
q21	5342	1711	1918	1711
q22	331	274	233	233
Total cold run time: 91939 ms
Total hot run time: 17193 ms
----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL ============================================
q1	3360	3289	3287	3287
q2	q3	2248	2375	2155	2155
q4	1203	1182	888	888
q5	2215	2186	2157	2157
q6	173	120	93	93
q7	1091	924	851	851
q8	1608	1431	1396	1396
q9	3188	3136	3107	3107
q10	1977	1884	1677	1677
q11	353	271	257	257
q12	452	432	346	346
q13	1499	1562	1161	1161
q14	170	182	185	182
q15	q16	407	396	363	363
q17	1038	1034	1015	1015
q18	5187	4489	4867	4489
q19	863	858	847	847
q20	1015	967	825	825
q21	3776	3220	3310	3220
q22	410	339	313	313
Total cold run time: 32233 ms
Total hot run time: 28629 ms

hello-stephen · 2026-08-17T14:55:18Z

TPC-DS: Total hot run time: 91896 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4ba3cf68954b0ff90691746713f897c94a1db645, data reload: false
query5	4296	450	345	345
query6	428	170	163	163
query7	4800	441	272	272
query8	292	133	140	133
query9	8677	2943	2928	2928
query10	458	281	242	242
query11	5470	1299	1186	1186
query12	134	76	76	76
query13	1204	455	341	341
query14	6249	3090	2802	2802
query14_1	2706	2707	2724	2707
query15	173	139	130	130
query16	947	381	359	359
query17	790	471	368	368
query18	2338	335	268	268
query19	175	141	108	108
query20	76	76	77	76
query21	218	120	101	101
query22	8184	8192	8120	8120
query23	9931	9237	8660	8660
query23_1	8724	8941	8887	8887
query24	7445	1239	899	899
query24_1	898	868	794	794
query25	407	281	235	235
query26	1261	266	166	166
query27	2729	438	288	288
query28	4612	1498	1489	1489
query29	932	420	368	368
query30	274	177	153	153
query31	774	442	373	373
query32	98	48	50	48
query33	457	225	176	176
query34	991	830	564	564
query35	460	420	368	368
query36	570	600	531	531
query37	116	83	76	76
query38	1038	914	862	862
query39	548	492	489	489
query39_1	507	482	491	482
query40	222	167	117	117
query41	51	51	48	48
query42	81	78	81	78
query43	255	262	223	223
query44 query45	111	106	102	102
query46	813	855	557	557
query47	789	778	759	759
query48	305	294	229	229
query49	545	282	260	260
query50	854	315	268	268
query51	7978	8010	7894	7894
query52	80	76	69	69
query53	212	222	169	169
query54	254	215	209	209
query55	67	69	59	59
query56	235	243	236	236
query57	719	653	630	630
query58	231	196	196	196
query59	1217	1208	1067	1067
query60	258	218	202	202
query61	140	153	146	146
query62	348	193	185	185
query63	180	155	158	155
query64	2701	780	543	543
query65 query66	1790	293	242	242
query67	9708	9601	9915	9601
query68 query69	373	251	201	201
query70	609	608	597	597
query71	310	248	244	244
query72	2339	1739	1580	1580
query73	657	583	334	334
query74	1574	1229	1166	1166
query75	1246	1150	1068	1068
query76	2292	746	562	562
query77	274	281	213	213
query78	3972	3778	3336	3336
query79	2990	821	593	593
query80	1657	406	377	377
query81	529	220	181	181
query82	662	141	106	106
query83	322	277	250	250
query84 query85	1090	434	413	413
query86	418	176	171	171
query87	1023	1000	949	949
query88	2917	2146	2184	2146
query89	311	229	216	216
query90	2122	155	159	155
query91	162	152	127	127
query92	50	50	43	43
query93	1828	1206	834	834
query94	631	252	230	230
query95	617	358	414	358
query96	832	615	282	282
query97	1127	1095	1043	1043
query98	179	138	137	137
query99	433	359	321	321
Total cold run time: 183498 ms
Total hot run time: 91896 ms

foxtail463 · 2026-08-18T02:40:32Z

run feut

hello-stephen · 2026-08-18T05:34:43Z

FE UT Coverage Report

Increment line coverage 83.57% (351/420) 🎉
Increment coverage report
Complete coverage report

foxtail463 · 2026-08-18T06:09:46Z

run buildall

hello-stephen · 2026-08-18T06:29:33Z

TPC-H: Total hot run time: 17189 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4ba3cf68954b0ff90691746713f897c94a1db645, data reload: false
------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL ============================================
q1	17578	3111	3062	3062
q2	q3	10894	840	490	490
q4	4677	250	212	212
q5	7670	575	398	398
q6	138	118	95	95
q7	528	492	383	383
q8	9428	911	857	857
q9	3498	2407	2375	2375
q10	6512	841	711	711
q11	499	262	255	255
q12	695	393	331	331
q13	17919	1523	1175	1175
q14	163	151	141	141
q15	q16	441	395	357	357
q17	812	805	843	805
q18	3048	2243	2231	2231
q19	1103	907	785	785
q20	669	532	454	454
q21	4889	1851	1845	1845
q22	319	269	227	227
Total cold run time: 91480 ms
Total hot run time: 17189 ms
----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL ============================================
q1	3416	3401	3411	3401
q2	q3	2184	2282	2221	2221
q4	1185	1148	891	891
q5	2182	2103	2108	2103
q6	166	116	90	90
q7	1012	910	843	843
q8	1601	1406	1408	1406
q9	3162	3092	3057	3057
q10	1833	1815	1597	1597
q11	359	268	259	259
q12	455	421	338	338
q13	1490	1540	1140	1140
q14	175	182	170	170
q15	q16	396	391	359	359
q17	1030	1030	1030	1030
q18	4958	4404	4716	4404
q19	1253	797	812	797
q20	951	915	790	790
q21	3537	3307	3257	3257
q22	404	355	328	328
Total cold run time: 31749 ms
Total hot run time: 28481 ms

hello-stephen · 2026-08-18T06:36:02Z

TPC-DS: Total hot run time: 80143 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4ba3cf68954b0ff90691746713f897c94a1db645, data reload: false
query5	4264	422	341	341
query6	473	159	149	149
query7	4803	412	251	251
query8	354	126	117	117
query9	8702	2906	2916	2906
query10	409	260	230	230
query11	5450	1035	918	918
query12	121	71	73	71
query13	1193	447	314	314
query14	5953	2196	2081	2081
query14_1	1973	1968	1969	1968
query15	177	119	116	116
query16	1085	376	364	364
query17	912	441	360	360
query18	2332	319	226	226
query19	178	136	107	107
query20	69	67	69	67
query21	232	114	99	99
query22	5320	5249	5303	5249
query23	6447	6141	5899	5899
query23_1	6153	6127	6121	6121
query24	7335	1079	743	743
query24_1	762	755	762	755
query25	440	280	269	269
query26	1273	267	194	194
query27	2684	421	283	283
query28	4670	1469	1519	1469
query29	1107	422	346	346
query30	292	179	150	150
query31	938	422	353	353
query32	113	47	46	46
query33	492	212	173	173
query34	1152	816	485	485
query35	394	396	341	341
query36	616	546	543	543
query37	175	78	71	71
query38	1063	835	801	801
query39	508	503	467	467
query39_1	493	444	474	444
query40	240	129	107	107
query41	53	50	51	50
query42	79	78	77	77
query43	238	245	207	207
query44 query45	107	109	99	99
query46	818	819	526	526
query47	752	773	700	700
query48	304	308	226	226
query49	533	246	181	181
query50	821	321	272	272
query51	7912	7949	7935	7935
query52	78	79	71	71
query53	204	215	161	161
query54	288	191	201	191
query55	108	60	57	57
query56	265	245	225	225
query57	750	660	588	588
query58	252	195	191	191
query59	1193	1236	1110	1110
query60	284	216	211	211
query61	163	149	142	142
query62	437	216	179	179
query63	185	154	159	154
query64	2852	659	600	600
query65 query66	1893	295	261	261
query67	10053	9980	9627	9627
query68 query69	395	218	190	190
query70	627	623	623	623
query71	321	242	239	239
query72	2374	1700	1792	1700
query73	751	554	368	368
query74	2033	1229	1116	1116
query75	1235	1129	998	998
query76	2437	701	515	515
query77	253	253	208	208
query78	3954	3857	3254	3254
query79	2632	777	579	579
query80	1618	388	341	341
query81	593	199	177	177
query82	919	132	101	101
query83	358	249	228	228
query84 query85	961	452	393	393
query86	538	172	169	169
query87	1026	979	890	890
query88	3019	2139	2162	2139
query89	377	232	205	205
query90	2000	151	146	146
query91	183	153	145	145
query92	49	54	41	41
query93	2074	1152	778	778
query94	819	261	259	259
query95	685	424	352	352
query96	807	575	278	278
query97	1090	1073	1037	1037
query98	213	130	130	130
query99	498	343	315	315
Total cold run time: 176238 ms
Total hot run time: 80143 ms

hello-stephen · 2026-08-18T10:39:19Z

FE Regression Coverage Report

Increment line coverage 58.39% (320/548) 🎉
Increment coverage report
Complete coverage report

foxtail463 · 2026-08-18T17:02:04Z

run cloud_p0

hello-stephen · 2026-08-18T21:19:01Z

FE Regression Coverage Report

Increment line coverage 46.51% (320/688) 🎉
Increment coverage report
Complete coverage report

924060929 · 2026-08-19T02:53:48Z

/review

github-actions · 2026-08-19T03:00:15Z

Codex automated review failed and did not complete.

Error: Review context preparation failed before Codex ran; inspect the 'Prepare authoritative PR context and required AGENTS guides' step.
Workflow run: https://github.com/apache/doris/actions/runs/32210189660

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

924060929 · 2026-08-20T06:50:10Z

/review

github-actions

Automated review summary

No new valuable findings beyond the existing live review threads. This is a comment-only review with zero inline comments; two full review rounds converged, and every candidate was independently validated, deduplicated, or dismissed with base/head evidence.

Critical checkpoints:

Goal, scope, and tests: The 29-file change consistently targets external row-count invalidation across refresh, DDL, event, replay, action, rewrite, and insert paths. The added tests cover exact/scoped invalidation, mapped identities, concurrency, replay, and post-commit fencing; their negative space was reviewed as well.
Concurrency and lifecycle: Row-count mapping publication/removal is serialized, connector -> engine -> row-count ordering is preserved, and passive object eviction remains separate from explicit structural invalidation. No new publication, teardown, or exception-ordering defect was established.
Identity, configuration, and compatibility: Modes 0/1/2, cross-database rename, hot/cold/uninitialized replay, property resets, and current/legacy log shapes were traced. Potential concerns either predate this PR or are already covered by live compatibility/identity threads, so they were not duplicated.
Transactions, persistence, and data writes: Single actions, distributed rewrite, and normal inserts place the newly added final fence after the owning commit and before later fallible result work. Failure/no-op paths avoid the new final fence; no new durability or retry-visible regression was found.
Parallel paths and conditions: Coordinator/follower, mapped/unmapped, hot/cold, exact/case-insensitive, empty/non-empty, and single/distributed branches were checked. No distinct head-only issue remained.
Performance, observability, and variables: The known O(N) invalidation concerns are already live-thread fenced. No new FE/BE variable propagation, logging, or configuration defect was introduced.
Test execution: Per the review-run prompt, no builds or tests were run; conclusions are from the authoritative diff, production control-flow tracing, bundled tests, and live-thread deduplication.

User focus: No additional review focus was provided.

Review status: complete and converged in Round 2; zero new inline comments.

1

Problem Summary: PR apache#64160 added row-count cache invalidation and a test-only engine cache replacement hook after this branch diverged. Merging current master conflicted with the scoped metadata cache test setup and left the old registry-based test hook incompatible with the new cacheTypes framework. Solution: Preserve the row-count cache implementation and invalidation ordering from apache#64160. Resolve RefreshManagerTest against FeMetaCacheEntry and migrate replaceEngineCachesForTest to register replacement engines and aliases through the new cacheTypes model. Tests: - 116 focused FE unit tests passed - 104 connector cache unit tests passed - Maven Checkstyle reactor passed with 0 violations Issue Number: closeapache#66633 Related PR: apache#64160

### What problem does this PR solve? Issue Number: closeapache#64159 Related PR: apache#64160 Problem Summary: Incremental metadata events published the ID-to-name mapping before the matching database or table object was committed to the shared cache. A concurrent lookup by ID could therefore load and publish a different object in that gap, which was then replaced and cleaned up by the event update. Publish the ID mapping and object inside one guarded cache commit, and fence loads that overlap either side of the auxiliary action so they can not publish stale objects. ### Release note None ### Check List (For Author) - Test: Unit Test - ScopedMetaCacheConcurrencyTest - FeMetaCacheEntryTest - ExternalCatalogTest - ExternalDatabaseTest - ./build.sh --fe - Behavior changed: No - Does this need documentation: No

### What problem does this PR solve? Issue Number: closeapache#64159 Related PR: apache#64160 Problem Summary: Incremental metadata events published the ID-to-name mapping before the matching database or table object was committed to the shared cache. A concurrent lookup by ID could therefore load and publish a different object in that gap, which was then replaced and cleaned up by the event update. Publish the ID mapping and object inside one guarded cache commit, fence overlapping loads, and make the object-and-ID step run only once when the outer name-cache CAS retries so a newer object is never replaced by replaying an already completed event update. ### Release note None ### Check List (For Author) - Test: Unit Test - ScopedMetaCacheConcurrencyTest - FeMetaCacheEntryTest - ExternalCatalogTest - ExternalDatabaseTest - ./build.sh --fe - Behavior changed: No - Does this need documentation: No

foxtail463 requested review from 924060929, englefly, morrySnow and starocean999 as code owners June 5, 2026 12:48

morrySnow added dev/3.1.x dev/4.0.x dev/4.1.x labels Jun 8, 2026

morrySnow previously approved these changes Jun 8, 2026
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Jun 8, 2026

github-actionsBot added the reviewed label Jun 8, 2026

github-actionsBot reviewed Jun 8, 2026
View reviewed changes

foxtail463 dismissed morrySnow’s stale review via 3934f5aJune 9, 2026 07:32

github-actionsBot removed the approved Indicates a PR has been approved by one committer. label Jun 9, 2026

github-actionsBot reviewed Jun 17, 2026
View reviewed changes

[fix](meta cache) Invalidate external row count cache after metadata …
4ba3cf6
…changes

foxtail463 force-pushed the fix/invalidate-external-rowcount-cache branch from db93496 to 4ba3cf6Compare August 17, 2026 09:11

github-actionsBot reviewed Aug 20, 2026
View reviewed changes

gavinchou approved these changes Aug 20, 2026
View reviewed changes

CalvinKirs approved these changes Aug 20, 2026
View reviewed changes

starocean999 approved these changes Aug 20, 2026
View reviewed changes

924060929 merged commit 86ed4ca into apache:masterAug 20, 2026
34 checks passed

github-actionsBot added the dev/4.1.x-conflict label Aug 20, 2026

Uh oh!

Conversation

foxtail463 commented Jun 5, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem Summary

Solution

Uh oh!

hello-stephen commented Jun 5, 2026

Uh oh!

foxtail463 commented Jun 5, 2026

Uh oh!

hello-stephen commented Jun 5, 2026

Uh oh!

hello-stephen commented Jun 5, 2026

Uh oh!

hello-stephen commented Jun 5, 2026

FE UT Coverage Report

Uh oh!

hello-stephen commented Jun 5, 2026

FE Regression Coverage Report

Uh oh!

morrySnowJun 8, 2026

Choose a reason for hiding this comment

Uh oh!

morrySnow commented Jun 8, 2026

Uh oh!

github-actionsBot commented Jun 8, 2026

Uh oh!

github-actionsBot commented Jun 8, 2026

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

foxtail463 commented Jun 9, 2026

Uh oh!

hello-stephen commented Jun 9, 2026

Uh oh!

hello-stephen commented Jun 9, 2026

Uh oh!

hello-stephen commented Jun 9, 2026

FE Regression Coverage Report

Uh oh!

foxtail463 commented Jun 10, 2026

Uh oh!

hello-stephen commented Jun 10, 2026

FE Regression Coverage Report

Uh oh!

morrySnow commented Jun 17, 2026

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

foxtail463 commented Aug 17, 2026

Uh oh!

foxtail463 commented Aug 17, 2026

Uh oh!

foxtail463 commented Aug 17, 2026

Uh oh!

foxtail463 commented Aug 17, 2026

Uh oh!

foxtail463 commented Aug 17, 2026

Uh oh!

hello-stephen commented Aug 17, 2026

Uh oh!

hello-stephen commented Aug 17, 2026

Uh oh!

foxtail463 commented Aug 18, 2026

Uh oh!

hello-stephen commented Aug 18, 2026

FE UT Coverage Report

Uh oh!

foxtail463 commented Aug 18, 2026

Uh oh!

hello-stephen commented Aug 18, 2026

Uh oh!

hello-stephen commented Aug 18, 2026

Uh oh!

hello-stephen commented Aug 18, 2026

FE Regression Coverage Report

Uh oh!

foxtail463 commented Jun 5, 2026•
edited
Loading