[enhance](iceberg) Refactor Iceberg metadata cache structure and add table cache test cases by suxiaogang223 · Pull Request #59716 · apache/doris

suxiaogang223 · 2026-01-09T08:14:42Z

What problem does this PR solve?

Description

Changes

This PR refactors the Iceberg metadata cache structure to improve code organization and adds comprehensive test cases for table cache behavior.

Main Changes

1. Refactored IcebergMetadataCache

Introduced IcebergTableCacheValue to encapsulate table-related metadata
Removed redundant snapshotListCache and snapshotCache
Merged snapshot information into IcebergTableCacheValue with lazy loading
Simplified cache structure from 3 separate caches to 2: tableCache and viewCache

Before:

privateLoadingCache<IcebergMetadataCacheKey, List<Snapshot>> snapshotListCache;
privateLoadingCache<IcebergMetadataCacheKey, Table> tableCache;
privateLoadingCache<IcebergMetadataCacheKey, IcebergSnapshotCacheValue> snapshotCache;

After:

privateLoadingCache<IcebergMetadataCacheKey, IcebergTableCacheValue> tableCache;
privateLoadingCache<IcebergMetadataCacheKey, View> viewCache;

2. Lazy Loading for Snapshot Cache

Snapshot cache is now loaded on-demand through IcebergTableCacheValue.getSnapshotCacheValue()
Reduced unnecessary memory footprint for queries that don't require snapshot information
Snapshot information is mainly used for MTMV scenarios

3. Simplified Cache API

getIcebergTable(): Returns the Table object directly from IcebergTableCacheValue
getSnapshotCache(): Returns snapshot cache value with lazy loading
getSnapshotList(): Returns snapshot list from the Table object

4. Test Cases

Added comprehensive test case test_iceberg_table_cache to verify cache behavior
Tests cover both cache-enabled and cache-disabled scenarios
Validated external modifications (INSERT, DELETE, UPDATE, schema changes) are properly handled

Benefits

Aspect	Improvement
Memory Usage	Reduced by eliminating duplicate caching of snapshot information
Code Structure	Cleaner with single `IcebergTableCacheValue` instead of multiple separate caches
Performance	Better with lazy loading of snapshot cache only when needed
Maintainability	Simpler cache management logic

Test Results

Added regression test: test_iceberg_table_cache.groovy
Tests validate cache behavior with TTL and external modifications
Verified cache invalidation works correctly with REFRESH TABLE
Test scenarios include:
- DML operations (INSERT, DELETE, UPDATE, INSERT OVERWRITE)
- Schema changes (ADD/DROP/RENAME COLUMN, ALTER COLUMN TYPE)
- Partition evolution (ADD/DROP/REPLACE PARTITION FIELD)

Related Files

Core Changes:

IcebergMetadataCache.java - Refactored cache structure
IcebergTableCacheValue.java - New class to encapsulate table metadata
IcebergExternalCatalog.java - Updated cache-related configurations

Tests:

test_iceberg_table_cache.groovy - Comprehensive cache behavior tests
Suite.groovy - Updated getSparkIcebergContainerName() implementation

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2026-01-09T08:14:47Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

suxiaogang223 · 2026-01-09T08:36:33Z

run external

suxiaogang223 · 2026-01-12T08:53:25Z

run buildall

hello-stephen · 2026-01-12T10:12:38Z

FE UT Coverage Report

Increment line coverage 0.00% (0/36) 🎉
Increment coverage report
Complete coverage report

doris-robot · 2026-01-12T10:15:14Z

TPC-H: Total hot run time: 32189 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1574f3a0840831ea973580b5275b27182d07b2bf, data reload: false
------ Round 1 ----------------------------------
q1	17636	4183	4086	4086
q2	2077	377	245	245
q3	10116	1275	724	724
q4	10208	855	309	309
q5	7538	2107	1841	1841
q6	187	168	135	135
q7	911	811	671	671
q8	9273	1414	1159	1159
q9	5100	4604	4559	4559
q10	6839	1852	1436	1436
q11	537	277	275	275
q12	735	756	590	590
q13	17802	3842	3078	3078
q14	297	298	274	274
q15	609	517	510	510
q16	667	684	647	647
q17	676	764	578	578
q18	6747	6583	6973	6583
q19	1355	996	669	669
q20	451	404	269	269
q21	3285	2654	2506	2506
q22	1170	1126	1045	1045
Total cold run time: 104216 ms
Total hot run time: 32189 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4395	4248	4257	4248
q2	324	411	333	333
q3	2299	2747	2449	2449
q4	1516	1815	1484	1484
q5	4601	4340	4422	4340
q6	224	177	129	129
q7	1958	1941	1816	1816
q8	2570	2413	2394	2394
q9	7135	7364	7264	7264
q10	2508	2691	2307	2307
q11	579	495	464	464
q12	701	723	561	561
q13	3349	3817	3040	3040
q14	268	289	266	266
q15	525	490	482	482
q16	616	674	616	616
q17	1099	1321	1362	1321
q18	7417	7489	7337	7337
q19	808	774	777	774
q20	1896	1974	1872	1872
q21	4485	4339	4196	4196
q22	1096	1020	995	995
Total cold run time: 50369 ms
Total hot run time: 48688 ms

doris-robot · 2026-01-12T10:26:06Z

TPC-DS: Total hot run time: 173580 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1574f3a0840831ea973580b5275b27182d07b2bf, data reload: false
query5	4444	598	436	436
query6	334	231	222	222
query7	4218	466	269	269
query8	330	258	246	246
query9	8770	2715	2657	2657
query10	508	354	336	336
query11	15270	15196	14803	14803
query12	173	112	111	111
query13	1261	477	368	368
query14	6214	2967	2704	2704
query14_1	2632	2581	2635	2581
query15	206	194	175	175
query16	1012	462	468	462
query17	1112	690	588	588
query18	2507	450	353	353
query19	229	228	198	198
query20	126	117	115	115
query21	224	145	125	125
query22	3994	3948	3896	3896
query23	16088	15712	15492	15492
query23_1	15825	15645	15676	15645
query24	7694	1600	1175	1175
query24_1	1185	1186	1213	1186
query25	580	521	451	451
query26	1240	281	165	165
query27	2754	481	311	311
query28	4515	2151	2142	2142
query29	796	569	476	476
query30	315	247	211	211
query31	799	639	560	560
query32	82	77	76	76
query33	554	346	300	300
query34	909	886	521	521
query35	718	784	681	681
query36	921	925	824	824
query37	133	95	79	79
query38	2740	2667	2641	2641
query39	797	756	729	729
query39_1	721	726	747	726
query40	229	136	118	118
query41	74	68	69	68
query42	106	103	102	102
query43	446	433	412	412
query44	1324	729	717	717
query45	192	188	181	181
query46	854	950	598	598
query47	1370	1456	1308	1308
query48	319	341	248	248
query49	638	419	353	353
query50	647	280	216	216
query51	3752	3799	3749	3749
query52	106	116	111	111
query53	304	327	274	274
query54	304	268	266	266
query55	81	77	72	72
query56	343	299	278	278
query57	1036	983	954	954
query58	267	246	254	246
query59	2187	2247	2036	2036
query60	316	321	298	298
query61	165	157	153	153
query62	420	352	312	312
query63	294	261	266	261
query64	4860	1316	994	994
query65	3835	3776	3749	3749
query66	1428	420	311	311
query67	15460	16105	15120	15120
query68	6029	992	713	713
query69	517	353	312	312
query70	1076	958	927	927
query71	418	294	276	276
query72	6008	3420	3527	3420
query73	774	721	300	300
query74	8871	8891	8623	8623
query75	2817	2831	2450	2450
query76	3456	1050	633	633
query77	534	377	274	274
query78	9874	9836	9158	9158
query79	1584	944	598	598
query80	704	572	463	463
query81	513	257	226	226
query82	449	147	110	110
query83	272	258	239	239
query84	257	118	101	101
query85	902	512	465	465
query86	433	302	318	302
query87	2872	2850	2763	2763
query88	3224	2230	2231	2230
query89	399	351	345	345
query90	2148	156	158	156
query91	175	169	141	141
query92	80	64	68	64
query93	1495	904	530	530
query94	637	319	297	297
query95	582	381	296	296
query96	580	452	214	214
query97	2371	2391	2331	2331
query98	239	200	202	200
query99	578	594	490	490
Total cold run time: 254239 ms
Total hot run time: 173580 ms

hello-stephen · 2026-01-12T12:37:34Z

FE Regression Coverage Report

Increment line coverage 0.00% (0/36) 🎉
Increment coverage report
Complete coverage report

…nd remove unused caches

suxiaogang223 · 2026-01-26T12:19:08Z

run buildall

doris-robot · 2026-01-26T12:46:08Z

TPC-H: Total hot run time: 32499 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 35ab7872eaf6d9ed159f68593e694e5cd4ba731d, data reload: false
------ Round 1 ----------------------------------
q1	17650	5227	5099	5099
q2	2080	307	200	200
q3	10154	1297	735	735
q4	10219	885	315	315
q5	7555	2161	1894	1894
q6	196	178	147	147
q7	884	745	627	627
q8	9257	1375	1077	1077
q9	5452	4941	4818	4818
q10	6775	1952	1561	1561
q11	521	309	275	275
q12	337	387	230	230
q13	17782	4035	3236	3236
q14	230	244	217	217
q15	879	838	808	808
q16	685	680	613	613
q17	626	819	478	478
q18	6771	6449	6522	6449
q19	1240	983	594	594
q20	398	337	227	227
q21	2611	2008	1933	1933
q22	1046	992	966	966
Total cold run time: 103348 ms
Total hot run time: 32499 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5324	5299	5284	5284
q2	261	334	262	262
q3	2207	2650	2266	2266
q4	1344	1737	1321	1321
q5	4285	4164	4322	4164
q6	302	205	163	163
q7	2351	1994	1836	1836
q8	2581	2397	2380	2380
q9	7611	7677	7487	7487
q10	2874	3173	2599	2599
q11	556	471	466	466
q12	702	781	680	680
q13	3922	4415	3537	3537
q14	318	328	289	289
q15	854	800	825	800
q16	680	766	720	720
q17	1175	1348	1383	1348
q18	8413	7967	7980	7967
q19	913	882	891	882
q20	2060	2176	1991	1991
q21	4828	4597	4186	4186
q22	1037	1012	971	971
Total cold run time: 54598 ms
Total hot run time: 51599 ms

doris-robot · 2026-01-26T13:02:57Z

ClickBench: Total hot run time: 28.32 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 35ab7872eaf6d9ed159f68593e694e5cd4ba731d, data reload: false
query1	0.06	0.05	0.05
query2	0.10	0.05	0.04
query3	0.25	0.08	0.08
query4	1.60	0.11	0.11
query5	0.28	0.26	0.25
query6	1.17	0.67	0.67
query7	0.03	0.03	0.03
query8	0.05	0.04	0.04
query9	0.56	0.51	0.50
query10	0.55	0.55	0.54
query11	0.15	0.10	0.10
query12	0.14	0.11	0.11
query13	0.63	0.62	0.61
query14	1.07	1.07	1.07
query15	0.86	0.87	0.88
query16	0.40	0.39	0.39
query17	1.16	1.16	1.14
query18	0.23	0.21	0.22
query19	2.09	2.04	1.96
query20	0.02	0.01	0.02
query21	15.40	0.27	0.14
query22	5.50	0.05	0.05
query23	15.82	0.28	0.10
query24	2.22	0.25	0.56
query25	0.09	0.07	0.11
query26	0.14	0.13	0.12
query27	0.09	0.08	0.07
query28	4.74	1.11	0.96
query29	12.57	3.97	3.19
query30	0.27	0.14	0.11
query31	2.82	0.63	0.39
query32	3.28	0.60	0.49
query33	3.27	3.22	3.20
query34	16.47	5.43	4.74
query35	4.77	4.84	4.80
query36	0.66	0.49	0.50
query37	0.11	0.08	0.07
query38	0.08	0.04	0.04
query39	0.04	0.02	0.02
query40	0.18	0.16	0.15
query41	0.09	0.04	0.03
query42	0.04	0.04	0.02
query43	0.05	0.04	0.04
Total cold run time: 100.1 s
Total hot run time: 28.32 s

doris-robot · 2026-01-27T11:25:16Z

ClickBench: Total hot run time: 28.19 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 991bc75b6530dc1f39b1b3d9dec7880b810253b7, data reload: false
query1	0.05	0.04	0.04
query2	0.10	0.04	0.04
query3	0.25	0.08	0.09
query4	1.60	0.11	0.11
query5	0.29	0.24	0.25
query6	1.17	0.67	0.68
query7	0.04	0.03	0.02
query8	0.06	0.04	0.03
query9	0.56	0.50	0.48
query10	0.56	0.55	0.55
query11	0.14	0.09	0.09
query12	0.14	0.10	0.11
query13	0.63	0.61	0.62
query14	1.04	1.05	1.07
query15	0.87	0.88	0.88
query16	0.39	0.39	0.38
query17	1.13	1.08	1.16
query18	0.23	0.21	0.20
query19	2.14	2.03	2.06
query20	0.02	0.01	0.02
query21	15.43	0.24	0.14
query22	5.28	0.05	0.04
query23	16.10	0.28	0.10
query24	1.04	0.79	0.26
query25	0.09	0.05	0.07
query26	0.14	0.13	0.12
query27	0.09	0.06	0.05
query28	4.13	1.15	0.96
query29	12.57	3.88	3.15
query30	0.28	0.13	0.12
query31	2.82	0.67	0.40
query32	3.23	0.60	0.49
query33	3.20	3.21	3.32
query34	16.40	5.39	4.75
query35	4.83	4.81	4.79
query36	0.66	0.49	0.48
query37	0.11	0.08	0.07
query38	0.07	0.04	0.04
query39	0.05	0.03	0.03
query40	0.18	0.17	0.15
query41	0.10	0.03	0.03
query42	0.04	0.03	0.03
query43	0.05	0.04	0.04
Total cold run time: 98.3 s
Total hot run time: 28.19 s

hello-stephen · 2026-01-27T11:51:01Z

FE UT Coverage Report

Increment line coverage 0.00% (0/92) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-01-27T14:40:15Z

FE Regression Coverage Report

Increment line coverage 46.74% (43/92) 🎉
Increment coverage report
Complete coverage report

Copilot

Pull request overview

This PR refactors the Iceberg metadata cache structure to improve code organization, reduce memory overhead, and add comprehensive test coverage for table cache behavior. The main improvements include consolidating three separate caches into two, implementing lazy loading for snapshot cache, and fixing a spelling error in a method name.

Changes:

Introduced IcebergTableCacheValue to encapsulate table metadata with lazy-loaded snapshot cache
Removed redundant snapshotListCache and snapshotCache, consolidating them into IcebergTableCacheValue
Renamed getLastedIcebergSnapshot to getLatestIcebergSnapshot (spelling correction)
Added comprehensive test suite covering DML operations, schema changes, and partition evolution

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
`IcebergTableCacheValue.java`	New class implementing lazy-loaded snapshot cache with thread-safe double-checked locking
`IcebergMetadataCache.java`	Simplified cache structure from 3 caches to 2; removed snapshot-specific caches and cache statistics
`IcebergUtils.java`	Refactored method signatures to support passing Table instances; renamed methods for consistency
`IcebergExternalCatalog.java`	Removed deprecated `ICEBERG_SNAPSHOT_META_CACHE_TTL_SECOND` property
`IcebergExternalTable.java`	Updated to use new simplified cache API methods
`IcebergDlaTable.java`	Updated to use new cache API for Hive-based Iceberg tables
`HMSExternalTable.java`	Updated to use renamed snapshot cache methods
`test_iceberg_table_cache.groovy`	Comprehensive new test covering cache behavior with DML, DDL, and partition operations

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

suxiaogang223 · 2026-01-28T07:21:59Z

run buildall

doris-robot · 2026-01-28T07:58:22Z

TPC-H: Total hot run time: 32384 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b94d3ff7cf85ab07e06a96e72967af235065ef7f, data reload: false
------ Round 1 ----------------------------------
q1	17655	5260	5017	5017
q2	2018	309	198	198
q3	10239	1306	758	758
q4	10211	861	319	319
q5	7560	2134	1946	1946
q6	203	180	151	151
q7	905	734	599	599
q8	9268	1395	1089	1089
q9	5161	4863	4929	4863
q10	6811	1984	1544	1544
q11	527	290	285	285
q12	347	388	237	237
q13	17814	4117	3260	3260
q14	238	242	225	225
q15	889	836	822	822
q16	675	678	620	620
q17	644	793	554	554
q18	6760	6553	7347	6553
q19	1555	1030	701	701
q20	428	386	240	240
q21	2846	2357	2106	2106
q22	378	323	297	297
Total cold run time: 103132 ms
Total hot run time: 32384 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5488	5440	5657	5440
q2	263	391	272	272
q3	2416	2839	2487	2487
q4	1467	1941	1403	1403
q5	4851	4487	4588	4487
q6	230	181	143	143
q7	2055	1939	1829	1829
q8	2680	2494	2581	2494
q9	7607	7489	7863	7489
q10	2974	3259	2582	2582
q11	540	470	447	447
q12	682	776	650	650
q13	3889	4128	3384	3384
q14	276	297	270	270
q15	858	827	791	791
q16	664	708	662	662
q17	1187	1331	1338	1331
q18	7575	7490	7425	7425
q19	838	857	875	857
q20	1962	2048	1888	1888
q21	4462	4299	4170	4170
q22	573	554	529	529
Total cold run time: 53537 ms
Total hot run time: 51030 ms

doris-robot · 2026-01-28T08:15:04Z

ClickBench: Total hot run time: 28.46 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b94d3ff7cf85ab07e06a96e72967af235065ef7f, data reload: false
query1	0.06	0.05	0.04
query2	0.09	0.04	0.05
query3	0.25	0.08	0.08
query4	1.61	0.11	0.11
query5	0.26	0.24	0.24
query6	1.16	0.68	0.69
query7	0.03	0.02	0.03
query8	0.06	0.04	0.05
query9	0.56	0.49	0.50
query10	0.55	0.54	0.54
query11	0.14	0.09	0.10
query12	0.15	0.11	0.11
query13	0.64	0.62	0.63
query14	1.07	1.08	1.07
query15	0.87	0.87	0.88
query16	0.40	0.40	0.38
query17	1.13	1.12	1.15
query18	0.24	0.22	0.22
query19	2.07	2.03	2.04
query20	0.02	0.02	0.01
query21	15.44	0.26	0.14
query22	5.25	0.06	0.05
query23	15.97	0.28	0.10
query24	2.51	0.34	0.58
query25	0.10	0.06	0.09
query26	0.13	0.13	0.13
query27	0.08	0.07	0.06
query28	4.30	1.17	0.97
query29	12.60	3.95	3.16
query30	0.27	0.15	0.11
query31	2.82	0.64	0.41
query32	3.24	0.60	0.50
query33	3.28	3.21	3.26
query34	16.43	5.37	4.77
query35	4.86	4.81	4.78
query36	0.65	0.50	0.49
query37	0.12	0.07	0.07
query38	0.07	0.04	0.03
query39	0.05	0.03	0.03
query40	0.19	0.17	0.15
query41	0.09	0.04	0.03
query42	0.05	0.03	0.03
query43	0.05	0.04	0.04
Total cold run time: 99.91 s
Total hot run time: 28.46 s

hello-stephen · 2026-01-28T08:36:01Z

FE UT Coverage Report

Increment line coverage 0.00% (0/92) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-01-28T10:13:47Z

FE Regression Coverage Report

Increment line coverage 72.83% (67/92) 🎉
Increment coverage report
Complete coverage report

github-actions · 2026-01-29T05:53:33Z

PR approved by at least one committer and no changes requested.

github-actions · 2026-01-29T05:53:36Z

PR approved by anyone and no changes requested.

…table cache test cases (#59716) ### What problem does this PR solve? ## Description ### Changes This PR refactors the Iceberg metadata cache structure to improve code organization and adds comprehensive test cases for table cache behavior. ### Main Changes #### 1. Refactored IcebergMetadataCache - Introduced `IcebergTableCacheValue` to encapsulate table-related metadata - Removed redundant `snapshotListCache` and `snapshotCache` - Merged snapshot information into `IcebergTableCacheValue` with lazy loading - Simplified cache structure from 3 separate caches to 2: `tableCache` and `viewCache` **Before:** ```java private LoadingCache<IcebergMetadataCacheKey, List<Snapshot>> snapshotListCache; private LoadingCache<IcebergMetadataCacheKey, Table> tableCache; private LoadingCache<IcebergMetadataCacheKey, IcebergSnapshotCacheValue> snapshotCache; ``` **After:** ```java private LoadingCache<IcebergMetadataCacheKey, IcebergTableCacheValue> tableCache; private LoadingCache<IcebergMetadataCacheKey, View> viewCache; ``` #### 2. Lazy Loading for Snapshot Cache - Snapshot cache is now loaded on-demand through `IcebergTableCacheValue.getSnapshotCacheValue()` - Reduced unnecessary memory footprint for queries that don't require snapshot information - Snapshot information is mainly used for MTMV scenarios #### 3. Simplified Cache API - `getIcebergTable()`: Returns the Table object directly from `IcebergTableCacheValue` - `getSnapshotCache()`: Returns snapshot cache value with lazy loading - `getSnapshotList()`: Returns snapshot list from the Table object #### 4. Test Cases - Added comprehensive test case `test_iceberg_table_cache` to verify cache behavior - Tests cover both cache-enabled and cache-disabled scenarios - Validated external modifications (INSERT, DELETE, UPDATE, schema changes) are properly handled ### Benefits | Aspect | Improvement | |--------|-------------| | **Memory Usage** | Reduced by eliminating duplicate caching of snapshot information | | **Code Structure** | Cleaner with single `IcebergTableCacheValue` instead of multiple separate caches | | **Performance** | Better with lazy loading of snapshot cache only when needed | | **Maintainability** | Simpler cache management logic | ### Test Results - Added regression test: `test_iceberg_table_cache.groovy` - Tests validate cache behavior with TTL and external modifications - Verified cache invalidation works correctly with `REFRESH TABLE` - Test scenarios include: - DML operations (INSERT, DELETE, UPDATE, INSERT OVERWRITE) - Schema changes (ADD/DROP/RENAME COLUMN, ALTER COLUMN TYPE) - Partition evolution (ADD/DROP/REPLACE PARTITION FIELD) ### Related Files **Core Changes:** - `IcebergMetadataCache.java` - Refactored cache structure - `IcebergTableCacheValue.java` - New class to encapsulate table metadata - `IcebergExternalCatalog.java` - Updated cache-related configurations **Tests:** - `test_iceberg_table_cache.groovy` - Comprehensive cache behavior tests - `Suite.groovy` - Updated `getSparkIcebergContainerName()` implementation

…ure and add table cache test cases #59716 (#60341) Cherry-picked from #59716 Co-authored-by: Socrates <suyiteng@selectdb.com>

…ure and add table cache test cases apache#59716 (apache#60341) Cherry-picked from apache#59716 Co-authored-by: Socrates <suyiteng@selectdb.com>

suxiaogang223 marked this pull request as draft January 9, 2026 08:14

suxiaogang223 marked this pull request as ready for review January 12, 2026 08:48

suxiaogang223 changed the title ~~Refact meta cache~~[improvement](iceberg) Refactor Iceberg metadata cache structure and add table cache test casesJan 12, 2026

suxiaogang223 changed the title ~~[improvement](iceberg) Refactor Iceberg metadata cache structure and add table cache test cases~~[enhance](iceberg) Refactor Iceberg metadata cache structure and add table cache test casesJan 12, 2026

suxiaogang223 mentioned this pull request Jan 12, 2026
doc: Update Iceberg cache documentation for clarity and detail apache/doris-website#3277
Closed
8 tasks

morningman reviewed Jan 16, 2026
View reviewed changes

Comment threadfe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergMetadataCache.java Outdated

morningman reviewed Jan 16, 2026
View reviewed changes

Comment threadfe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergMetadataCache.java

morningman added need-back dev/4.0.x kind/need-document and removed need-back labels Jan 26, 2026

suxiaogang223 added 6 commits January 26, 2026 20:04

refactor: update IcebergMetadataCache to use IcebergTableCacheValue a…
f5015d0
…nd remove unused caches

fix build
f99c04c

remove snapshotList cache
435fccc

clean code
0994657

add cases
50374a7

fix cases
2391454

suxiaogang223 force-pushed the refact_meta_cache branch from 1574f3a to 2391454Compare January 26, 2026 12:05

fix
35ab787

Refactor IcebergUtils cache access
991bc75

morningman requested a review from Copilot January 27, 2026 14:42

Copilotstarted reviewing on behalf of morningman January 27, 2026 14:43 View session

CopilotAI reviewed Jan 27, 2026
View reviewed changes

Fix latest Iceberg schema selection
b94d3ff

morningman approved these changes Jan 29, 2026
View reviewed changes

github-actionsBot added approved Indicates a PR has been approved by one committer. reviewed labels Jan 29, 2026

zhangstar333 approved these changes Jan 29, 2026
View reviewed changes

morningman merged commit f64ab1d into apache:masterJan 29, 2026
29 of 30 checks passed

github-actionsBot mentioned this pull request Jan 29, 2026
branch-4.0: [enhance](iceberg) Refactor Iceberg metadata cache structure and add table cache test cases #59716 #60341
Merged

yiguolei pushed a commit that referenced this pull request Feb 2, 2026
branch-4.0: [enhance](iceberg) Refactor Iceberg metadata cache struct…
6f311cc
…ure and add table cache test cases #59716 (#60341) Cherry-picked from #59716 Co-authored-by: Socrates <suyiteng@selectdb.com>

yiguolei added dev/4.0.4-merged and removed dev/4.0.x labels Feb 2, 2026

morningman added kind/need-document-merged and removed kind/need-document labels Feb 11, 2026

suxiaogang223 mentioned this pull request Feb 11, 2026
[Refactor][Tracking] Unified external table metadata cache framework #60686
Closed

suxiaogang223 deleted the refact_meta_cache branch February 13, 2026 16:25

morningman mentioned this pull request Jul 3, 2026
[Discuss] Doris Roadmap 2026 #60036
Open
76 tasks

Uh oh!

Conversation

suxiaogang223 commented Jan 9, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Description

Changes

Main Changes

1. Refactored IcebergMetadataCache

2. Lazy Loading for Snapshot Cache

3. Simplified Cache API

4. Test Cases

Benefits

Test Results

Related Files

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Jan 9, 2026

Uh oh!

suxiaogang223 commented Jan 9, 2026

Uh oh!

suxiaogang223 commented Jan 12, 2026

Uh oh!

hello-stephen commented Jan 12, 2026

FE UT Coverage Report

Uh oh!

doris-robot commented Jan 12, 2026

Uh oh!

doris-robot commented Jan 12, 2026

Uh oh!

hello-stephen commented Jan 12, 2026

FE Regression Coverage Report

Uh oh!

Uh oh!

Uh oh!

suxiaogang223 commented Jan 26, 2026

Uh oh!

doris-robot commented Jan 26, 2026

Uh oh!

doris-robot commented Jan 26, 2026

Uh oh!

doris-robot commented Jan 27, 2026

Uh oh!

hello-stephen commented Jan 27, 2026

FE UT Coverage Report

Uh oh!

hello-stephen commented Jan 27, 2026

FE Regression Coverage Report

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

suxiaogang223 commented Jan 28, 2026

Uh oh!

doris-robot commented Jan 28, 2026

Uh oh!

doris-robot commented Jan 28, 2026

Uh oh!

hello-stephen commented Jan 28, 2026

FE UT Coverage Report

Uh oh!

hello-stephen commented Jan 28, 2026

FE Regression Coverage Report

Uh oh!

github-actionsBot commented Jan 29, 2026

Uh oh!

github-actionsBot commented Jan 29, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

suxiaogang223 commented Jan 9, 2026•
edited
Loading