[fix](iceberg)fix core when read a iceberg table that have schema change and equality delete. by hubgeter · Pull Request #59984 · apache/doris

hubgeter · 2026-01-17T02:53:48Z

What problem does this PR solve?

Problem Summary:
This PR addresses three major issues related to reading Iceberg tables:

When reading an Iceberg table that contains equality delete files, a crash may occur if schema changes have been applied to the affected columns.
The root cause is that the previous implementation matched columns by column name when applying equality deletes. However, Iceberg requires matching columns by Iceberg field ID to correctly identify the original column across schema evolution.
This PR updates the equality delete logic to resolve columns using Iceberg IDs instead of column names.
For Iceberg tables migrated from Hive, legacy data files do not contain Iceberg field IDs.
In this case, columns must be resolved using the table property schema.name-mapping.default.
This PR adds support for using the name mapping defined in table properties to correctly match columns when reading such data files.
Reduce Thrift metadata overhead during table reads. To optimize read performance, this PR reduces the amount of Thrift metadata transmitted when reading Iceberg tables. History_schema_info now includes only the columns required for the read, including pruning of nested fields within complex types.

Release note

Handle Iceberg data files with missing field IDs.
Fix equality delete handling for Iceberg tables with schema changes.

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2026-01-17T02:53:54Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

hubgeter · 2026-01-17T02:53:58Z

run buildall

doris-robot · 2026-01-17T03:38:49Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	79.52% (1786/2246)
Line Coverage	64.77% (31810/49114)
Region Coverage	65.51% (15854/24201)
Branch Coverage	56.02% (8419/15028)

hello-stephen · 2026-01-17T04:16:05Z

FE UT Coverage Report

Increment line coverage 0.00% (0/84) 🎉
Increment coverage report
Complete coverage report

hubgeter · 2026-01-17T16:59:32Z

run buildall

hello-stephen · 2026-01-17T17:43:19Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	79.52% (1786/2246)
Line Coverage	64.76% (31807/49114)
Region Coverage	65.48% (15847/24201)
Branch Coverage	56.04% (8422/15028)

hello-stephen · 2026-01-17T18:21:15Z

FE UT Coverage Report

Increment line coverage 0.00% (0/84) 🎉
Increment coverage report
Complete coverage report

doris-robot · 2026-01-17T18:32:27Z

TPC-H: Total hot run time: 31578 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 10863fa72fbef54a86c98b31a1ef360c86245121, data reload: false
------ Round 1 ----------------------------------
q1	17590	4214	4047	4047
q2	2027	363	232	232
q3	10121	1260	704	704
q4	10192	789	300	300
q5	7515	2026	1940	1940
q6	195	174	138	138
q7	922	776	668	668
q8	9271	1405	1201	1201
q9	5084	4707	4514	4514
q10	6784	1791	1371	1371
q11	512	309	271	271
q12	703	746	613	613
q13	17774	3834	3048	3048
q14	280	290	265	265
q15	598	514	501	501
q16	697	668	655	655
q17	666	831	468	468
q18	6834	6543	6499	6499
q19	1222	983	642	642
q20	406	361	250	250
q21	2990	2519	2289	2289
q22	1030	1020	962	962
Total cold run time: 103413 ms
Total hot run time: 31578 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4148	4069	4062	4062
q2	308	394	332	332
q3	2087	2611	2180	2180
q4	1334	1756	1332	1332
q5	4177	3988	4079	3988
q6	212	171	131	131
q7	1899	1841	1704	1704
q8	2875	2505	2537	2505
q9	7345	7343	7237	7237
q10	2508	2716	2242	2242
q11	564	481	465	465
q12	753	737	622	622
q13	3654	4124	3526	3526
q14	307	313	285	285
q15	540	510	512	510
q16	657	687	672	672
q17	1223	1483	1379	1379
q18	8225	7981	7749	7749
q19	897	829	891	829
q20	2013	2225	1850	1850
q21	4742	4686	4108	4108
q22	1125	1032	957	957
Total cold run time: 51593 ms
Total hot run time: 48665 ms

doris-robot · 2026-01-17T18:43:10Z

TPC-DS: Total hot run time: 173933 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 10863fa72fbef54a86c98b31a1ef360c86245121, data reload: false
query5	4426	618	489	489
query6	345	232	213	213
query7	4223	476	265	265
query8	344	249	237	237
query9	8746	2911	2895	2895
query10	523	369	344	344
query11	15279	15120	15078	15078
query12	170	118	122	118
query13	1261	471	378	378
query14	6357	3073	2828	2828
query14_1	2680	2672	2638	2638
query15	199	192	172	172
query16	983	499	462	462
query17	1119	656	557	557
query18	2525	442	341	341
query19	221	248	187	187
query20	118	112	110	110
query21	219	133	113	113
query22	3933	4132	3968	3968
query23	16018	15837	15263	15263
query23_1	15424	15420	15437	15420
query24	7104	1535	1187	1187
query24_1	1183	1159	1157	1157
query25	512	417	378	378
query26	1245	270	150	150
query27	2765	450	275	275
query28	4527	2176	2146	2146
query29	766	511	410	410
query30	297	244	205	205
query31	797	636	541	541
query32	84	69	76	69
query33	516	340	308	308
query34	898	883	525	525
query35	702	739	667	667
query36	902	913	840	840
query37	131	94	81	81
query38	2780	2760	2682	2682
query39	762	742	737	737
query39_1	706	705	706	705
query40	215	133	124	124
query41	63	62	63	62
query42	102	103	99	99
query43	443	462	413	413
query44	1326	742	745	742
query45	184	189	175	175
query46	846	948	573	573
query47	1416	1437	1399	1399
query48	312	323	235	235
query49	605	444	333	333
query50	611	274	200	200
query51	3831	3757	3780	3757
query52	101	106	94	94
query53	291	336	273	273
query54	280	260	270	260
query55	80	80	76	76
query56	296	302	301	301
query57	983	1022	926	926
query58	260	254	257	254
query59	2020	2173	2132	2132
query60	321	319	307	307
query61	167	141	144	141
query62	383	350	294	294
query63	308	261	263	261
query64	4955	1248	937	937
query65	3718	3746	3728	3728
query66	1426	432	324	324
query67	15538	15575	15533	15533
query68	2417	1100	775	775
query69	444	350	323	323
query70	993	941	867	867
query71	328	309	294	294
query72	5207	3189	3177	3177
query73	600	727	306	306
query74	8873	8659	8581	8581
query75	2725	2799	2458	2458
query76	2265	1058	708	708
query77	351	382	306	306
query78	9617	9734	9145	9145
query79	1064	893	566	566
query80	781	580	490	490
query81	483	261	228	228
query82	1268	149	111	111
query83	359	250	245	245
query84	248	119	90	90
query85	860	473	435	435
query86	386	300	286	286
query87	2904	2865	2790	2790
query88	3470	2577	2557	2557
query89	378	364	323	323
query90	1833	175	156	156
query91	166	159	141	141
query92	76	74	70	70
query93	954	906	536	536
query94	472	341	276	276
query95	578	348	309	309
query96	652	507	227	227
query97	2360	2383	2292	2292
query98	211	210	201	201
query99	602	581	504	504
Total cold run time: 245819 ms
Total hot run time: 173933 ms

doris-robot · 2026-01-17T18:48:14Z

ClickBench: Total hot run time: 27.86 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 10863fa72fbef54a86c98b31a1ef360c86245121, data reload: false
query1	0.05	0.05	0.05
query2	0.13	0.07	0.07
query3	0.32	0.08	0.08
query4	1.60	0.09	0.10
query5	0.27	0.24	0.25
query6	1.14	0.66	0.67
query7	0.04	0.03	0.02
query8	0.07	0.06	0.06
query9	0.58	0.53	0.51
query10	0.55	0.55	0.55
query11	0.26	0.12	0.13
query12	0.26	0.15	0.14
query13	0.63	0.62	0.61
query14	0.99	0.99	0.97
query15	0.89	0.83	0.82
query16	0.39	0.38	0.40
query17	1.03	1.06	1.05
query18	0.24	0.22	0.22
query19	1.98	1.87	1.89
query20	0.02	0.01	0.02
query21	15.40	0.29	0.24
query22	4.99	0.09	0.10
query23	15.41	0.40	0.22
query24	2.54	0.53	0.32
query25	0.11	0.09	0.09
query26	0.18	0.17	0.17
query27	0.09	0.10	0.09
query28	3.62	1.14	0.97
query29	12.53	4.05	3.22
query30	0.32	0.13	0.10
query31	2.89	0.67	0.44
query32	3.23	0.61	0.51
query33	2.99	3.13	3.02
query34	16.11	5.11	4.45
query35	4.47	4.46	4.42
query36	0.62	0.49	0.49
query37	0.24	0.08	0.09
query38	0.20	0.05	0.05
query39	0.07	0.05	0.04
query40	0.19	0.18	0.17
query41	0.14	0.06	0.06
query42	0.08	0.05	0.05
query43	0.06	0.05	0.05
Total cold run time: 97.92 s
Total hot run time: 27.86 s

hello-stephen · 2026-01-17T23:07:31Z

FE Regression Coverage Report

Increment line coverage 0.00% (0/84) 🎉
Increment coverage report
Complete coverage report

hubgeter · 2026-01-18T08:53:55Z

run buildall

doris-robot · 2026-01-18T09:37:54Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	79.52% (1786/2246)
Line Coverage	64.73% (31791/49114)
Region Coverage	65.42% (15832/24201)
Branch Coverage	55.98% (8413/15028)

hello-stephen · 2026-01-18T10:16:39Z

FE UT Coverage Report

Increment line coverage 0.00% (0/106) 🎉
Increment coverage report
Complete coverage report

doris-robot · 2026-01-18T11:08:09Z

TPC-H: Total hot run time: 31924 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5160ec02c32713ab2ccf221630a14a5dff0ff463, data reload: false
------ Round 1 ----------------------------------
q1	17640	4226	4063	4063
q2	2024	361	250	250
q3	10163	1285	750	750
q4	10281	871	314	314
q5	8492	2176	1822	1822
q6	218	167	136	136
q7	930	784	664	664
q8	9274	1365	1080	1080
q9	5230	4652	4640	4640
q10	6813	1809	1383	1383
q11	534	301	284	284
q12	728	749	580	580
q13	17792	3821	3085	3085
q14	291	315	279	279
q15	582	512	514	512
q16	744	672	628	628
q17	729	839	477	477
q18	6901	6512	6954	6512
q19	1491	1070	655	655
q20	411	396	256	256
q21	3230	2643	2499	2499
q22	1131	1112	1055	1055
Total cold run time: 105629 ms
Total hot run time: 31924 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4496	4322	4368	4322
q2	330	386	316	316
q3	2264	2811	2482	2482
q4	1519	1869	1651	1651
q5	4583	4417	4266	4266
q6	213	173	125	125
q7	1950	1947	1699	1699
q8	2581	2504	2398	2398
q9	7385	7222	7063	7063
q10	2575	2758	2111	2111
q11	510	452	432	432
q12	653	700	560	560
q13	3317	3766	3105	3105
q14	283	281	256	256
q15	525	492	484	484
q16	615	654	624	624
q17	1111	1319	1314	1314
q18	7202	7319	7275	7275
q19	849	833	818	818
q20	1894	1967	1847	1847
q21	4475	4288	4019	4019
q22	1043	1016	966	966
Total cold run time: 50373 ms
Total hot run time: 48133 ms

doris-robot · 2026-01-18T11:18:55Z

TPC-DS: Total hot run time: 174572 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5160ec02c32713ab2ccf221630a14a5dff0ff463, data reload: false
query5	4395	626	477	477
query6	364	218	202	202
query7	4213	465	264	264
query8	333	249	238	238
query9	8715	2916	2909	2909
query10	489	380	311	311
query11	15278	15203	15080	15080
query12	184	118	116	116
query13	1240	485	375	375
query14	6215	3107	2871	2871
query14_1	2717	2746	2734	2734
query15	205	199	174	174
query16	1019	500	469	469
query17	1090	671	560	560
query18	2471	449	344	344
query19	236	227	195	195
query20	124	118	118	118
query21	215	140	122	122
query22	3961	4118	4076	4076
query23	16258	15557	15377	15377
query23_1	15438	15540	15427	15427
query24	7267	1563	1181	1181
query24_1	1167	1203	1172	1172
query25	559	429	383	383
query26	1240	272	151	151
query27	2765	442	278	278
query28	4548	2166	2166	2166
query29	756	520	407	407
query30	308	240	205	205
query31	817	619	581	581
query32	88	77	74	74
query33	539	353	298	298
query34	917	907	540	540
query35	693	763	670	670
query36	843	913	838	838
query37	137	98	81	81
query38	2760	2714	2672	2672
query39	780	760	760	760
query39_1	715	721	703	703
query40	223	136	116	116
query41	72	63	64	63
query42	107	102	102	102
query43	477	446	430	430
query44	1362	757	753	753
query45	187	186	178	178
query46	867	941	578	578
query47	1361	1414	1377	1377
query48	320	330	255	255
query49	608	437	344	344
query50	625	271	207	207
query51	3826	3870	3852	3852
query52	107	109	96	96
query53	295	331	274	274
query54	299	284	260	260
query55	88	83	90	83
query56	327	307	316	307
query57	1006	991	949	949
query58	274	258	257	257
query59	2010	2164	2149	2149
query60	349	322	311	311
query61	147	150	147	147
query62	396	354	327	327
query63	302	271	267	267
query64	4919	1237	941	941
query65	3819	3699	3721	3699
query66	1444	416	311	311
query67	15455	15628	15592	15592
query68	2405	1106	771	771
query69	447	368	323	323
query70	1044	957	944	944
query71	329	313	291	291
query72	5315	2465	3280	2465
query73	601	748	339	339
query74	8768	8695	8540	8540
query75	2728	2804	2488	2488
query76	2258	1084	694	694
query77	366	380	316	316
query78	9742	9796	9122	9122
query79	1063	916	607	607
query80	1290	602	514	514
query81	513	274	239	239
query82	1330	152	113	113
query83	363	265	253	253
query84	257	122	93	93
query85	994	546	499	499
query86	398	326	305	305
query87	2859	2958	2762	2762
query88	3544	2630	2620	2620
query89	389	350	332	332
query90	1933	174	164	164
query91	166	160	137	137
query92	78	72	71	71
query93	963	904	556	556
query94	599	329	300	300
query95	595	334	316	316
query96	642	518	249	249
query97	2367	2400	2322	2322
query98	219	203	200	200
query99	579	567	498	498
Total cold run time: 247294 ms
Total hot run time: 174572 ms

doris-robot · 2026-01-18T11:24:03Z

ClickBench: Total hot run time: 27.85 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5160ec02c32713ab2ccf221630a14a5dff0ff463, data reload: false
query1	0.05	0.04	0.04
query2	0.14	0.07	0.07
query3	0.32	0.08	0.08
query4	1.61	0.10	0.10
query5	0.26	0.25	0.25
query6	1.14	0.64	0.65
query7	0.02	0.02	0.02
query8	0.07	0.06	0.05
query9	0.59	0.51	0.49
query10	0.56	0.57	0.54
query11	0.26	0.14	0.13
query12	0.26	0.15	0.14
query13	0.61	0.62	0.61
query14	0.99	0.98	0.97
query15	0.90	0.83	0.82
query16	0.39	0.38	0.41
query17	1.00	1.00	1.03
query18	0.24	0.23	0.22
query19	1.99	1.82	1.86
query20	0.02	0.01	0.01
query21	15.41	0.28	0.25
query22	4.99	0.10	0.10
query23	15.40	0.40	0.22
query24	2.35	0.51	0.31
query25	0.10	0.09	0.10
query26	0.18	0.17	0.18
query27	0.09	0.10	0.10
query28	3.56	1.13	0.98
query29	12.52	3.99	3.22
query30	0.34	0.13	0.13
query31	2.82	0.66	0.44
query32	3.24	0.62	0.50
query33	2.99	3.01	3.10
query34	16.24	5.03	4.50
query35	4.50	4.45	4.44
query36	0.62	0.51	0.48
query37	0.25	0.09	0.09
query38	0.21	0.06	0.06
query39	0.08	0.05	0.05
query40	0.20	0.19	0.17
query41	0.13	0.06	0.07
query42	0.07	0.05	0.06
query43	0.06	0.06	0.05
Total cold run time: 97.77 s
Total hot run time: 27.85 s

hello-stephen · 2026-01-18T13:49:37Z

FE Regression Coverage Report

Increment line coverage 0.00% (0/106) 🎉
Increment coverage report
Complete coverage report

hubgeter · 2026-01-18T14:36:33Z

run buildall

hello-stephen · 2026-01-18T15:07:56Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	79.52% (1786/2246)
Line Coverage	64.75% (31802/49114)
Region Coverage	65.45% (15839/24201)
Branch Coverage	56.01% (8417/15028)

hubgeter · 2026-01-19T10:02:35Z

run buildall

doris-robot · 2026-01-19T11:11:39Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	79.52% (1786/2246)
Line Coverage	64.75% (31800/49114)
Region Coverage	65.48% (15847/24201)
Branch Coverage	56.01% (8417/15028)

hello-stephen · 2026-01-19T11:52:38Z

FE UT Coverage Report

Increment line coverage 0.00% (0/106) 🎉
Increment coverage report
Complete coverage report

doris-robot · 2026-01-19T12:11:37Z

TPC-H: Total hot run time: 32029 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ac0efd13d860451af1fb3ab08b8bdbc89c2eba42, data reload: false
------ Round 1 ----------------------------------
q1	17651	4217	4082	4082
q2	2031	353	245	245
q3	10082	1271	713	713
q4	10224	858	306	306
q5	7548	2120	1843	1843
q6	192	172	140	140
q7	926	782	649	649
q8	9274	1474	1163	1163
q9	4902	4634	4549	4549
q10	6781	1834	1406	1406
q11	533	303	284	284
q12	697	731	606	606
q13	17794	3845	3056	3056
q14	294	290	274	274
q15	580	522	513	513
q16	696	682	628	628
q17	661	844	476	476
q18	7187	6608	6850	6608
q19	1352	1049	673	673
q20	402	375	257	257
q21	3353	2677	2539	2539
q22	1150	1086	1019	1019
Total cold run time: 104310 ms
Total hot run time: 32029 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4317	4274	4232	4232
q2	326	385	340	340
q3	2308	2862	2403	2403
q4	1498	1955	1545	1545
q5	4421	4321	4453	4321
q6	225	204	160	160
q7	2039	1839	1844	1839
q8	2597	2409	2316	2316
q9	7121	7331	7167	7167
q10	2444	2739	2326	2326
q11	539	461	420	420
q12	686	717	568	568
q13	3295	3812	3092	3092
q14	276	277	266	266
q15	544	493	486	486
q16	623	657	618	618
q17	1099	1237	1293	1237
q18	7282	7155	7343	7155
q19	840	796	807	796
q20	1869	1944	1805	1805
q21	4503	4255	4062	4062
q22	1057	1028	983	983
Total cold run time: 49909 ms
Total hot run time: 48137 ms

doris-robot · 2026-01-19T12:22:20Z

TPC-DS: Total hot run time: 173146 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ac0efd13d860451af1fb3ab08b8bdbc89c2eba42, data reload: false
query5	4304	599	478	478
query6	330	215	218	215
query7	4229	459	265	265
query8	360	250	234	234
query9	8729	2851	2897	2851
query10	514	406	342	342
query11	15170	15050	14807	14807
query12	177	111	114	111
query13	1262	489	383	383
query14	6259	3055	3016	3016
query14_1	2614	2601	2597	2597
query15	199	186	169	169
query16	978	494	447	447
query17	1082	638	523	523
query18	2454	414	309	309
query19	210	212	180	180
query20	118	111	107	107
query21	212	143	112	112
query22	4026	4075	3953	3953
query23	15923	15718	15248	15248
query23_1	15453	15443	15474	15443
query24	7138	1530	1165	1165
query24_1	1139	1187	1171	1171
query25	538	453	407	407
query26	1256	265	153	153
query27	2766	437	278	278
query28	4555	2167	2153	2153
query29	769	533	437	437
query30	310	235	208	208
query31	792	629	577	577
query32	80	75	76	75
query33	535	366	315	315
query34	892	858	527	527
query35	722	758	682	682
query36	866	883	788	788
query37	137	100	88	88
query38	2700	2713	2608	2608
query39	794	756	734	734
query39_1	705	697	715	697
query40	224	137	121	121
query41	71	67	68	67
query42	104	97	101	97
query43	444	438	397	397
query44	1335	763	756	756
query45	186	184	174	174
query46	837	943	574	574
query47	1437	1506	1412	1412
query48	319	341	242	242
query49	634	456	347	347
query50	652	269	205	205
query51	3739	3789	3681	3681
query52	104	108	95	95
query53	289	321	271	271
query54	295	274	266	266
query55	80	82	79	79
query56	308	307	303	303
query57	1049	1031	1000	1000
query58	287	265	259	259
query59	2105	2107	2007	2007
query60	341	340	322	322
query61	176	167	164	164
query62	392	374	309	309
query63	301	265	267	265
query64	4972	1231	925	925
query65	3812	3726	3731	3726
query66	1436	430	303	303
query67	15369	15517	15255	15255
query68	2420	1098	760	760
query69	459	352	319	319
query70	1017	948	827	827
query71	327	305	283	283
query72	5213	3113	3008	3008
query73	608	716	313	313
query74	8751	8775	8511	8511
query75	2764	2804	2446	2446
query76	2275	1063	648	648
query77	359	366	304	304
query78	9846	9925	9164	9164
query79	1093	901	579	579
query80	1278	557	472	472
query81	543	260	237	237
query82	969	143	114	114
query83	324	246	236	236
query84	252	111	89	89
query85	885	478	417	417
query86	406	287	283	283
query87	2836	2822	2765	2765
query88	3483	2580	2572	2572
query89	380	349	325	325
query90	2004	168	154	154
query91	168	156	128	128
query92	80	73	70	70
query93	983	924	542	542
query94	638	315	293	293
query95	571	387	310	310
query96	647	501	239	239
query97	2315	2374	2326	2326
query98	227	198	200	198
query99	598	590	587	587
Total cold run time: 246516 ms
Total hot run time: 173146 ms

doris-robot · 2026-01-19T12:27:28Z

ClickBench: Total hot run time: 27.65 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ac0efd13d860451af1fb3ab08b8bdbc89c2eba42, data reload: false
query1	0.05	0.05	0.04
query2	0.13	0.06	0.07
query3	0.31	0.07	0.07
query4	1.60	0.10	0.10
query5	0.27	0.24	0.25
query6	1.14	0.64	0.64
query7	0.03	0.03	0.03
query8	0.06	0.05	0.05
query9	0.57	0.50	0.49
query10	0.55	0.55	0.56
query11	0.25	0.13	0.14
query12	0.26	0.14	0.14
query13	0.63	0.61	0.60
query14	0.99	0.98	0.98
query15	0.90	0.80	0.81
query16	0.39	0.38	0.38
query17	1.03	1.07	1.06
query18	0.25	0.22	0.22
query19	1.92	1.87	1.78
query20	0.02	0.02	0.01
query21	15.40	0.28	0.25
query22	4.97	0.10	0.10
query23	15.40	0.40	0.22
query24	2.44	0.48	0.31
query25	0.10	0.09	0.10
query26	0.18	0.17	0.16
query27	0.10	0.08	0.09
query28	3.55	1.13	0.97
query29	12.50	4.03	3.24
query30	0.35	0.13	0.11
query31	2.82	0.65	0.43
query32	3.24	0.61	0.50
query33	2.95	3.05	3.04
query34	16.51	5.07	4.41
query35	4.47	4.37	4.44
query36	0.62	0.50	0.48
query37	0.24	0.09	0.09
query38	0.20	0.06	0.06
query39	0.07	0.05	0.06
query40	0.20	0.19	0.15
query41	0.13	0.06	0.07
query42	0.08	0.05	0.05
query43	0.06	0.06	0.06
Total cold run time: 97.93 s
Total hot run time: 27.65 s

hello-stephen · 2026-01-19T14:16:36Z

FE Regression Coverage Report

Increment line coverage 93.40% (99/106) 🎉
Increment coverage report
Complete coverage report

hubgeter · 2026-01-19T15:19:28Z

run buildall

doris-robot · 2026-03-11T02:56:56Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	79.45% (1798/2263)
Line Coverage	64.68% (32265/49881)
Region Coverage	65.61% (16148/24611)
Branch Coverage	56.00% (8602/15360)

doris-robot · 2026-03-11T02:58:03Z

TPC-H: Total hot run time: 27897 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 73f720ac919fc9b5631dfe4335c31b2c13e7e90f, data reload: false
------ Round 1 ----------------------------------
============================================
q1	16978	4468	4269	4269
q2	q3	10317	783	506	506
q4	4710	359	258	258
q5	7605	1203	1044	1044
q6	176	172	146	146
q7	782	824	670	670
q8	9386	1433	1345	1345
q9	4854	4725	4712	4712
q10	6247	1902	1673	1673
q11	480	278	234	234
q12	752	570	468	468
q13	18050	2919	2170	2170
q14	237	222	224	222
q15	931	801	817	801
q16	750	737	676	676
q17	711	847	422	422
q18	6050	5343	5261	5261
q19	1286	975	604	604
q20	495	507	425	425
q21	4431	1818	1702	1702
q22	359	349	289	289
Total cold run time: 95587 ms
Total hot run time: 27897 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4711	4703	4594	4594
q2	q3	3890	4378	3848	3848
q4	856	1191	761	761
q5	4032	4403	4332	4332
q6	171	189	139	139
q7	1751	1684	1528	1528
q8	2454	2716	2572	2572
q9	7597	7272	7401	7272
q10	3830	3936	3593	3593
q11	540	464	425	425
q12	506	600	488	488
q13	2787	3125	2476	2476
q14	308	300	273	273
q15	882	826	805	805
q16	697	748	712	712
q17	1173	1450	1415	1415
q18	7380	6819	6772	6772
q19	893	880	895	880
q20	2104	2208	2030	2030
q21	3942	3451	3345	3345
q22	459	427	384	384
Total cold run time: 50963 ms
Total hot run time: 48644 ms

doris-robot · 2026-03-11T03:08:01Z

TPC-DS: Total hot run time: 153398 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 73f720ac919fc9b5631dfe4335c31b2c13e7e90f, data reload: false
query5	4317	629	512	512
query6	314	226	214	214
query7	4216	469	260	260
query8	321	245	247	245
query9	8694	2742	2716	2716
query10	482	373	365	365
query11	7418	5932	5547	5547
query12	183	125	128	125
query13	1271	439	359	359
query14	5656	3825	3577	3577
query14_1	2837	2812	2814	2812
query15	209	193	179	179
query16	988	498	463	463
query17	1110	711	623	623
query18	2445	454	355	355
query19	213	208	185	185
query20	135	130	135	130
query21	226	144	125	125
query22	4775	4946	4722	4722
query23	16592	16102	15825	15825
query23_1	15779	16059	15968	15968
query24	7269	1659	1265	1265
query24_1	1292	1383	1281	1281
query25	560	501	431	431
query26	1284	268	154	154
query27	2820	476	286	286
query28	4450	1852	1849	1849
query29	823	571	467	467
query30	316	242	210	210
query31	1386	1271	1217	1217
query32	81	72	73	72
query33	505	327	291	291
query34	948	908	564	564
query35	640	671	643	643
query36	1082	1127	1009	1009
query37	130	95	85	85
query38	2947	2994	2892	2892
query39	983	875	847	847
query39_1	808	841	824	824
query40	227	151	137	137
query41	62	59	58	58
query42	297	295	301	295
query43	239	255	218	218
query44 query45	199	189	181	181
query46	883	981	606	606
query47	2125	2137	2056	2056
query48	317	319	227	227
query49	632	463	376	376
query50	684	280	210	210
query51	4133	4112	4141	4112
query52	288	294	274	274
query53	292	340	280	280
query54	298	271	295	271
query55	94	83	88	83
query56	320	312	296	296
query57	1371	1342	1281	1281
query58	330	275	267	267
query59	1375	1432	1309	1309
query60	349	327	321	321
query61	148	141	142	141
query62	614	585	541	541
query63	310	274	273	273
query64	4939	1278	986	986
query65 query66	1444	466	357	357
query67	16330	16558	16335	16335
query68 query69	390	305	284	284
query70	1011	932	968	932
query71	347	312	291	291
query72	2813	2656	2624	2624
query73	528	567	315	315
query74	10065	9979	9723	9723
query75	2855	2773	2441	2441
query76	2152	1040	670	670
query77	368	391	312	312
query78	11102	11391	10677	10677
query79	1179	812	604	604
query80	1371	645	524	524
query81	571	287	252	252
query82	964	152	113	113
query83	331	265	246	246
query84	248	132	99	99
query85	898	487	444	444
query86	434	302	277	277
query87	3189	3155	3092	3092
query88	3516	2629	2642	2629
query89	423	362	344	344
query90	2047	178	171	171
query91	168	166	134	134
query92	84	75	67	67
query93	1003	825	497	497
query94	655	313	283	283
query95	580	334	380	334
query96	641	519	225	225
query97	2492	2485	2436	2436
query98	228	219	219	219
query99	1008	985	913	913
Total cold run time: 232974 ms
Total hot run time: 153398 ms

hello-stephen · 2026-03-11T03:30:42Z

FE UT Coverage Report

Increment line coverage 70.75% (75/106) 🎉
Increment coverage report
Complete coverage report

doris-robot · 2026-03-11T03:34:55Z

BE UT Coverage Report

Increment line coverage 11.11% (67/603) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	52.56% (19650/37383)
Line Coverage	36.14% (183168/506876)
Region Coverage	32.28% (141536/438450)
Branch Coverage	33.43% (61739/184657)

doris-robot · 2026-03-11T04:54:30Z

TPC-H: Total hot run time: 27694 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 73f720ac919fc9b5631dfe4335c31b2c13e7e90f, data reload: false
------ Round 1 ----------------------------------
============================================
q1	16775	4438	4289	4289
q2	q3	10304	754	514	514
q4	4653	350	251	251
q5	7640	1191	1044	1044
q6	173	184	147	147
q7	788	830	658	658
q8	9416	1466	1300	1300
q9	4774	4760	4705	4705
q10	6240	1878	1634	1634
q11	450	257	246	246
q12	712	584	471	471
q13	18033	2947	2221	2221
q14	227	232	216	216
q15	901	794	797	794
q16	729	729	674	674
q17	698	845	418	418
q18	5866	5291	5292	5291
q19	1111	999	644	644
q20	502	514	387	387
q21	4373	1822	1481	1481
q22	453	314	309	309
Total cold run time: 94818 ms
Total hot run time: 27694 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4789	4538	4567	4538
q2	q3	3897	4308	3830	3830
q4	914	1187	756	756
q5	4044	4359	4316	4316
q6	178	177	144	144
q7	1730	1627	1543	1543
q8	2488	2732	2617	2617
q9	7542	7351	7360	7351
q10	3773	4000	3612	3612
q11	515	496	427	427
q12	501	590	439	439
q13	2648	3078	2408	2408
q14	299	307	280	280
q15	821	797	798	797
q16	709	767	760	760
q17	1234	1548	1369	1369
q18	7247	6752	6679	6679
q19	884	844	881	844
q20	2068	2119	2134	2119
q21	3875	3499	3388	3388
q22	467	420	396	396
Total cold run time: 50623 ms
Total hot run time: 48613 ms

doris-robot · 2026-03-11T05:04:30Z

TPC-DS: Total hot run time: 153255 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 73f720ac919fc9b5631dfe4335c31b2c13e7e90f, data reload: false
query5	4330	637	529	529
query6	316	224	212	212
query7	4218	475	262	262
query8	326	243	232	232
query9	8701	2743	2724	2724
query10	521	374	366	366
query11	7273	5870	5559	5559
query12	182	126	124	124
query13	1268	442	356	356
query14	5654	3840	3575	3575
query14_1	2810	2837	2822	2822
query15	212	196	173	173
query16	1000	496	455	455
query17	1113	737	638	638
query18	2441	445	353	353
query19	209	206	186	186
query20	135	131	133	131
query21	232	146	130	130
query22	5183	4971	4849	4849
query23	16524	15905	15949	15905
query23_1	15924	15734	15923	15734
query24	7064	1792	1279	1279
query24_1	1293	1314	1302	1302
query25	636	619	418	418
query26	1612	271	156	156
query27	2880	499	314	314
query28	4743	1858	1866	1858
query29	831	562	474	474
query30	313	245	214	214
query31	1381	1284	1211	1211
query32	81	85	76	76
query33	512	325	284	284
query34	926	932	551	551
query35	646	671	578	578
query36	1100	1147	1033	1033
query37	135	89	86	86
query38	2983	2868	2873	2868
query39	895	911	844	844
query39_1	822	818	854	818
query40	231	149	140	140
query41	64	61	57	57
query42	296	299	300	299
query43	234	242	234	234
query44 query45	198	187	183	183
query46	881	983	603	603
query47	2093	2122	2020	2020
query48	314	310	235	235
query49	634	460	385	385
query50	673	280	204	204
query51	4068	4015	4039	4015
query52	284	291	279	279
query53	291	337	284	284
query54	298	269	268	268
query55	95	87	87	87
query56	329	322	312	312
query57	1389	1346	1300	1300
query58	287	285	273	273
query59	1322	1411	1302	1302
query60	335	336	321	321
query61	153	145	142	142
query62	639	586	543	543
query63	308	275	273	273
query64	4896	1280	986	986
query65 query66	1432	466	366	366
query67	16485	16589	16371	16371
query68 query69	398	328	298	298
query70	961	975	986	975
query71	345	315	300	300
query72	2939	2887	2531	2531
query73	527	537	330	330
query74	10011	9929	9764	9764
query75	2827	2746	2453	2453
query76	2114	1027	662	662
query77	361	396	300	300
query78	11194	11368	10643	10643
query79	1891	809	595	595
query80	1445	603	551	551
query81	567	281	246	246
query82	960	152	119	119
query83	325	272	249	249
query84	252	122	98	98
query85	903	476	425	425
query86	418	307	295	295
query87	3157	3100	3020	3020
query88	3589	2673	2644	2644
query89	427	372	340	340
query90	2032	176	178	176
query91	167	158	133	133
query92	84	74	71	71
query93	1141	809	496	496
query94	635	324	297	297
query95	588	343	325	325
query96	643	515	232	232
query97	2520	2517	2387	2387
query98	234	226	218	218
query99	987	938	833	833
Total cold run time: 235116 ms
Total hot run time: 153255 ms

hello-stephen · 2026-03-11T05:14:57Z

BE Regression && UT Coverage Report

Increment line coverage 71.14% (429/603) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	73.19% (26791/36605)
Line Coverage	56.47% (285348/505337)
Region Coverage	53.78% (238012/442598)
Branch Coverage	55.49% (102784/185233)

hello-stephen · 2026-03-11T08:42:06Z

BE Regression && UT Coverage Report

Increment line coverage 71.14% (429/603) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	73.18% (26788/36605)
Line Coverage	56.46% (285313/505337)
Region Coverage	53.76% (237939/442598)
Branch Coverage	55.48% (102769/185233)

github-actions · 2026-03-11T17:22:22Z

PR approved by at least one committer and no changes requested.

kaka11chen

LGTM

…nge and equality delete. (apache#59984) Problem Summary: This PR addresses three major issues related to reading Iceberg tables: 1. When reading an Iceberg table that contains equality delete files, a crash may occur if schema changes have been applied to the affected columns. The root cause is that the previous implementation matched columns by column name when applying equality deletes. However, Iceberg requires matching columns by Iceberg field ID to correctly identify the original column across schema evolution. This PR updates the equality delete logic to resolve columns using Iceberg IDs instead of column names. 2. For Iceberg tables migrated from Hive, legacy data files do not contain Iceberg field IDs. In this case, columns must be resolved using the table property `schema.name-mapping.default.` This PR adds support for using the name mapping defined in table properties to correctly match columns when reading such data files. 3. Reduce Thrift metadata overhead during table reads. To optimize read performance, this PR reduces the amount of Thrift metadata transmitted when reading Iceberg tables. `History_schema_info` now includes only the columns required for the read, including pruning of nested fields within complex types. Handle Iceberg data files with missing field IDs. Fix equality delete handling for Iceberg tables with schema changes.

… schema change and equality delete. (#59984) (#61348) bp #59984 Problem Summary: This PR addresses three major issues related to reading Iceberg tables: 1. When reading an Iceberg table that contains equality delete files, a crash may occur if schema changes have been applied to the affected columns. The root cause is that the previous implementation matched columns by column name when applying equality deletes. However, Iceberg requires matching columns by Iceberg field ID to correctly identify the original column across schema evolution. This PR updates the equality delete logic to resolve columns using Iceberg IDs instead of column names. 2. For Iceberg tables migrated from Hive, legacy data files do not contain Iceberg field IDs. In this case, columns must be resolved using the table property `schema.name-mapping.default.` This PR adds support for using the name mapping defined in table properties to correctly match columns when reading such data files. 3. Reduce Thrift metadata overhead during table reads. To optimize read performance, this PR reduces the amount of Thrift metadata transmitted when reading Iceberg tables. `History_schema_info` now includes only the columns required for the read, including pruning of nested fields within complex types. Handle Iceberg data files with missing field IDs. Fix equality delete handling for Iceberg tables with schema changes. ### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test  - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason  - Behavior changed: - [ ] No. - [ ] Yes.  - Does this need documentation? - [ ] No. - [ ] Yes.  ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label

…nge and equality delete. (apache#59984) Problem Summary: This PR addresses three major issues related to reading Iceberg tables: 1. When reading an Iceberg table that contains equality delete files, a crash may occur if schema changes have been applied to the affected columns. The root cause is that the previous implementation matched columns by column name when applying equality deletes. However, Iceberg requires matching columns by Iceberg field ID to correctly identify the original column across schema evolution. This PR updates the equality delete logic to resolve columns using Iceberg IDs instead of column names. 2. For Iceberg tables migrated from Hive, legacy data files do not contain Iceberg field IDs. In this case, columns must be resolved using the table property `schema.name-mapping.default.` This PR adds support for using the name mapping defined in table properties to correctly match columns when reading such data files. 3. Reduce Thrift metadata overhead during table reads. To optimize read performance, this PR reduces the amount of Thrift metadata transmitted when reading Iceberg tables. `History_schema_info` now includes only the columns required for the read, including pruning of nested fields within complex types. Handle Iceberg data files with missing field IDs. Fix equality delete handling for Iceberg tables with schema changes.

… schema change and equality delete. (#59984) (#61689) bp #59984 Problem Summary: This PR addresses three major issues related to reading Iceberg tables: 1. When reading an Iceberg table that contains equality delete files, a crash may occur if schema changes have been applied to the affected columns. The root cause is that the previous implementation matched columns by column name when applying equality deletes. However, Iceberg requires matching columns by Iceberg field ID to correctly identify the original column across schema evolution. This PR updates the equality delete logic to resolve columns using Iceberg IDs instead of column names. 2. For Iceberg tables migrated from Hive, legacy data files do not contain Iceberg field IDs. In this case, columns must be resolved using the table property `schema.name-mapping.default.` This PR adds support for using the name mapping defined in table properties to correctly match columns when reading such data files. 3. Reduce Thrift metadata overhead during table reads. To optimize read performance, this PR reduces the amount of Thrift metadata transmitted when reading Iceberg tables. `History_schema_info` now includes only the columns required for the read, including pruning of nested fields within complex types. Handle Iceberg data files with missing field IDs. Fix equality delete handling for Iceberg tables with schema changes.

hubgeter force-pushed the iceberg_eq_core branch from 54149f6 to 10863faCompare January 17, 2026 16:59

morningman added the dev/4.0.x label Jan 19, 2026

morningman added the dev/4.1.x label Mar 11, 2026

morningman approved these changes Mar 11, 2026
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Mar 11, 2026

kaka11chen approved these changes Mar 12, 2026
View reviewed changes

hubgeter mentioned this pull request Mar 12, 2026
[Bug] BE Crash with SIGSEGV in ByteArrayDictDecoder and std::out_of_range during Iceberg table scanning/loading #61225
Open
3 tasks

morningman merged commit 431ac05 into apache:masterMar 12, 2026
28 of 31 checks passed

github-actionsBot added the dev/4.0.x-conflict label Mar 12, 2026

hubgeter mentioned this pull request Mar 15, 2026
branch-4.0:[fix](iceberg)fix core when read a iceberg table that have schema change and equality delete. (#59984) #61348
Merged
16 tasks

yiguolei added dev/4.0.5-merged and removed dev/4.0.x dev/4.0.x-conflict labels Mar 17, 2026

hubgeter mentioned this pull request Mar 25, 2026
branch-4.1:[fix](iceberg)fix core when read a iceberg table that have schema change and equality delete. (#59984) #61689
Merged

yiguolei added dev/4.1.0-merged and removed dev/4.1.x labels Mar 26, 2026

yiguolei mentioned this pull request Mar 27, 2026
4.0.5 Release Notes #61817
Closed

Uh oh!

Conversation

hubgeter commented Jan 17, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Jan 17, 2026

Uh oh!

hubgeter commented Jan 17, 2026

Uh oh!

doris-robot commented Jan 17, 2026

Cloud UT Coverage Report

Uh oh!

hello-stephen commented Jan 17, 2026

FE UT Coverage Report

Uh oh!

hubgeter commented Jan 17, 2026

Uh oh!

hello-stephen commented Jan 17, 2026

Cloud UT Coverage Report

Uh oh!

hello-stephen commented Jan 17, 2026

FE UT Coverage Report

Uh oh!

doris-robot commented Jan 17, 2026

Uh oh!

doris-robot commented Jan 17, 2026

Uh oh!

doris-robot commented Jan 17, 2026

Uh oh!

hello-stephen commented Jan 17, 2026

FE Regression Coverage Report

Uh oh!

hubgeter commented Jan 18, 2026

Uh oh!

doris-robot commented Jan 18, 2026

Cloud UT Coverage Report

Uh oh!

hello-stephen commented Jan 18, 2026

FE UT Coverage Report

Uh oh!

doris-robot commented Jan 18, 2026

Uh oh!

doris-robot commented Jan 18, 2026

Uh oh!

doris-robot commented Jan 18, 2026

Uh oh!

hello-stephen commented Jan 18, 2026

FE Regression Coverage Report

Uh oh!

hubgeter commented Jan 18, 2026

Uh oh!

hello-stephen commented Jan 18, 2026

Cloud UT Coverage Report

Uh oh!

hubgeter commented Jan 19, 2026

Uh oh!

doris-robot commented Jan 19, 2026

Cloud UT Coverage Report

Uh oh!

hello-stephen commented Jan 19, 2026

FE UT Coverage Report

Uh oh!

doris-robot commented Jan 19, 2026

Uh oh!

doris-robot commented Jan 19, 2026

Uh oh!

doris-robot commented Jan 19, 2026

Uh oh!

hello-stephen commented Jan 19, 2026

FE Regression Coverage Report

Uh oh!

hubgeter commented Jan 19, 2026

Uh oh!

doris-robot commented Mar 11, 2026

Cloud UT Coverage Report

Uh oh!

hubgeter commented Jan 17, 2026•
edited
Loading