[opt](iceberg)Optimize the batch mode conditions for Iceberg. by wuwenchi · Pull Request #49489 · apache/doris

wuwenchi · 2025-03-26T01:25:42Z

What problem does this PR solve?

Problem Summary:

Add control variables for batch mode.
Determine whether to use batch mode by matching the number of manifest files.

Related #46398

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Thearas · 2025-03-26T01:25:47Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

wuwenchi · 2025-03-27T05:30:24Z

run buildall

wuwenchi · 2025-03-27T05:36:25Z

run buildall

wuwenchi · 2025-03-27T06:58:53Z

run buildall

doris-robot · 2025-03-27T07:34:02Z

TPC-H: Total hot run time: 33977 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f4d164bbf3457682065f81f8611222cb81c081b8, data reload: false
------ Round 1 ----------------------------------
q1	26147	5203	5027	5027
q2	2074	294	167	167
q3	10403	1263	705	705
q4	10234	1008	553	553
q5	7542	2372	2345	2345
q6	201	164	131	131
q7	928	758	620	620
q8	9351	1291	1061	1061
q9	6930	5208	5146	5146
q10	6819	2319	1891	1891
q11	475	270	258	258
q12	356	366	226	226
q13	17775	3712	3057	3057
q14	231	225	206	206
q15	547	496	498	496
q16	629	622	576	576
q17	584	868	341	341
q18	7638	7367	7007	7007
q19	1304	946	575	575
q20	340	350	193	193
q21	3976	2619	2433	2433
q22	1097	1066	963	963
Total cold run time: 115581 ms
Total hot run time: 33977 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5257	5125	5221	5125
q2	243	327	230	230
q3	2156	2661	2288	2288
q4	1483	1851	1426	1426
q5	4587	4388	4403	4388
q6	208	170	124	124
q7	1949	1917	1763	1763
q8	2593	2520	2489	2489
q9	7227	7214	7121	7121
q10	3000	3224	2785	2785
q11	596	504	486	486
q12	706	742	647	647
q13	3454	4120	3275	3275
q14	282	282	273	273
q15	541	517	513	513
q16	661	692	656	656
q17	1146	1529	1417	1417
q18	7704	7412	7421	7412
q19	831	934	1077	934
q20	1968	2002	1882	1882
q21	5157	4618	4624	4618
q22	1097	1058	971	971
Total cold run time: 52846 ms
Total hot run time: 50823 ms

doris-robot · 2025-03-27T07:45:37Z

TPC-DS: Total hot run time: 186453 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f4d164bbf3457682065f81f8611222cb81c081b8, data reload: false
query1	997	487	475	475
query2	6545	1938	1928	1928
query3	6803	222	213	213
query4	26343	23348	23261	23261
query5	4397	717	483	483
query6	306	207	209	207
query7	4609	485	283	283
query8	311	248	242	242
query9	8623	2529	2534	2529
query10	489	310	251	251
query11	15613	15496	14886	14886
query12	168	115	111	111
query13	1651	528	406	406
query14	9497	6165	6309	6165
query15	203	190	170	170
query16	7265	649	491	491
query17	1180	699	539	539
query18	1980	395	295	295
query19	189	186	167	167
query20	121	117	115	115
query21	213	120	100	100
query22	4123	4368	4255	4255
query23	33965	33200	33127	33127
query24	8436	2398	2380	2380
query25	534	460	386	386
query26	1238	274	144	144
query27	2750	506	332	332
query28	4360	2425	2375	2375
query29	766	576	422	422
query30	289	215	188	188
query31	952	861	787	787
query32	74	68	63	63
query33	556	377	358	358
query34	794	868	504	504
query35	780	803	753	753
query36	965	981	888	888
query37	125	106	81	81
query38	4188	4048	4110	4048
query39	1490	1443	1408	1408
query40	204	117	103	103
query41	57	59	54	54
query42	117	104	104	104
query43	522	513	483	483
query44	1338	801	818	801
query45	175	176	210	176
query46	881	1042	644	644
query47	1745	1831	1708	1708
query48	374	413	300	300
query49	785	498	418	418
query50	713	732	403	403
query51	4152	4146	4179	4146
query52	109	105	98	98
query53	221	261	177	177
query54	497	490	420	420
query55	85	82	81	81
query56	266	272	263	263
query57	1175	1152	1087	1087
query58	247	246	244	244
query59	2511	2629	2489	2489
query60	288	300	264	264
query61	133	127	135	127
query62	776	737	642	642
query63	224	182	184	182
query64	4347	1025	676	676
query65	4324	4245	4309	4245
query66	1165	421	319	319
query67	15888	15512	15244	15244
query68	7904	910	525	525
query69	466	302	254	254
query70	1173	1116	1083	1083
query71	467	302	259	259
query72	5523	4716	4842	4716
query73	686	628	349	349
query74	8928	9212	8824	8824
query75	3871	3248	2701	2701
query76	3576	1204	776	776
query77	799	404	309	309
query78	9899	10253	9375	9375
query79	2005	822	581	581
query80	615	596	445	445
query81	472	258	225	225
query82	436	126	100	100
query83	185	181	157	157
query84	236	99	82	82
query85	805	364	314	314
query86	352	328	281	281
query87	4358	4593	4412	4412
query88	3464	2243	2257	2243
query89	393	319	280	280
query90	1956	218	224	218
query91	146	157	113	113
query92	76	64	62	62
query93	1162	1076	600	600
query94	682	422	309	309
query95	361	276	276	276
query96	491	561	277	277
query97	3198	3303	3102	3102
query98	233	210	204	204
query99	1407	1393	1287	1287
Total cold run time: 273532 ms
Total hot run time: 186453 ms

doris-robot · 2025-03-27T07:50:57Z

ClickBench: Total hot run time: 31.3 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f4d164bbf3457682065f81f8611222cb81c081b8, data reload: false
query1	0.04	0.04	0.03
query2	0.14	0.10	0.11
query3	0.26	0.20	0.20
query4	1.59	0.19	0.20
query5	0.59	0.60	0.59
query6	1.18	0.74	0.73
query7	0.03	0.02	0.03
query8	0.04	0.03	0.03
query9	0.58	0.54	0.53
query10	0.57	0.58	0.58
query11	0.16	0.10	0.11
query12	0.16	0.11	0.12
query13	0.62	0.61	0.61
query14	2.67	2.79	2.70
query15	0.94	0.85	0.87
query16	0.40	0.39	0.39
query17	1.03	1.06	1.05
query18	0.22	0.21	0.20
query19	1.96	2.02	1.84
query20	0.01	0.01	0.01
query21	15.36	0.92	0.55
query22	0.75	1.19	0.63
query23	15.06	1.42	0.63
query24	7.78	0.86	0.65
query25	0.50	0.23	0.13
query26	0.63	0.15	0.13
query27	0.05	0.05	0.05
query28	9.91	0.96	0.44
query29	12.54	4.04	3.31
query30	0.26	0.10	0.06
query31	2.82	0.61	0.40
query32	3.22	0.56	0.47
query33	3.09	3.06	3.09
query34	15.73	5.26	4.59
query35	4.53	4.54	4.57
query36	0.67	0.51	0.49
query37	0.09	0.06	0.06
query38	0.06	0.04	0.03
query39	0.03	0.02	0.02
query40	0.16	0.13	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.04	0.03
Total cold run time: 106.58 s
Total hot run time: 31.3 s

wuwenchi · 2025-03-28T02:18:04Z

run buildall

doris-robot · 2025-03-28T02:57:36Z

TPC-H: Total hot run time: 33916 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9dae4648216ba3bbc43fbf2598591dd18af3efcf, data reload: false
------ Round 1 ----------------------------------
q1	25800	5099	5546	5099
q2	2082	280	156	156
q3	10429	1232	687	687
q4	10227	987	523	523
q5	7517	2404	2310	2310
q6	183	158	135	135
q7	922	743	605	605
q8	9329	1283	1072	1072
q9	6853	5258	5149	5149
q10	6838	2318	1870	1870
q11	478	280	258	258
q12	355	353	213	213
q13	17767	3662	3041	3041
q14	232	225	209	209
q15	524	485	485	485
q16	632	626	576	576
q17	559	849	338	338
q18	7487	7255	7118	7118
q19	1769	941	558	558
q20	306	327	192	192
q21	3830	2589	2347	2347
q22	1078	1051	975	975
Total cold run time: 115197 ms
Total hot run time: 33916 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5212	5223	5156	5156
q2	243	325	242	242
q3	2263	2743	2385	2385
q4	1424	1870	1432	1432
q5	4527	4500	4544	4500
q6	213	170	130	130
q7	1970	1905	1748	1748
q8	2582	2629	2536	2536
q9	7221	7271	7146	7146
q10	3002	3194	2739	2739
q11	598	523	486	486
q12	718	761	618	618
q13	3501	3942	3323	3323
q14	291	292	268	268
q15	520	478	468	468
q16	630	697	625	625
q17	1140	1516	1348	1348
q18	7949	7586	7523	7523
q19	809	814	859	814
q20	1909	1997	1847	1847
q21	5260	4870	4765	4765
q22	1123	1084	1018	1018
Total cold run time: 53105 ms
Total hot run time: 51117 ms

doris-robot · 2025-03-28T03:09:50Z

TPC-DS: Total hot run time: 192534 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9dae4648216ba3bbc43fbf2598591dd18af3efcf, data reload: false
query1	1389	1067	1036	1036
query2	6209	1943	1886	1886
query3	11149	4515	4545	4515
query4	54095	25648	22863	22863
query5	4986	514	459	459
query6	323	193	178	178
query7	4895	483	264	264
query8	317	254	231	231
query9	5712	2566	2559	2559
query10	427	330	255	255
query11	15198	15017	14827	14827
query12	158	114	107	107
query13	1047	539	424	424
query14	10140	6300	6393	6300
query15	193	203	174	174
query16	7139	647	500	500
query17	1100	723	608	608
query18	1549	425	317	317
query19	198	204	177	177
query20	129	125	118	118
query21	223	128	106	106
query22	4511	4417	4404	4404
query23	34060	33344	33571	33344
query24	6686	2373	2394	2373
query25	451	464	415	415
query26	726	275	145	145
query27	2441	515	330	330
query28	3169	2417	2445	2417
query29	610	595	421	421
query30	278	217	200	200
query31	877	871	767	767
query32	78	61	59	59
query33	450	376	293	293
query34	797	870	501	501
query35	799	867	772	772
query36	945	1050	911	911
query37	119	96	75	75
query38	4200	4259	4167	4167
query39	1563	1450	1424	1424
query40	209	121	114	114
query41	58	53	53	53
query42	128	120	113	113
query43	510	498	473	473
query44	1285	824	811	811
query45	182	170	165	165
query46	860	1038	655	655
query47	1862	1933	1843	1843
query48	404	426	305	305
query49	709	505	450	450
query50	705	752	407	407
query51	4277	4244	4244	4244
query52	110	104	100	100
query53	229	252	182	182
query54	516	498	426	426
query55	79	82	83	82
query56	294	268	260	260
query57	1156	1234	1140	1140
query58	254	243	239	239
query59	2788	2914	2715	2715
query60	311	297	268	268
query61	132	138	127	127
query62	718	740	677	677
query63	225	185	176	176
query64	2043	1074	724	724
query65	4320	4296	4264	4264
query66	771	378	291	291
query67	16106	15794	15541	15541
query68	7751	882	504	504
query69	534	301	261	261
query70	1235	1101	1075	1075
query71	539	282	260	260
query72	5575	4836	5059	4836
query73	1423	694	348	348
query74	9177	9209	8881	8881
query75	3816	3206	2736	2736
query76	4244	1179	755	755
query77	614	376	277	277
query78	10144	10213	9381	9381
query79	1935	814	616	616
query80	625	537	447	447
query81	480	253	215	215
query82	465	132	94	94
query83	365	171	160	160
query84	289	96	72	72
query85	816	359	302	302
query86	399	303	256	256
query87	4348	4632	4392	4392
query88	3327	2213	2202	2202
query89	390	322	283	283
query90	1950	205	205	205
query91	148	142	111	111
query92	81	62	54	54
query93	1243	1070	594	594
query94	774	413	310	310
query95	349	270	262	262
query96	481	561	270	270
query97	3185	3257	3145	3145
query98	233	207	204	204
query99	1432	1397	1259	1259
Total cold run time: 298944 ms
Total hot run time: 192534 ms

doris-robot · 2025-03-28T03:15:08Z

ClickBench: Total hot run time: 31.01 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9dae4648216ba3bbc43fbf2598591dd18af3efcf, data reload: false
query1	0.05	0.04	0.03
query2	0.12	0.11	0.10
query3	0.27	0.19	0.18
query4	1.58	0.19	0.20
query5	0.59	0.59	0.59
query6	1.19	0.73	0.70
query7	0.02	0.02	0.01
query8	0.05	0.04	0.03
query9	0.57	0.53	0.53
query10	0.63	0.58	0.56
query11	0.16	0.11	0.11
query12	0.14	0.11	0.11
query13	0.62	0.60	0.60
query14	2.64	2.69	2.77
query15	0.92	0.84	0.84
query16	0.39	0.38	0.37
query17	1.02	1.03	1.02
query18	0.21	0.20	0.21
query19	1.88	1.96	1.82
query20	0.01	0.01	0.01
query21	15.36	0.91	0.53
query22	0.76	1.08	0.64
query23	15.09	1.36	0.60
query24	7.35	0.81	0.67
query25	0.49	0.19	0.05
query26	0.51	0.18	0.14
query27	0.05	0.05	0.05
query28	9.30	0.86	0.41
query29	12.56	3.97	3.37
query30	0.26	0.08	0.06
query31	2.84	0.59	0.37
query32	3.23	0.56	0.46
query33	3.07	3.10	3.18
query34	15.83	5.09	4.55
query35	4.56	4.53	4.53
query36	0.66	0.49	0.48
query37	0.08	0.07	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.03
query40	0.17	0.13	0.13
query41	0.07	0.02	0.02
query42	0.03	0.03	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.44 s
Total hot run time: 31.01 s

morningman · 2025-03-31T08:46:23Z

+ int cnt = 0;
+ while (matchingManifest.hasNext()) {
+ cnt += matchingManifest.next().addedFilesCount();
+ if (cnt >= sessionVariable.getNumPartitionsInBatchMode()) {


It is strange to use file number to compare with partition number.

wuwenchi · 2025-04-01T06:58:14Z

run buildall

doris-robot · 2025-04-01T07:34:13Z

TPC-H: Total hot run time: 34617 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9a27f7ab7661e664cf14683ae99d5666a472e681, data reload: false
------ Round 1 ----------------------------------
q1	25782	5124	5090	5090
q2	2084	290	185	185
q3	10388	1260	717	717
q4	10222	1036	571	571
q5	7552	2301	2415	2301
q6	193	173	132	132
q7	903	775	623	623
q8	9312	1374	1219	1219
q9	6790	5212	5197	5197
q10	6867	2290	1891	1891
q11	485	292	270	270
q12	348	359	221	221
q13	17775	3685	3089	3089
q14	229	223	208	208
q15	538	502	490	490
q16	639	621	583	583
q17	623	915	393	393
q18	7569	7179	7169	7169
q19	1637	959	553	553
q20	333	339	235	235
q21	4180	3533	2490	2490
q22	1098	1022	990	990
Total cold run time: 115547 ms
Total hot run time: 34617 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5280	5171	5166	5166
q2	258	335	231	231
q3	2135	2647	2260	2260
q4	1474	1908	1544	1544
q5	4498	4504	4410	4410
q6	220	179	136	136
q7	2058	1936	1770	1770
q8	2789	2728	2665	2665
q9	7274	7062	7157	7062
q10	3028	3223	2813	2813
q11	624	514	501	501
q12	714	808	605	605
q13	3512	3946	3395	3395
q14	285	288	270	270
q15	551	509	492	492
q16	678	707	661	661
q17	1199	1706	1439	1439
q18	7676	7487	7447	7447
q19	862	838	907	838
q20	1965	1969	1859	1859
q21	5479	4857	4801	4801
q22	1143	1067	1002	1002
Total cold run time: 53702 ms
Total hot run time: 51367 ms

doris-robot · 2025-04-01T07:46:29Z

TPC-DS: Total hot run time: 192853 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9a27f7ab7661e664cf14683ae99d5666a472e681, data reload: false
query1	1410	1070	1050	1050
query2	6157	1876	1843	1843
query3	11155	4627	4641	4627
query4	54165	24773	22848	22848
query5	5034	565	445	445
query6	329	212	195	195
query7	4874	490	284	284
query8	288	252	236	236
query9	5227	2577	2571	2571
query10	369	323	274	274
query11	15113	15535	14783	14783
query12	155	115	110	110
query13	1025	521	416	416
query14	10153	6342	6392	6342
query15	213	205	172	172
query16	7063	639	525	525
query17	1097	767	626	626
query18	1549	424	324	324
query19	211	194	189	189
query20	136	134	122	122
query21	212	130	111	111
query22	4514	4500	4473	4473
query23	34095	33452	33279	33279
query24	6602	2428	2416	2416
query25	480	472	406	406
query26	723	277	152	152
query27	2351	514	336	336
query28	3555	2476	2408	2408
query29	595	571	449	449
query30	272	235	196	196
query31	866	874	771	771
query32	75	63	66	63
query33	462	372	328	328
query34	800	863	525	525
query35	802	825	783	783
query36	943	1005	881	881
query37	123	98	78	78
query38	4080	4166	4136	4136
query39	1527	1434	1407	1407
query40	216	118	101	101
query41	52	50	51	50
query42	128	107	108	107
query43	496	501	471	471
query44	1280	817	818	817
query45	177	172	169	169
query46	862	1031	636	636
query47	1851	1928	1837	1837
query48	406	448	313	313
query49	694	518	414	414
query50	667	705	417	417
query51	4326	4303	4198	4198
query52	105	107	95	95
query53	231	261	181	181
query54	590	587	507	507
query55	80	81	80	80
query56	301	316	291	291
query57	1167	1225	1107	1107
query58	270	254	261	254
query59	2797	2830	2649	2649
query60	331	342	313	313
query61	138	129	131	129
query62	771	722	662	662
query63	222	192	182	182
query64	1990	1054	710	710
query65	4437	4401	4319	4319
query66	754	432	315	315
query67	16114	15780	15471	15471
query68	7454	900	520	520
query69	541	302	262	262
query70	1218	1083	1114	1083
query71	509	321	289	289
query72	6017	4928	5007	4928
query73	1543	730	355	355
query74	9151	8788	8892	8788
query75	3921	3209	2706	2706
query76	4244	1175	760	760
query77	652	453	280	280
query78	10067	9978	9208	9208
query79	3018	807	568	568
query80	687	500	438	438
query81	489	254	224	224
query82	486	127	94	94
query83	345	255	234	234
query84	296	102	87	87
query85	770	364	423	364
query86	373	312	285	285
query87	4405	4467	4561	4467
query88	3857	2270	2228	2228
query89	404	314	276	276
query90	1817	209	229	209
query91	143	146	116	116
query92	70	58	57	57
query93	2058	971	585	585
query94	664	441	309	309
query95	363	303	296	296
query96	493	634	284	284
query97	3210	3232	3130	3130
query98	248	213	198	198
query99	1430	1402	1250	1250
Total cold run time: 301196 ms
Total hot run time: 192853 ms

doris-robot · 2025-04-01T07:51:49Z

ClickBench: Total hot run time: 31.49 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9a27f7ab7661e664cf14683ae99d5666a472e681, data reload: false
query1	0.05	0.04	0.02
query2	0.12	0.11	0.10
query3	0.26	0.19	0.19
query4	1.60	0.20	0.19
query5	0.59	0.57	0.58
query6	1.18	0.73	0.70
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.58	0.53	0.52
query10	0.57	0.58	0.59
query11	0.16	0.10	0.11
query12	0.15	0.11	0.11
query13	0.61	0.60	0.60
query14	2.66	2.78	2.72
query15	0.94	0.86	0.84
query16	0.37	0.39	0.39
query17	1.06	1.00	1.02
query18	0.21	0.21	0.20
query19	1.95	1.97	1.82
query20	0.02	0.01	0.01
query21	15.38	0.92	0.53
query22	0.77	1.17	0.77
query23	14.81	1.41	0.62
query24	7.34	1.13	1.03
query25	0.46	0.20	0.08
query26	0.61	0.16	0.15
query27	0.05	0.04	0.04
query28	9.99	0.87	0.44
query29	12.62	4.04	3.31
query30	0.26	0.09	0.06
query31	2.82	0.58	0.38
query32	3.23	0.56	0.46
query33	3.11	3.13	3.05
query34	15.76	5.19	4.52
query35	4.55	4.52	4.54
query36	0.65	0.48	0.48
query37	0.08	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.14	0.14
query41	0.08	0.03	0.03
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.03 s
Total hot run time: 31.49 s

github-actions · 2025-04-02T03:40:42Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-04-02T03:40:45Z

PR approved by anyone and no changes requested.

suxiaogang223

+1

) ### What problem does this PR solve? related pr: #49489 Problem Summary: The manifest file needs to read the "exists" type.

…#49489) ### What problem does this PR solve? Problem Summary: 1. Add control variables for batch mode. 2. Determine whether to use batch mode by matching the number of manifest files. Related apache#46398

…che#50434) ### What problem does this PR solve? related pr: apache#49489 Problem Summary: The manifest file needs to read the "exists" type.

…#49489) Problem Summary: 1. Add control variables for batch mode. 2. Determine whether to use batch mode by matching the number of manifest files. Related apache#46398

…che#50434) ### What problem does this PR solve? related pr: apache#49489 Problem Summary: The manifest file needs to read the "exists" type.

…#49489) Problem Summary: 1. Add control variables for batch mode. 2. Determine whether to use batch mode by matching the number of manifest files. Related apache#46398

…che#50434) ### What problem does this PR solve? related pr: apache#49489 Problem Summary: The manifest file needs to read the "exists" type.

…52095) bp #46398 #49025 #49489 #50434 #51185 #51694 --------- Co-authored-by: wuwenchi <wuwenchi@selectdb.com> Co-authored-by: daidai <changyuwei@selectdb.com>

wuwenchi marked this pull request as draft March 26, 2025 01:25

fix
4ee0018

wuwenchi force-pushed the opt-iceberg-batch branch from 57f1cec to 4ee0018Compare March 27, 2025 05:30

wuwenchi marked this pull request as ready for review March 27, 2025 05:30

fix
92e0b0e

fix
f4d164b

wuwenchi added 3 commits March 28, 2025 09:44

fix
9e573e5

fix
93f54a3

fix
9dae464

morningman reviewed Mar 31, 2025
View reviewed changes

morningman added dev/2.1.x-experimental dev/3.0.x-experimental labels Mar 31, 2025

fix
9a27f7a

morningman approved these changes Apr 2, 2025
View reviewed changes

github-actionsBot added approved Indicates a PR has been approved by one committer. reviewed labels Apr 2, 2025

suxiaogang223 approved these changes Apr 2, 2025
View reviewed changes

morningman merged commit 032b639 into apache:masterApr 2, 2025

wuwenchi mentioned this pull request Apr 28, 2025
[fix](iceberg)The manifest file needs to read the "exists" type. #50434
Merged
16 tasks

morningman pushed a commit that referenced this pull request Apr 29, 2025
[fix](iceberg)The manifest file needs to read the "exists" type. (#50434
a8418cc
) ### What problem does this PR solve? related pr: #49489 Problem Summary: The manifest file needs to read the "exists" type.

wuwenchi mentioned this pull request Jun 5, 2025
[opt](iceberg)support iceberg in batch mode for 3.0 #51515
Closed

This was referenced Jun 20, 2025
branch-3.1: [feat](iceberg) implement iceberg partition batch mode #52094
Closed
branch-3.1: [feat](iceberg) implement iceberg partition batch mode #52095
Merged

morningman added the dev/3.1.x label Jun 20, 2025

github-actionsBot added the dev/3.1.x-conflict label Jun 20, 2025

morrySnow added dev/3.1.0-merged and removed dev/3.1.x dev/3.1.x-conflict labels Jun 21, 2025

morningman removed the dev/3.0.x-experimental label Jul 4, 2025

Uh oh!

Conversation

wuwenchi commented Mar 26, 2025• edited by morningman Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented Mar 26, 2025

Uh oh!

wuwenchi commented Mar 27, 2025

Uh oh!

wuwenchi commented Mar 27, 2025

Uh oh!

wuwenchi commented Mar 27, 2025

Uh oh!

doris-robot commented Mar 27, 2025

Uh oh!

doris-robot commented Mar 27, 2025

Uh oh!

doris-robot commented Mar 27, 2025

Uh oh!

wuwenchi commented Mar 28, 2025

Uh oh!

doris-robot commented Mar 28, 2025

Uh oh!

doris-robot commented Mar 28, 2025

Uh oh!

doris-robot commented Mar 28, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

morningmanMar 31, 2025

Choose a reason for hiding this comment

Uh oh!

wuwenchi commented Apr 1, 2025

Uh oh!

doris-robot commented Apr 1, 2025

Uh oh!

doris-robot commented Apr 1, 2025

Uh oh!

doris-robot commented Apr 1, 2025

Uh oh!

github-actionsBot commented Apr 2, 2025

Uh oh!

github-actionsBot commented Apr 2, 2025

Uh oh!

suxiaogang223 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

wuwenchi commented Mar 26, 2025•
edited by morningman
Loading