Skip to content

branch-3.1: [fix](iceberg)Fix the thread pool issue used for commit. (#51508) - #52572

Merged
morrySnow merged 2 commits into
apache:branch-3.1from
morningman:31_bp51508
Jul 2, 2025
Merged

branch-3.1: [fix](iceberg)Fix the thread pool issue used for commit. (#51508)#52572
morrySnow merged 2 commits into
apache:branch-3.1from
morningman:31_bp51508

Conversation

@morningman

@morningmanmorningman commented Jul 1, 2025

Copy link
Copy Markdown
Contributor

@wuwenchi@morningman
[fix](iceberg)Fix the thread pool issue used for commit. (apache#51508)
dc65e18
Problem Summary:
When Iceberg generates a new snapshot, it performs a merge operation
based on the previous snapshot. This operation reads manifest files, and
the file reading process uses a global thread pool. However, users may
have their own authentication information, which requires the use of
doAs to ensure context. Therefore, the thread pool provided by Iceberg
cannot be used.
None
@morningman

morningman commented Jul 1, 2025

Copy link
Copy Markdown
ContributorAuthor

run buildall

@hello-stephen

hello-stephen commented Jul 1, 2025

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@morningman

morningman commented Jul 1, 2025

Copy link
Copy Markdown
ContributorAuthor

run buildall

@doris-robot

doris-robot commented Jul 1, 2025

Copy link
Copy Markdown
TPC-H: Total hot run time: 39899 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit dc9586cec78e759291ecaad48a5ecdf4354453bd, data reload: false
------ Round 1 ----------------------------------
q1	17977	6975	6656	6656
q2	2072	186	159	159
q3	10659	1122	1181	1122
q4	10380	737	696	696
q5	7760	2949	2884	2884
q6	225	134	138	134
q7	979	644	606	606
q8	9375	1978	2041	1978
q9	6708	6425	6439	6425
q10	6976	2274	2333	2274
q11	471	258	260	258
q12	402	214	212	212
q13	17808	3007	2975	2975
q14	249	207	211	207
q15	514	463	469	463
q16	464	386	383	383
q17	995	597	594	594
q18	7577	6664	6642	6642
q19	1368	943	912	912
q20	487	208	211	208
q21	4041	3126	3163	3126
q22	1048	986	985	985
Total cold run time: 108535 ms
Total hot run time: 39899 ms
----- Round 2, with runtime_filter_mode=off -----
q1	6651	6578	6555	6555
q2	326	230	233	230
q3	2951	2965	3022	2965
q4	2127	1861	1851	1851
q5	5935	5945	5845	5845
q6	210	126	129	126
q7	2336	1942	1865	1865
q8	3451	3596	3585	3585
q9	8747	8959	8929	8929
q10	3570	3537	3518	3518
q11	608	489	475	475
q12	836	628	595	595
q13	7751	3241	3173	3173
q14	290	276	270	270
q15	523	467	456	456
q16	512	456	441	441
q17	1854	1635	1617	1617
q18	8126	7781	7819	7781
q19	1695	1643	1567	1567
q20	2084	1832	1855	1832
q21	5275	5072	5015	5015
q22	1117	1018	995	995
Total cold run time: 66975 ms
Total hot run time: 59686 ms

@doris-robot

doris-robot commented Jul 1, 2025

Copy link
Copy Markdown
TPC-DS: Total hot run time: 197572 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit dc9586cec78e759291ecaad48a5ecdf4354453bd, data reload: false
query1	1283	937	916	916
query2	6252	1869	1857	1857
query3	10790	4420	4383	4383
query4	33033	23929	23968	23929
query5	4789	476	460	460
query6	304	184	181	181
query7	4009	323	324	323
query8	294	231	228	228
query9	9488	2577	2579	2577
query10	496	285	258	258
query11	18223	15320	15523	15320
query12	156	104	103	103
query13	1553	433	426	426
query14	9246	6845	7170	6845
query15	265	183	180	180
query16	8022	513	496	496
query17	1563	593	596	593
query18	2139	312	316	312
query19	313	156	163	156
query20	123	112	114	112
query21	211	104	107	104
query22	4956	4636	4600	4600
query23	35030	34167	33997	33997
query24	11616	2946	2919	2919
query25	673	400	394	394
query26	1796	173	163	163
query27	2738	345	353	345
query28	7601	2187	2179	2179
query29	1023	453	471	453
query30	262	170	162	162
query31	1044	857	833	833
query32	98	58	59	58
query33	800	330	318	318
query34	967	514	556	514
query35	902	758	748	748
query36	1135	936	971	936
query37	162	68	73	68
query38	4089	4005	3966	3966
query39	1535	1481	1453	1453
query40	258	109	114	109
query41	52	49	53	49
query42	121	108	107	107
query43	521	473	475	473
query44	1256	853	844	844
query45	194	175	171	171
query46	1184	728	719	719
query47	2016	1952	1978	1952
query48	429	361	358	358
query49	992	406	432	406
query50	856	429	426	426
query51	7490	7201	7198	7198
query52	102	100	98	98
query53	264	193	188	188
query54	1329	492	491	491
query55	83	81	82	81
query56	308	261	272	261
query57	1348	1193	1232	1193
query58	246	233	213	213
query59	3185	2914	2994	2914
query60	283	270	281	270
query61	111	106	110	106
query62	901	690	722	690
query63	227	193	197	193
query64	5039	695	659	659
query65	3386	3324	3280	3280
query66	1354	318	313	313
query67	16151	15860	15485	15485
query68	4871	603	573	573
query69	442	274	261	261
query70	1120	1131	1130	1130
query71	369	255	256	255
query72	6212	4072	4005	4005
query73	765	343	355	343
query74	10766	9240	9140	9140
query75	3359	2664	2674	2664
query76	2757	980	1064	980
query77	387	280	275	275
query78	10624	9612	9536	9536
query79	1714	597	599	597
query80	1094	442	417	417
query81	557	221	226	221
query82	879	91	88	88
query83	235	146	142	142
query84	243	77	81	77
query85	1370	313	297	297
query86	431	295	263	263
query87	4332	4284	4216	4216
query88	3726	2406	2400	2400
query89	415	292	292	292
query90	1970	186	189	186
query91	141	115	106	106
query92	69	50	50	50
query93	2343	569	559	559
query94	824	299	251	251
query95	368	260	259	259
query96	694	291	288	288
query97	3299	3170	3117	3117
query98	228	195	197	195
query99	1492	1277	1300	1277
Total cold run time: 306761 ms
Total hot run time: 197572 ms

@doris-robot

doris-robot commented Jul 1, 2025

Copy link
Copy Markdown
ClickBench: Total hot run time: 30.13 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit dc9586cec78e759291ecaad48a5ecdf4354453bd, data reload: false
query1	0.03	0.03	0.04
query2	0.07	0.03	0.03
query3	0.23	0.06	0.07
query4	1.63	0.10	0.10
query5	0.53	0.51	0.51
query6	1.12	0.73	0.73
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.54	0.51	0.49
query10	0.56	0.54	0.58
query11	0.14	0.10	0.10
query12	0.14	0.11	0.11
query13	0.61	0.62	0.60
query14	0.76	0.80	0.81
query15	0.84	0.83	0.84
query16	0.36	0.39	0.38
query17	1.03	1.09	1.05
query18	0.24	0.22	0.22
query19	1.94	1.88	1.86
query20	0.01	0.01	0.02
query21	15.39	0.61	0.60
query22	2.31	2.22	1.18
query23	17.12	0.84	0.87
query24	3.21	1.18	1.67
query25	0.15	0.19	0.26
query26	0.26	0.14	0.15
query27	0.05	0.06	0.04
query28	9.52	0.53	0.50
query29	12.65	3.26	3.21
query30	0.25	0.06	0.07
query31	2.83	0.39	0.39
query32	3.24	0.47	0.46
query33	2.96	3.03	3.06
query34	16.79	4.47	4.49
query35	4.53	4.56	4.57
query36	0.67	0.47	0.47
query37	0.10	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.02	0.03
query40	0.15	0.12	0.12
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.03	0.04	0.03
Total cold run time: 103.25 s
Total hot run time: 30.13 s

@morrySnow
morrySnow merged commit 54c9d9f into apache:branch-3.1Jul 2, 2025
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnowmorrySnowmorrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@morningman@hello-stephen@doris-robot@morrySnow@wuwenchi