[fix](multi-catalog) Fix multi-thread issue in hive/iceberg writer commit meta-info to fe. by kaka11chen · Pull Request #49842 · apache/doris

kaka11chen · 2025-04-07T08:55:36Z

What problem does this PR solve?

Related PR: #48709

Problem Summary:

Refer to the changes in #48709. In hive/iceberg writer, the information related to partiton_update_info and iceberg_commit_data will be submitted to FE for processing. However, there will be multiple threads accessing the related vectors, so thread-safe processing is required.

Release note

Refer to the changes in #48709. In hive/iceberg writer, the information related to partiton_update_info and iceberg_commit_data will be submitted to FE for processing. However, there will be multiple threads accessing the related vectors, so thread-safe processing is required.

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Thearas · 2025-04-07T08:55:41Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

kaka11chen · 2025-04-07T08:55:47Z

run buildall

doris-robot · 2025-04-07T09:48:59Z

TPC-H: Total hot run time: 34462 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 125fcc9f98ead392c148295ae1d102d6e54c9bee, data reload: false
------ Round 1 ----------------------------------
q1	26131	5107	5039	5039
q2	2072	278	175	175
q3	10403	1265	721	721
q4	10230	1011	562	562
q5	7562	2339	2438	2339
q6	190	168	137	137
q7	922	753	623	623
q8	9337	1375	1194	1194
q9	6854	5186	5133	5133
q10	6902	2307	1887	1887
q11	505	282	270	270
q12	368	358	227	227
q13	17775	3733	3122	3122
q14	226	230	207	207
q15	541	491	482	482
q16	614	632	603	603
q17	585	867	350	350
q18	7500	7234	7032	7032
q19	1845	964	569	569
q20	340	335	224	224
q21	4444	3649	2613	2613
q22	1039	988	953	953
Total cold run time: 116385 ms
Total hot run time: 34462 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5279	5151	5169	5151
q2	256	324	223	223
q3	2203	2701	2257	2257
q4	1469	1991	1485	1485
q5	4513	4433	4448	4433
q6	223	178	136	136
q7	1982	1945	1793	1793
q8	2667	2535	2528	2528
q9	7209	7011	7086	7011
q10	3019	3175	2707	2707
q11	598	512	486	486
q12	677	779	629	629
q13	3548	3969	3403	3403
q14	278	291	269	269
q15	532	480	477	477
q16	675	692	657	657
q17	1184	1564	1404	1404
q18	7874	7403	7572	7403
q19	843	787	880	787
q20	1935	2050	1883	1883
q21	5364	4970	4929	4929
q22	1099	1045	1046	1045
Total cold run time: 53427 ms
Total hot run time: 51096 ms

doris-robot · 2025-04-07T10:01:17Z

TPC-DS: Total hot run time: 193547 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 125fcc9f98ead392c148295ae1d102d6e54c9bee, data reload: false
query1	1427	1063	1054	1054
query2	6020	1972	1957	1957
query3	11037	4595	4579	4579
query4	52522	25342	23196	23196
query5	5218	541	463	463
query6	360	228	203	203
query7	4920	500	287	287
query8	316	253	241	241
query9	5786	2558	2577	2558
query10	450	327	270	270
query11	15137	14983	14997	14983
query12	169	107	104	104
query13	1048	514	407	407
query14	10018	6435	6579	6435
query15	219	194	183	183
query16	7129	695	494	494
query17	1097	761	641	641
query18	1521	403	313	313
query19	200	212	176	176
query20	127	118	122	118
query21	214	125	108	108
query22	4349	4560	4281	4281
query23	33999	33399	33343	33343
query24	6760	2446	2452	2446
query25	485	460	415	415
query26	696	277	154	154
query27	2389	501	333	333
query28	3345	2459	2513	2459
query29	604	568	451	451
query30	282	229	192	192
query31	840	858	800	800
query32	73	65	66	65
query33	469	355	349	349
query34	766	863	520	520
query35	821	836	756	756
query36	946	998	918	918
query37	119	104	75	75
query38	4294	4299	4297	4297
query39	1494	1452	1474	1452
query40	223	122	109	109
query41	53	57	54	54
query42	122	111	114	111
query43	498	531	501	501
query44	1324	811	821	811
query45	184	175	169	169
query46	849	1048	666	666
query47	1827	1875	1775	1775
query48	378	427	302	302
query49	729	535	421	421
query50	676	689	415	415
query51	4339	4350	4278	4278
query52	118	116	110	110
query53	243	264	190	190
query54	584	574	533	533
query55	87	91	87	87
query56	311	288	295	288
query57	1170	1219	1143	1143
query58	273	257	268	257
query59	2827	2752	2613	2613
query60	329	323	321	321
query61	168	156	165	156
query62	733	754	697	697
query63	233	200	218	200
query64	1624	1145	811	811
query65	4429	4339	4422	4339
query66	796	403	308	308
query67	15935	15717	15217	15217
query68	7113	829	520	520
query69	533	327	269	269
query70	1231	1110	1107	1107
query71	497	312	299	299
query72	5604	4802	4857	4802
query73	1378	628	347	347
query74	9107	9223	8692	8692
query75	3893	3190	2717	2717
query76	4276	1204	762	762
query77	629	379	290	290
query78	10082	10002	9203	9203
query79	5645	799	546	546
query80	652	565	441	441
query81	476	261	225	225
query82	640	129	96	96
query83	356	268	246	246
query84	292	101	85	85
query85	797	345	322	322
query86	334	280	282	280
query87	4460	4444	4427	4427
query88	3196	2195	2207	2195
query89	528	305	279	279
query90	1964	213	213	213
query91	144	144	109	109
query92	82	59	56	56
query93	2636	930	575	575
query94	688	426	301	301
query95	367	296	296	296
query96	486	568	275	275
query97	3192	3213	3130	3130
query98	220	218	214	214
query99	1428	1413	1270	1270
Total cold run time: 301654 ms
Total hot run time: 193547 ms

doris-robot · 2025-04-07T10:06:36Z

ClickBench: Total hot run time: 30.97 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 125fcc9f98ead392c148295ae1d102d6e54c9bee, data reload: false
query1	0.03	0.04	0.03
query2	0.12	0.11	0.11
query3	0.25	0.20	0.19
query4	1.59	0.20	0.12
query5	0.57	0.54	0.56
query6	1.18	0.71	0.72
query7	0.02	0.02	0.02
query8	0.04	0.04	0.03
query9	0.58	0.51	0.53
query10	0.58	0.59	0.56
query11	0.16	0.10	0.11
query12	0.15	0.12	0.12
query13	0.61	0.59	0.60
query14	2.66	2.69	2.72
query15	0.92	0.84	0.86
query16	0.38	0.39	0.38
query17	1.03	1.01	1.02
query18	0.21	0.20	0.19
query19	1.92	2.00	1.86
query20	0.01	0.01	0.01
query21	15.36	0.87	0.55
query22	0.75	1.18	0.62
query23	15.01	1.38	0.63
query24	7.50	1.51	0.77
query25	0.51	0.19	0.09
query26	0.68	0.16	0.13
query27	0.05	0.06	0.05
query28	8.98	0.89	0.43
query29	12.54	4.00	3.32
query30	0.25	0.09	0.07
query31	2.82	0.60	0.38
query32	3.23	0.55	0.46
query33	3.00	3.15	3.07
query34	15.74	5.13	4.48
query35	4.51	4.54	4.51
query36	0.65	0.49	0.48
query37	0.09	0.07	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.03
query40	0.16	0.14	0.13
query41	0.08	0.03	0.03
query42	0.03	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 105.07 s
Total hot run time: 30.97 s

doris-robot · 2025-04-07T11:57:44Z

BE UT Coverage Report

Increment line coverage 0.00% (0/30) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	52.25% (14020/26833)
Line Coverage	41.03% (121019/294952)
Region Coverage	39.77% (61594/154877)
Branch Coverage	34.45% (30843/89534)

…mmit meta-info to fe.

kaka11chen · 2025-04-07T13:04:16Z

run buildall

doris-robot · 2025-04-07T13:45:03Z

TPC-H: Total hot run time: 33855 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 10e3a7fea3a0f659dcf9a30b183bc0211e3d7e2d, data reload: false
------ Round 1 ----------------------------------
q1	25977	5008	4957	4957
q2	2070	290	198	198
q3	10381	1221	664	664
q4	10231	999	520	520
q5	7516	2381	2293	2293
q6	189	167	134	134
q7	899	721	614	614
q8	9329	1233	1066	1066
q9	6899	5170	5094	5094
q10	6817	2307	1892	1892
q11	466	285	275	275
q12	342	356	220	220
q13	17780	3674	3098	3098
q14	218	231	217	217
q15	525	488	484	484
q16	627	625	583	583
q17	588	856	353	353
q18	7422	7272	7102	7102
q19	1261	946	534	534
q20	325	314	213	213
q21	4004	3410	2409	2409
q22	1060	1005	935	935
Total cold run time: 114926 ms
Total hot run time: 33855 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5104	5075	5088	5075
q2	242	326	231	231
q3	2137	2847	2310	2310
q4	1442	1802	1391	1391
q5	4454	4423	4398	4398
q6	217	168	130	130
q7	1996	1922	1781	1781
q8	2606	2690	2544	2544
q9	7253	7141	7086	7086
q10	2965	3156	2756	2756
q11	561	497	487	487
q12	703	750	647	647
q13	3550	3915	3254	3254
q14	310	301	265	265
q15	535	476	466	466
q16	652	688	646	646
q17	1137	1533	1397	1397
q18	7727	7558	7505	7505
q19	779	838	883	838
q20	1978	2006	1819	1819
q21	5206	4972	4870	4870
q22	1107	1039	1000	1000
Total cold run time: 52661 ms
Total hot run time: 50896 ms

doris-robot · 2025-04-07T13:56:57Z

TPC-DS: Total hot run time: 193884 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 10e3a7fea3a0f659dcf9a30b183bc0211e3d7e2d, data reload: false
query1	1414	1064	1051	1051
query2	6119	1930	1939	1930
query3	10998	4440	4738	4440
query4	25076	23723	23920	23723
query5	5300	646	477	477
query6	313	212	191	191
query7	3992	498	284	284
query8	300	268	257	257
query9	8548	2595	2573	2573
query10	512	328	262	262
query11	15391	15023	14844	14844
query12	163	113	120	113
query13	1581	514	395	395
query14	8944	6154	6205	6154
query15	201	185	178	178
query16	7647	631	469	469
query17	1153	781	628	628
query18	2048	445	333	333
query19	211	193	170	170
query20	134	129	117	117
query21	212	130	117	117
query22	4489	4744	4560	4560
query23	34437	33320	33471	33320
query24	8163	2411	2438	2411
query25	504	468	398	398
query26	844	280	150	150
query27	2709	516	337	337
query28	4849	2450	2454	2450
query29	639	590	434	434
query30	284	223	190	190
query31	867	853	807	807
query32	77	64	62	62
query33	555	405	317	317
query34	803	889	520	520
query35	814	841	772	772
query36	1006	1047	917	917
query37	124	100	75	75
query38	4161	4153	4232	4153
query39	1688	1439	1438	1438
query40	209	127	113	113
query41	54	54	53	53
query42	126	109	112	109
query43	512	513	489	489
query44	1374	801	819	801
query45	180	173	171	171
query46	834	1020	627	627
query47	1870	1853	1789	1789
query48	390	426	319	319
query49	746	524	428	428
query50	674	706	428	428
query51	4202	4258	4262	4258
query52	113	109	101	101
query53	236	269	214	214
query54	587	598	526	526
query55	81	81	81	81
query56	301	304	298	298
query57	1154	1216	1129	1129
query58	265	266	269	266
query59	2753	2875	2714	2714
query60	339	320	286	286
query61	130	132	126	126
query62	756	732	686	686
query63	226	187	190	187
query64	3073	1072	714	714
query65	4491	4374	4321	4321
query66	846	423	307	307
query67	16373	15500	15481	15481
query68	9520	892	515	515
query69	564	302	267	267
query70	1157	1130	1084	1084
query71	470	314	289	289
query72	5240	4610	4595	4595
query73	689	549	344	344
query74	9222	9341	9216	9216
query75	4483	3250	2713	2713
query76	3748	1191	736	736
query77	996	370	281	281
query78	10025	10198	9242	9242
query79	1454	812	565	565
query80	680	508	454	454
query81	504	257	228	228
query82	346	128	104	104
query83	287	254	242	242
query84	295	105	86	86
query85	746	354	316	316
query86	328	308	306	306
query87	4668	4538	4317	4317
query88	2856	2217	2209	2209
query89	394	320	281	281
query90	1934	213	215	213
query91	141	145	114	114
query92	76	62	55	55
query93	1134	956	599	599
query94	677	419	288	288
query95	373	302	290	290
query96	481	560	273	273
query97	3111	3344	3144	3144
query98	224	211	202	202
query99	1399	1396	1277	1277
Total cold run time: 278387 ms
Total hot run time: 193884 ms

doris-robot · 2025-04-07T14:02:16Z

ClickBench: Total hot run time: 30.72 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 10e3a7fea3a0f659dcf9a30b183bc0211e3d7e2d, data reload: false
query1	0.04	0.04	0.02
query2	0.13	0.11	0.11
query3	0.25	0.19	0.19
query4	1.59	0.19	0.20
query5	0.60	0.57	0.58
query6	1.18	0.71	0.72
query7	0.02	0.02	0.02
query8	0.04	0.04	0.03
query9	0.60	0.52	0.51
query10	0.58	0.58	0.57
query11	0.15	0.11	0.11
query12	0.15	0.12	0.11
query13	0.61	0.60	0.60
query14	2.70	2.80	2.72
query15	0.92	0.85	0.83
query16	0.38	0.36	0.39
query17	1.01	1.01	1.04
query18	0.21	0.19	0.19
query19	1.95	1.84	1.93
query20	0.02	0.01	0.02
query21	15.37	0.99	0.56
query22	0.76	1.06	0.70
query23	15.06	1.40	0.67
query24	6.71	1.49	0.31
query25	0.41	0.14	0.07
query26	0.61	0.17	0.14
query27	0.05	0.06	0.05
query28	9.80	0.94	0.42
query29	12.55	4.02	3.30
query30	0.27	0.09	0.06
query31	3.15	0.61	0.40
query32	3.24	0.56	0.48
query33	3.00	3.02	3.06
query34	15.75	5.15	4.51
query35	4.53	4.54	4.52
query36	0.67	0.49	0.48
query37	0.08	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.03	0.03
query40	0.16	0.14	0.12
query41	0.08	0.03	0.03
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 105.53 s
Total hot run time: 30.72 s

hello-stephen · 2025-04-07T15:21:19Z

BE UT Coverage Report

Increment line coverage 0.00% (0/30) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	52.26% (14022/26833)
Line Coverage	41.05% (121074/294953)
Region Coverage	39.78% (61610/154877)
Branch Coverage	34.45% (30844/89534)

morningman

LGTM

github-actions · 2025-04-08T03:09:32Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-04-08T03:09:34Z

PR approved by anyone and no changes requested.

…mmit meta-info to fe. (#49842) ### What problem does this PR solve? Related PR: #48709 Problem Summary: Refer to the changes in #48709. In hive/iceberg writer, the information related to `partiton_update_info` and `iceberg_commit_data` will be submitted to FE for processing. However, there will be multiple threads accessing the related vectors, so thread-safe processing is required.

…mmit meta-info to fe. (apache#49842) Related PR: apache#48709 Problem Summary: Refer to the changes in apache#48709. In hive/iceberg writer, the information related to `partiton_update_info` and `iceberg_commit_data` will be submitted to FE for processing. However, there will be multiple threads accessing the related vectors, so thread-safe processing is required.

…rg writer commit meta-info to fe. #49842 (#49863) Cherry-picked from #49842 Co-authored-by: Qi Chen <chenqi@selectdb.com>

…mmit meta-info to fe. (apache#49842) ### What problem does this PR solve? Related PR: apache#48709 Problem Summary: Refer to the changes in apache#48709. In hive/iceberg writer, the information related to `partiton_update_info` and `iceberg_commit_data` will be submitted to FE for processing. However, there will be multiple threads accessing the related vectors, so thread-safe processing is required.

kaka11chen force-pushed the fix_hive_iceberg_commit_info_thread_issue branch from 125fcc9 to 56e3f5fCompare April 7, 2025 12:58

[fix](multi-catalog) Fix multi-thread issue in hive/iceberg writer co…
10e3a7f
…mmit meta-info to fe.

kaka11chen force-pushed the fix_hive_iceberg_commit_info_thread_issue branch from 56e3f5f to 10e3a7fCompare April 7, 2025 13:02

morningman added dev/2.1.x dev/3.0.x labels Apr 8, 2025

morningman approved these changes Apr 8, 2025
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Apr 8, 2025

github-actionsBot added the reviewed label Apr 8, 2025

wuwenchi approved these changes Apr 8, 2025
View reviewed changes

CalvinKirs approved these changes Apr 8, 2025
View reviewed changes

morningman merged commit d3a7e2f into apache:masterApr 8, 2025

github-actionsBot mentioned this pull request Apr 8, 2025
branch-3.0: [fix](multi-catalog) Fix multi-thread issue in hive/iceberg writer commit meta-info to fe. #49842 #49863
Merged

github-actionsBot added the dev/2.1.x-conflict label Apr 8, 2025

kaka11chen mentioned this pull request Apr 9, 2025
[fix](multi-catalog) Fix multi-thread issue in hive/iceberg writer commit meta-info to fe. #49920
Merged
16 tasks

yiguolei removed the dev/2.1.x label Apr 10, 2025

yiguolei added dev/2.1.10-merged and removed dev/2.1.x-conflict labels Apr 10, 2025

dataroaring pushed a commit that referenced this pull request Apr 11, 2025
branch-3.0: [fix](multi-catalog) Fix multi-thread issue in hive/icebe…
c5e18ef
…rg writer commit meta-info to fe. #49842 (#49863) Cherry-picked from #49842 Co-authored-by: Qi Chen <chenqi@selectdb.com>

dataroaring added dev/3.0.5-merged and removed dev/3.0.x labels Apr 11, 2025

gavinchou mentioned this pull request Apr 23, 2025
Release Notes 3.0.5 #50333
Closed

yiguolei mentioned this pull request May 13, 2025
2.1.10 Release Note #50853
Closed

Uh oh!

Conversation

kaka11chen commented Apr 7, 2025

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented Apr 7, 2025

Uh oh!

kaka11chen commented Apr 7, 2025

Uh oh!

doris-robot commented Apr 7, 2025

Uh oh!

doris-robot commented Apr 7, 2025

Uh oh!

doris-robot commented Apr 7, 2025

Uh oh!

doris-robot commented Apr 7, 2025

BE UT Coverage Report

Uh oh!

kaka11chen commented Apr 7, 2025

Uh oh!

doris-robot commented Apr 7, 2025

Uh oh!

doris-robot commented Apr 7, 2025

Uh oh!

doris-robot commented Apr 7, 2025

Uh oh!

hello-stephen commented Apr 7, 2025

BE UT Coverage Report

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Apr 8, 2025

Uh oh!

github-actionsBot commented Apr 8, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants