Skip to content

branch-3.0: [fix](create-resource) fix potential concurrent modification exception when creating resource (#50356, #50615) - #50533

Merged
dataroaring merged 2 commits into
apache:branch-3.0from
morningman:30_bp50356
May 15, 2025
Merged

branch-3.0: [fix](create-resource) fix potential concurrent modification exception when creating resource (#50356, #50615)#50533
dataroaring merged 2 commits into
apache:branch-3.0from
morningman:30_bp50356

Conversation

@morningman

@morningmanmorningman commented Apr 29, 2025•
edited
Loading

Copy link
Copy Markdown
Contributor

@hello-stephen

hello-stephen commented Apr 29, 2025

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@morningman

morningman commented Apr 29, 2025

Copy link
Copy Markdown
ContributorAuthor

run buildall

@morningmanmorningman changed the title [fix](create-resource) fix potential concurrent modification exception when creating resource (#50356)[fix](create-resource) fix potential concurrent modification exception when creating resource (#50356, #50615)May 7, 2025
@morningman

morningman commented May 7, 2025

Copy link
Copy Markdown
ContributorAuthor

run buildall

gavinchou
gavinchou previously approved these changes May 7, 2025
View reviewed changes
@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label May 7, 2025
@github-actions

github-actionsBot commented May 7, 2025

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actions

github-actionsBot commented May 7, 2025

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@morningman

morningman commented May 7, 2025

Copy link
Copy Markdown
ContributorAuthor

run buildall

@doris-robot

doris-robot commented May 7, 2025

Copy link
Copy Markdown
TPC-H: Total hot run time: 40004 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6c52b12e931315e484b90af0e08df1791c0d729b, data reload: false
------ Round 1 ----------------------------------
q1	17599	6791	6622	6622
q2	2069	180	170	170
q3	10614	1047	1191	1047
q4	10564	692	719	692
q5	7735	2815	2828	2815
q6	223	136	131	131
q7	975	610	605	605
q8	9347	1914	1954	1914
q9	6582	6366	6413	6366
q10	7026	2237	2286	2237
q11	459	246	259	246
q12	398	213	207	207
q13	17802	2945	2971	2945
q14	235	203	221	203
q15	483	455	468	455
q16	688	584	584	584
q17	973	606	602	602
q18	7235	6817	6694	6694
q19	1401	1053	1031	1031
q20	470	206	212	206
q21	4168	3241	3321	3241
q22	1077	991	1001	991
Total cold run time: 108123 ms
Total hot run time: 40004 ms
----- Round 2, with runtime_filter_mode=off -----
q1	6567	6599	6553	6553
q2	323	228	231	228
q3	2892	2747	2864	2747
q4	2051	1836	1799	1799
q5	5765	5713	5762	5713
q6	212	130	134	130
q7	2242	1896	1863	1863
q8	3385	3614	3540	3540
q9	9025	8990	8958	8958
q10	3601	3547	3540	3540
q11	603	504	500	500
q12	803	636	634	634
q13	8053	3196	3166	3166
q14	296	261	266	261
q15	507	469	465	465
q16	667	666	660	660
q17	1835	1732	1594	1594
q18	8183	7698	7726	7698
q19	1661	1478	1520	1478
q20	2062	1906	1858	1858
q21	5489	5238	5192	5192
q22	1140	1048	1031	1031
Total cold run time: 67362 ms
Total hot run time: 59608 ms

@doris-robot

doris-robot commented May 7, 2025

Copy link
Copy Markdown
TPC-DS: Total hot run time: 196614 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6c52b12e931315e484b90af0e08df1791c0d729b, data reload: false
query1	1328	915	880	880
query2	6309	2011	1977	1977
query3	10824	4103	4212	4103
query4	61322	29912	23670	23670
query5	5273	481	450	450
query6	394	180	181	180
query7	5477	307	302	302
query8	307	225	228	225
query9	8438	2592	2555	2555
query10	466	254	257	254
query11	17480	15204	15728	15204
query12	160	105	102	102
query13	1425	435	424	424
query14	10460	7167	6619	6619
query15	214	189	177	177
query16	6970	479	486	479
query17	1157	596	578	578
query18	1651	327	316	316
query19	210	159	160	159
query20	121	117	108	108
query21	206	102	106	102
query22	4717	4546	4733	4546
query23	35285	33949	33791	33791
query24	6180	2922	2937	2922
query25	550	417	419	417
query26	651	173	168	168
query27	1852	379	349	349
query28	4450	2466	2475	2466
query29	712	472	438	438
query30	236	163	165	163
query31	1033	818	796	796
query32	73	54	56	54
query33	411	270	280	270
query34	916	508	509	508
query35	868	765	731	731
query36	1069	944	957	944
query37	123	64	70	64
query38	4038	3974	4001	3974
query39	1539	1505	1460	1460
query40	215	97	95	95
query41	52	50	51	50
query42	114	100	98	98
query43	515	483	491	483
query44	1140	805	822	805
query45	188	178	179	178
query46	1147	711	728	711
query47	2042	1956	1965	1956
query48	470	383	385	383
query49	754	403	401	401
query50	821	435	430	430
query51	7470	7276	7299	7276
query52	102	92	93	92
query53	256	179	187	179
query54	569	454	449	449
query55	84	82	84	82
query56	273	263	259	259
query57	1277	1186	1158	1158
query58	224	209	228	209
query59	3151	2976	3182	2976
query60	287	272	284	272
query61	119	113	123	113
query62	786	677	680	677
query63	215	180	183	180
query64	1323	673	651	651
query65	3277	3190	3190	3190
query66	650	294	308	294
query67	16009	15471	15702	15471
query68	4141	569	573	569
query69	424	262	258	258
query70	1210	1131	1070	1070
query71	325	254	243	243
query72	6279	3939	4061	3939
query73	748	348	371	348
query74	10345	9273	9330	9273
query75	3364	2639	2661	2639
query76	1968	1080	1055	1055
query77	498	274	267	267
query78	10770	9789	9552	9552
query79	1803	598	602	598
query80	1085	429	427	427
query81	518	245	237	237
query82	1253	87	85	85
query83	167	136	146	136
query84	280	77	77	77
query85	948	302	285	285
query86	347	284	295	284
query87	4419	4279	4230	4230
query88	3878	2355	2350	2350
query89	423	292	287	287
query90	1899	184	182	182
query91	184	143	146	143
query92	67	48	51	48
query93	1954	539	545	539
query94	738	287	274	274
query95	348	252	255	252
query96	620	285	278	278
query97	3296	3168	3163	3163
query98	210	203	200	200
query99	1575	1283	1310	1283
Total cold run time: 314509 ms
Total hot run time: 196614 ms

@doris-robot

doris-robot commented May 7, 2025

Copy link
Copy Markdown
ClickBench: Total hot run time: 33.12 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6c52b12e931315e484b90af0e08df1791c0d729b, data reload: false
query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.23	0.06	0.06
query4	1.64	0.10	0.11
query5	0.53	0.51	0.51
query6	1.14	0.73	0.74
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.55	0.51	0.50
query10	0.57	0.55	0.55
query11	0.14	0.10	0.10
query12	0.14	0.11	0.11
query13	0.61	0.59	0.59
query14	2.71	2.72	2.73
query15	0.90	0.82	0.82
query16	0.38	0.38	0.37
query17	1.06	1.04	1.06
query18	0.23	0.22	0.22
query19	1.92	1.80	2.06
query20	0.01	0.01	0.01
query21	15.44	0.59	0.59
query22	2.90	2.85	2.15
query23	16.94	0.99	0.82
query24	3.08	1.71	2.13
query25	0.19	0.04	0.16
query26	0.70	0.13	0.14
query27	0.04	0.04	0.03
query28	8.66	0.54	0.49
query29	12.55	3.26	3.26
query30	0.24	0.06	0.06
query31	2.85	0.38	0.38
query32	3.26	0.47	0.45
query33	2.95	3.02	2.99
query34	17.26	4.45	4.42
query35	4.54	4.49	4.50
query36	0.66	0.48	0.50
query37	0.09	0.06	0.06
query38	0.04	0.03	0.04
query39	0.03	0.02	0.02
query40	0.17	0.13	0.13
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 105.67 s
Total hot run time: 33.12 s

@morningman

morningman commented May 7, 2025

Copy link
Copy Markdown
ContributorAuthor

run buildall

@morningmanmorningman changed the title [fix](create-resource) fix potential concurrent modification exception when creating resource (#50356, #50615)branch-3.0: [fix](create-resource) fix potential concurrent modification exception when creating resource (#50356, #50615)May 7, 2025
morningmanand others added 2 commits May 14, 2025 10:30
@morningman
[fix](create-resource) fix potential concurrent modification exceptio…
1c5ff56
…n when creating resource (apache#50356)
Related PR: apache#48424
Problem Summary:
The `CreateResourceCommandTest` may throw concurrent modification
exception.
This is because in `JdbcResource`, the origin property map may be
modified.
This PR changes；
1. Change the origin property map to ImmutableMap, to avoid potential
exception.
2. Copy the map to the jdbc resource to avoid modifying the origin map
@yagagagaga@morningman
[fix](vault) fix UnsupportedOperationException if no use_path_style…
d3608b4
… in CreateStorageVaultStmt (apache#50615)
Related PR: apache#50356apache#45155
Problem Summary:
```sql
CREATE STORAGE VAULT test PROPERTIES
(
"type"="S3",
"s3.endpoint"="cos.ap-guangzhou.myqcloud.com",
"s3.access_key" = "*****",
"s3.secret_key" = "*****",
"s3.region" = "ap-guangzhou",
"s3.root.path" = "prefix",
"s3.bucket" = "test",
"provider" = "COS"
)
```
throw Exception:
```java
2025-05-06 15:05:16,395 WARN (mysql-nio-pool-0|235) [StmtExecutor.handleDdlStmt():3216] DDL statement(CREATE STORAGE VAULT test PROPERTIES
(
"type"="S3",
"s3.endpoint"="cos.ap-guangzhou.myqcloud.com",
"s3.access_key" = "*****",
"s3.secret_key" = "*****",
"s3.region" = "ap-guangzhou",
"s3.root.path" = "prefix",
"s3.bucket" = "test",
"provider" = "COS"
)) process failed.
java.lang.UnsupportedOperationException: null
at com.google.common.collect.ImmutableMap.put(ImmutableMap.java:814) ~[guava-33.2.1-jre.jar:?]
at org.apache.doris.catalog.StorageVault.getStorageVaultInstance(StorageVault.java:151) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.catalog.StorageVault.fromStmt(StorageVault.java:103) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.catalog.StorageVaultMgr.createStorageVaultResource(StorageVaultMgr.java:69) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.DdlExecutor.execute(DdlExecutor.java:440) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.handleDdlStmt(StmtExecutor.java:3195) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.executeByLegacy(StmtExecutor.java:1124) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:644) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.queryRetry(StmtExecutor.java:574) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:559) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.ConnectProcessor.executeQuery(ConnectProcessor.java:349) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:249) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.MysqlConnectProcessor.handleQuery(MysqlConnectProcessor.java:233) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.MysqlConnectProcessor.dispatch(MysqlConnectProcessor.java:261) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.MysqlConnectProcessor.processOnce(MysqlConnectProcessor.java:444) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.mysql.ReadListener.lambda$handleEvent$0(ReadListener.java:52) ~[doris-fe.jar:1.2-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
at java.lang.Thread.run(Thread.java:833) ~[?:?]
```
@github-actionsgithub-actionsBot removed the approved Indicates a PR has been approved by one committer. label May 14, 2025
@morningman

morningman commented May 14, 2025

Copy link
Copy Markdown
ContributorAuthor

run buildall

@github-actions

github-actionsBot commented May 15, 2025

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label May 15, 2025
dataroaring
dataroaring approved these changes May 15, 2025
View reviewed changes

@dataroaringdataroaring left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring
dataroaring merged commit 3ae4211 into apache:branch-3.0May 15, 2025
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gavinchougavinchougavinchou approved these changes
@dataroaringdataroaringdataroaring approved these changes

Assignees

No one assigned

Labels

approvedIndicates a PR has been approved by one committer.reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@morningman@hello-stephen@doris-robot@gavinchou@dataroaring@yagagagaga