[feat](s3) support S3 storage vault credentials provider without role ARN by sollhui · Pull Request #64766 · apache/doris

sollhui · 2026-06-24T03:27:43Z

What problem does this PR solve?

S3 storage vault creation only treated role ARN as the credential-provider path. When users configured s3.credentials_provider_type without s3.role_arn, FE did not persist the provider type into ObjectStoreInfoPB, and Cloud meta-service still required AK/SK for the vault. The recycler also only read credential provider type inside the role ARN branch.

This change allows S3 storage vaults to use an explicit credentials provider type without role ARN. FE now writes cred_provider_type when s3.credentials_provider_type or AWS_CREDENTIALS_PROVIDER_TYPE is set, Cloud meta-service accepts credential-provider-based S3 vaults without AK/SK, and the recycler reads the provider type independently from role ARN.

hello-stephen · 2026-06-24T03:27:47Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

sollhui · 2026-06-24T03:28:02Z

run buildall

sollhui · 2026-06-24T03:28:39Z

/review

github-actions

Summary:
I found three correctness issues in the provider-only S3 vault support. The CREATE/add path is partially covered, but related ALTER, SHOW CREATE, and recycler credential-provider paths are not complete.

Critical checkpoint conclusions:

Goal/test proof: the PR aims to support S3 storage vault credential providers without a role ARN. The add path has tests, but ALTER, SHOW CREATE round-trip, and recycler v1 behavior are not covered.
Scope/focus: the code change is small, but it misses parallel paths that consume the same vault credential state.
Concurrency/lifecycle: no new concurrency or lifecycle hazard found in the changed code.
Config/compatibility: aws_credentials_provider_version=v1 remains an accepted mutable config and behaves differently for the new provider-only state.
Tests/results: added tests validate provider-only ADD only; missing tests correspond to the inline findings.
Observability/persistence/data correctness: persistence of the new state is explicit, but downstream consumers do not all serialize or honor it correctly.

Subagent conclusions:

optimizer-rewrite: OR-1 and OR-2 became inline comments M-1 and M-2; OR-3 was found during convergence and became M-3.
tests-session-config: TSC-1 duplicated OR-1 and was merged into M-1.
Convergence round 2 ended with both subagents replying NO_NEW_VALUABLE_FINDINGS for the final M-1/M-2/M-3 comment set.

User focus: no additional user-provided review focus was present.

liaoxin01

LGTM

github-actions · 2026-06-24T03:56:03Z

PR approved by at least one committer and no changes requested.

hello-stephen · 2026-06-24T04:09:11Z

TPC-H: Total hot run time: 29374 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 343f522e5a2083a50d837efa0c19617bc34ebdcc, data reload: false
------ Round 1 ----------------------------------
============================================
q1	17750	4144	4081	4081
q2	2009	325	191	191
q3	10295	1494	806	806
q4	4680	477	346	346
q5	7514	862	568	568
q6	190	179	143	143
q7	817	884	639	639
q8	9363	1639	1701	1639
q9	5598	4582	4554	4554
q10	6754	1802	1526	1526
q11	437	276	250	250
q12	634	430	316	316
q13	18070	3400	2785	2785
q14	268	273	237	237
q15	q16	791	787	716	716
q17	1105	966	924	924
q18	7132	5836	5590	5590
q19	1301	1406	1087	1087
q20	500	429	269	269
q21	6050	2630	2396	2396
q22	444	362	311	311
Total cold run time: 101702 ms
Total hot run time: 29374 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4371	4316	4311	4311
q2	333	356	236	236
q3	4608	5001	4466	4466
q4	2122	2226	1481	1481
q5	4481	4356	4360	4356
q6	240	184	133	133
q7	1771	2143	1801	1801
q8	2714	2251	2204	2204
q9	8346	8390	8131	8131
q10	4843	4824	4329	4329
q11	582	410	405	405
q12	743	762	541	541
q13	3293	3604	3024	3024
q14	285	301	285	285
q15	q16	722	747	670	670
q17	1408	1360	1528	1360
q18	7884	7245	7287	7245
q19	1167	1115	1112	1112
q20	2245	2223	1994	1994
q21	5340	4741	4544	4544
q22	525	472	430	430
Total cold run time: 58023 ms
Total hot run time: 53058 ms

hello-stephen · 2026-06-24T04:20:13Z

TPC-DS: Total hot run time: 173728 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 343f522e5a2083a50d837efa0c19617bc34ebdcc, data reload: false
query5	4347	648	497	497
query6	447	195	185	185
query7	4965	595	315	315
query8	362	219	200	200
query9	8776	4096	4103	4096
query10	481	331	257	257
query11	5845	2353	2133	2133
query12	161	105	97	97
query13	1285	641	421	421
query14	6372	5480	5099	5099
query14_1	4465	4381	4362	4362
query15	204	194	171	171
query16	1044	458	425	425
query17	1105	708	563	563
query18	2706	475	343	343
query19	200	189	138	138
query20	113	109	108	108
query21	214	138	115	115
query22	13725	13610	13388	13388
query23	17486	16576	16217	16217
query23_1	16309	16440	16217	16217
query24	7485	1782	1343	1343
query24_1	1343	1309	1288	1288
query25	543	440	407	407
query26	1324	328	175	175
query27	2644	529	352	352
query28	4399	2029	2031	2029
query29	1082	670	501	501
query30	307	239	203	203
query31	1117	1077	956	956
query32	102	62	62	62
query33	523	330	267	267
query34	1213	1168	654	654
query35	778	794	687	687
query36	1386	1440	1192	1192
query37	160	113	96	96
query38	1893	1725	1666	1666
query39	941	931	886	886
query39_1	884	893	888	888
query40	223	129	109	109
query41	72	68	68	68
query42	92	92	89	89
query43	338	328	296	296
query44	1489	809	794	794
query45	201	192	181	181
query46	1066	1192	773	773
query47	2404	2386	2260	2260
query48	401	403	293	293
query49	653	480	371	371
query50	1026	361	272	272
query51	4482	4479	4312	4312
query52	82	85	73	73
query53	258	274	192	192
query54	285	246	237	237
query55	77	73	68	68
query56	257	246	240	240
query57	1421	1428	1336	1336
query58	317	228	208	208
query59	1622	1676	1460	1460
query60	285	258	240	240
query61	172	168	203	168
query62	699	668	584	584
query63	226	187	196	187
query64	2491	767	599	599
query65	4858	4778	4789	4778
query66	1710	466	340	340
query67	29965	29831	29737	29737
query68	3300	1520	889	889
query69	424	293	260	260
query70	1081	994	973	973
query71	294	226	220	220
query72	2882	2618	2331	2331
query73	818	825	422	422
query74	5156	4999	4751	4751
query75	2658	2652	2249	2249
query76	2322	1211	816	816
query77	347	379	294	294
query78	12608	12651	11915	11915
query79	1407	1141	798	798
query80	1270	486	385	385
query81	518	290	240	240
query82	616	155	123	123
query83	326	278	249	249
query84	271	155	115	115
query85	903	524	422	422
query86	424	306	281	281
query87	1835	1840	1774	1774
query88	3757	2795	2732	2732
query89	431	381	334	334
query90	1928	190	190	190
query91	176	157	132	132
query92	64	64	58	58
query93	1607	1466	854	854
query94	725	351	337	337
query95	690	377	346	346
query96	1084	827	359	359
query97	2732	2776	2596	2596
query98	225	209	202	202
query99	1209	1187	1028	1028
Total cold run time: 260432 ms
Total hot run time: 173728 ms

hello-stephen · 2026-06-24T04:25:08Z

ClickBench: Total hot run time: 25.29 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 343f522e5a2083a50d837efa0c19617bc34ebdcc, data reload: false
query1	0.01	0.00	0.00
query2	0.10	0.05	0.05
query3	0.26	0.14	0.13
query4	1.60	0.14	0.13
query5	0.25	0.23	0.23
query6	1.23	1.11	1.06
query7	0.03	0.01	0.01
query8	0.06	0.04	0.04
query9	0.38	0.33	0.32
query10	0.57	0.56	0.54
query11	0.19	0.15	0.15
query12	0.22	0.14	0.15
query13	0.50	0.49	0.48
query14	1.01	1.01	1.03
query15	0.62	0.61	0.59
query16	0.31	0.33	0.31
query17	1.11	1.09	1.10
query18	0.23	0.22	0.22
query19	2.10	1.99	1.97
query20	0.02	0.01	0.01
query21	15.44	0.21	0.13
query22	4.93	0.05	0.06
query23	16.14	0.30	0.12
query24	2.99	0.40	0.30
query25	0.11	0.06	0.05
query26	0.73	0.23	0.15
query27	0.04	0.03	0.04
query28	3.45	0.93	0.54
query29	12.53	4.36	3.45
query30	0.29	0.15	0.16
query31	2.77	0.58	0.33
query32	3.23	0.62	0.49
query33	3.25	3.20	3.22
query34	15.65	4.24	3.54
query35	3.52	3.49	3.50
query36	0.55	0.45	0.42
query37	0.09	0.06	0.07
query38	0.05	0.04	0.03
query39	0.04	0.03	0.03
query40	0.18	0.16	0.16
query41	0.08	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 96.94 s
Total hot run time: 25.29 s

hello-stephen · 2026-06-24T06:12:49Z

FE Regression Coverage Report

Increment line coverage 0.00% (0/10) 🎉
Increment coverage report
Complete coverage report

skip

### What problem does this PR solve? Issue Number: close #xxx Related PR: apache#64766 Problem Summary: The branch-4.0 backport left an unused CredProviderTypePB import in S3Properties.java, causing the FE Code Style Checker to fail. Remove the unused import. ### Release note None ### Check List (For Author) - Test: mvn clean checkstyle:check - Behavior changed: No - Does this need documentation: No

### What problem does this PR solve? Issue Number: close #xxx Related PR: apache#64766 Problem Summary: The branch-4.0 backport used provider enum values such as CONTAINER and ENV in Cloud and FE tests, but branch-4.0's Cloud proto only defines DEFAULT, SIMPLE, and INSTANCE_PROFILE. This caused Cloud UT to fail while compiling meta_service_test.cpp. Adapt the provider-only storage vault coverage to use INSTANCE_PROFILE and add the missing FE conversion helper and AWS_CREDENTIALS_PROVIDER_TYPE compatibility handling for branch-4.0. ### Release note None ### Check List (For Author) - Test: mvn clean checkstyle:check; ./run-fe-ut.sh --run org.apache.doris.datasource.property.storage.S3PropertiesTest - Behavior changed: No - Does this need documentation: No

…without role ARN (#65013) pick #64766

…without role ARN (#65011) pick #64766

support S3 storage vault credentials provider without role ARN
343f522

sollhui requested review from gavinchou, liaoxin01 and luwei16 as code owners June 24, 2026 03:27

sollhui added dev/3.1.x dev/4.0.x dev/4.1.x labels Jun 24, 2026

github-actionsBot previously requested changes Jun 24, 2026
View reviewed changes

Comment threadfe/fe-core/src/main/java/org/apache/doris/datasource/property/storage/S3Properties.java
Comment threadcloud/src/meta-service/meta_service_resource.cpp
Comment threadcloud/src/recycler/s3_accessor.cpp

liaoxin01 approved these changes Jun 24, 2026
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Jun 24, 2026

sollhui merged commit 7b0d4ff into apache:masterJun 24, 2026
37 checks passed

sollhui deleted the support_s3_vault_credentials_provider_without_role_arn branch June 24, 2026 06:13

github-actionsBot added dev/4.0.x-conflict dev/4.1.x-conflict labels Jun 24, 2026

This was referenced Jun 30, 2026
branch-4.0: [feat](s3) support S3 storage vault credentials provider without role ARN #65011
Merged
branch-4.1: [feat](s3) support S3 storage vault credentials provider without role ARN #65013
Merged

yiguolei pushed a commit that referenced this pull request Jul 1, 2026
branch-4.1: [feat](s3) support S3 storage vault credentials provider …
d344f83
…without role ARN (#65013) pick #64766

yiguolei removed dev/4.1.x dev/4.1.x-conflict labels Jul 1, 2026

yiguolei added the dev/4.1.3-merged label Jul 1, 2026

morningman added dev/4.0.7-merged and removed dev/4.0.x dev/4.0.x-conflict labels Jul 2, 2026

morningman pushed a commit that referenced this pull request Jul 2, 2026
branch-4.0: [feat](s3) support S3 storage vault credentials provider …
8391ed0
…without role ARN (#65011) pick #64766

morningman mentioned this pull request Jul 9, 2026
4.0.7 Release Notes #65399
Closed

This was referenced Jul 9, 2026
[Feature] Support keyless GCS authentication (Application Default Credentials / Workload Identity) for storage vaults #65432
Open
[feat](vault) Support keyless GCS vaults with GKE Workload Identity #65433
Closed

sollhui removed the dev/3.1.x label Aug 2, 2026

cyriltovena mentioned this pull request Aug 5, 2026
[feature](vault) Support keyless GCS vaults with GKE Workload Identity #66484
Open
10 tasks

Navigation Menu

Uh oh!

[feat](s3) support S3 storage vault credentials provider without role ARN - #64766

[feat](s3) support S3 storage vault credentials provider without role ARN#64766
sollhui merged 1 commit into
apache:masterfrom
sollhui:support_s3_vault_credentials_provider_without_role_arn

sollhui commented Jun 24, 2026

Uh oh!

hello-stephen commented Jun 24, 2026

Uh oh!

sollhui commented Jun 24, 2026

Uh oh!

sollhui commented Jun 24, 2026

Uh oh!

github-actionsBot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

liaoxin01 left a comment

Uh oh!

github-actionsBot commented Jun 24, 2026

Uh oh!

hello-stephen commented Jun 24, 2026

Uh oh!

hello-stephen commented Jun 24, 2026

Uh oh!

hello-stephen commented Jun 24, 2026

Uh oh!

hello-stephen commented Jun 24, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

sollhui commented Jun 24, 2026

What problem does this PR solve?

Uh oh!

hello-stephen commented Jun 24, 2026

Uh oh!

sollhui commented Jun 24, 2026

Uh oh!

sollhui commented Jun 24, 2026

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

liaoxin01 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Jun 24, 2026

Uh oh!

hello-stephen commented Jun 24, 2026

Uh oh!

hello-stephen commented Jun 24, 2026

Uh oh!

hello-stephen commented Jun 24, 2026

Uh oh!

hello-stephen commented Jun 24, 2026

FE Regression Coverage Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants