Uh oh!
There was an error while loading. Please reload this page.
[feat](Iceberg)Rest & S3Table Support Iam-role - #60498
Conversation
hello-stephen
commented
Feb 4, 2026
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
CalvinKirs
commented
Feb 4, 2026
run buildall |
hello-stephen
commented
Feb 4, 2026
FE UT Coverage ReportIncrement line coverage |
doris-robot
commented
Feb 4, 2026
TPC-H: Total hot run time: 31840 ms |
doris-robot
commented
Feb 4, 2026
ClickBench: Total hot run time: 28.31 s |
hello-stephen
commented
Feb 4, 2026
FE Regression Coverage ReportIncrement line coverage |
Sbaia
commented
Feb 5, 2026
Thanks for picking this up and building on #59893, @CalvinKirs. I want to clarify an important distinction about what this PR actually enables vs. what users on EKS/GKE typically need. What this PR adds: cross-account assume-role support ( What this PR does NOT add: support for IRSA (IAM Roles for Service Accounts) or, more generally, the AWS default credentials chain without any explicit credential configuration. The validation in booleanhasAccessKeyAndSecret = ...;
booleanhasIamRole = ...;
return !hasAccessKeyAndSecret && !hasIamRole; // fails validationWith IRSA (and future equivalents like GKE Workload Identity or ECS task roles), the pod already has credentials injected via the service account. The user doesn't have a Regarding the assume-role path itself:
So the assume-role feature works, but it's solving a different use case (cross-account access) than what most EKS users need (direct IRSA credentials without specifying any role_arn). Our original PR #59893 takes a simpler approach: when no explicit credentials are provided, we just don't set any credential properties, allowing Iceberg/AWS SDK to resolve credentials through the default chain. This covers IRSA, instance profiles, environment variables, and any future pod identity mechanism — with zero configuration in the catalog DDL. I'd suggest either:
Happy to collaborate on merging the two approaches. |
CalvinKirs
commented
Feb 9, 2026
@Sbaia Hi, It turns out I can set |
CalvinKirs
commented
Feb 9, 2026
run buildall |
doris-robot
commented
Feb 9, 2026
TPC-H: Total hot run time: 30129 ms |
doris-robot
commented
Feb 9, 2026
ClickBench: Total hot run time: 28.15 s |
hello-stephen
commented
Feb 9, 2026
FE UT Coverage ReportIncrement line coverage |
hello-stephen
commented
Feb 9, 2026
FE Regression Coverage ReportIncrement line coverage |
CalvinKirs
commented
Feb 24, 2026
@Sbaia Hi, Could you take a look |
### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: Fix a rebase conflict residue in Iceberg REST custom signing-name test so explicit credentials are provided when credential properties are expected. ### Release note None ### Check List (For Author) - Test: Unit Test - ./run-fe-ut.sh --run org.apache.doris.datasource.property.metastore.IcebergRestPropertiesTest,org.apache.doris.datasource.property.metastore.IcebergS3TablesMetaStorePropertiesTest - Behavior changed: No - Does this need documentation: No
eaf74d2 to
71a2372Compare### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: Deduplicate Iceberg REST and S3Tables AWS credential property priority handling while keeping their explicit credential key formats separate. ### Release note None ### Check List (For Author) - Test: Unit Test - ./run-fe-ut.sh --run org.apache.doris.datasource.property.metastore.IcebergRestPropertiesTest,org.apache.doris.datasource.property.metastore.IcebergS3TablesMetaStorePropertiesTest - Behavior changed: No - Does this need documentation: No
### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: Use a Doris Iceberg AWS assume-role credentials provider so REST, S3Tables, and S3 FileIO assume-role credentials honor the configured AWS credentials provider type as the STS source provider. ### Release note None ### Check List (For Author) - Test: Unit Test - ./run-fe-ut.sh --run org.apache.doris.datasource.property.metastore.IcebergRestPropertiesTest,org.apache.doris.datasource.property.metastore.IcebergS3TablesMetaStorePropertiesTest - Behavior changed: No - Does this need documentation: No
### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: Pass standard Iceberg assume-role and source credentials-provider properties into the Doris Iceberg assume-role provider so the provider does not reparse Doris credentials_provider_type. ### Release note None ### Check List (For Author) - Test: Unit Test - ./run-fe-ut.sh --run org.apache.doris.datasource.property.metastore.IcebergRestPropertiesTest,org.apache.doris.datasource.property.metastore.IcebergS3TablesMetaStorePropertiesTest - Behavior changed: No - Does this need documentation: No
### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: Build Iceberg assume-role provider properties from Doris S3Properties and pass only the resolved primitive values into the Iceberg credentials provider. ### Release note None ### Check List (For Author) - Test: Unit Test - ./run-fe-ut.sh --run org.apache.doris.datasource.property.metastore.IcebergRestPropertiesTest,org.apache.doris.datasource.property.metastore.IcebergS3TablesMetaStorePropertiesTest - Behavior changed: No - Does this need documentation: No
### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: Reuse Doris S3Properties as the source for Iceberg assume-role credentials provider configuration and pass S3Properties-compatible provider parameters through Iceberg/S3Tables SPI. ### Release note None ### Check List (For Author) - Test: Unit Test - ./run-fe-ut.sh --run org.apache.doris.datasource.property.metastore.IcebergRestPropertiesTest,org.apache.doris.datasource.property.metastore.IcebergS3TablesMetaStorePropertiesTest - Behavior changed: No - Does this need documentation: No
### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: Share Iceberg REST and S3Tables AWS credential property construction, avoid provider-specific S3 parameters for standard AWS providers, and build S3Tables control-plane and S3 FileIO clients from S3Properties for explicit credentials and IAM role support. ### Release note None ### Check List (For Author) - Test: Unit Test - ./run-fe-ut.sh --run org.apache.doris.datasource.property.metastore.IcebergRestPropertiesTest,org.apache.doris.datasource.property.metastore.IcebergS3TablesMetaStorePropertiesTest - Behavior changed: No - Does this need documentation: No
### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: Remove the Doris-specific Iceberg S3 FileIO client factory and pass standard Iceberg assume-role properties directly. Explicit AK/SK and non-role credential provider modes continue to use Iceberg/AWS SDK standard properties. ### Release note None ### Check List (For Author) - Test: Unit Test - ./run-fe-ut.sh --run org.apache.doris.datasource.property.metastore.IcebergRestPropertiesTest,org.apache.doris.datasource.property.metastore.IcebergS3TablesMetaStorePropertiesTest - Behavior changed: No - Does this need documentation: No
### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: Move S3 FileIO credential property construction and S3Tables runtime AWS credentials provider selection into the shared Iceberg AWS credential helper, keeping S3Tables-specific client construction local. ### Release note None ### Check List (For Author) - Test: Unit Test - ./run-fe-ut.sh --run org.apache.doris.datasource.property.metastore.IcebergRestPropertiesTest,org.apache.doris.datasource.property.metastore.IcebergS3TablesMetaStorePropertiesTest - Behavior changed: No - Does this need documentation: No
CalvinKirs
commented
Apr 27, 2026
run buildall |
hello-stephen
commented
Apr 27, 2026
FE Regression Coverage ReportIncrement line coverage |
### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: Iceberg REST catalog exposed iceberg.rest.role_arn, but REST SigV4 signing does not support this Doris property without an explicit credentials provider wrapper. Reject the unsupported property and keep REST signing on AK/SK or credentials_provider_type. ### Release note None ### Check List (For Author) - Test: Unit Test - ./run-fe-ut.sh --run org.apache.doris.datasource.property.metastore.IcebergRestPropertiesTest - Behavior changed: Yes. Iceberg REST catalog now rejects unsupported iceberg.rest.role_arn instead of mapping it to Iceberg assume-role client factory properties. - Does this need documentation: No
PR approved by anyone and no changes requested. |
Uh oh!
There was an error while loading. Please reload this page.
FYI apache#59893 The initial implementation was contributed by https://github.com/Sbaia . Many thanks for his contributions.
FYI apache#59893 The initial implementation was contributed by https://github.com/Sbaia . Many thanks for his contributions.
…lugin The iceberg system tables ($snapshots, $files, ...) are the one part of iceberg BE reads through Java: FE plans the scan and Java-serializes each FileScanTask across, and only iceberg's own library can turn one back into rows. That makes this module a plugin like any other - its own directory, its own classloader, its own copy of everything it needs. What it needs turns out to be less than the shared preload layer was handing it. A plugin only ever deserializes a task and the FileIO inside it; it never builds a catalog, so the AWS catalog clients (glue, dynamodb, s3tables) stay in FE and the plugin carries only the FileIO half of the SDK. kms stays despite nothing here encrypting anything: AwsClientFactories resolves a kms class in its static initializer, so without it the S3 path dies before it starts. Hadoop, by contrast, has to be added - it used to arrive from lib/hadoop_hdfs on BE's system classpath, which a plugin cannot see. Logging needed one bridge rather than three. Dropping hadoop's reload4j - a second logging implementation, and reachable only from a servlet and a daemon banner - also takes commons-logging's discovery down to its JDK14 backend, so httpclient's output on the S3 path lands in jni.log by itself. Only fe-kerberos, which speaks log4j2, needs a bridge. CustomAwsCredentialsProvider goes. It was reached by class name from a catalog property FE no longer emits: its fe-core twin was deleted with apache#60498, which moved the s3tables catalog to a hand-built client, and FE now names only the AWS SDK's own providers there. Nothing in FE, BE, or the regression suites mentions it. The scanner keeps its factory named sys-table rather than the reader every other connector publishes, because it is not iceberg's reader: data files go through BE's native readers, and this serves the metadata tables alone.
…lugin The iceberg system tables ($snapshots, $files, ...) are the one part of iceberg BE reads through Java: FE plans the scan and Java-serializes each FileScanTask across, and only iceberg's own library can turn one back into rows. That makes this module a plugin like any other - its own directory, its own classloader, its own copy of everything it needs. What it needs turns out to be less than the shared preload layer was handing it. A plugin only ever deserializes a task and the FileIO inside it; it never builds a catalog, so the AWS catalog clients (glue, dynamodb, s3tables) stay in FE and the plugin carries only the FileIO half of the SDK. kms stays despite nothing here encrypting anything: AwsClientFactories resolves a kms class in its static initializer, so without it the S3 path dies before it starts. Hadoop, by contrast, has to be added - it used to arrive from lib/hadoop_hdfs on BE's system classpath, which a plugin cannot see. Logging needed one bridge rather than three. Dropping hadoop's reload4j - a second logging implementation, and reachable only from a servlet and a daemon banner - also takes commons-logging's discovery down to its JDK14 backend, so httpclient's output on the S3 path lands in jni.log by itself. Only fe-kerberos, which speaks log4j2, needs a bridge. CustomAwsCredentialsProvider goes. It was reached by class name from a catalog property FE no longer emits: its fe-core twin was deleted with apache#60498, which moved the s3tables catalog to a hand-built client, and FE now names only the AWS SDK's own providers there. Nothing in FE, BE, or the regression suites mentions it. The scanner keeps its factory named sys-table rather than the reader every other connector publishes, because it is not iceberg's reader: data files go through BE's native readers, and this serves the metadata tables alone.
…lugin The iceberg system tables ($snapshots, $files, ...) are the one part of iceberg BE reads through Java: FE plans the scan and Java-serializes each FileScanTask across, and only iceberg's own library can turn one back into rows. That makes this module a plugin like any other - its own directory, its own classloader, its own copy of everything it needs. What it needs turns out to be less than the shared preload layer was handing it. A plugin only ever deserializes a task and the FileIO inside it; it never builds a catalog, so the AWS catalog clients (glue, dynamodb, s3tables) stay in FE and the plugin carries only the FileIO half of the SDK. kms stays despite nothing here encrypting anything: AwsClientFactories resolves a kms class in its static initializer, so without it the S3 path dies before it starts. Hadoop, by contrast, has to be added - it used to arrive from lib/hadoop_hdfs on BE's system classpath, which a plugin cannot see. Logging needed one bridge rather than three. Dropping hadoop's reload4j - a second logging implementation, and reachable only from a servlet and a daemon banner - also takes commons-logging's discovery down to its JDK14 backend, so httpclient's output on the S3 path lands in jni.log by itself. Only fe-kerberos, which speaks log4j2, needs a bridge. CustomAwsCredentialsProvider goes. It was reached by class name from a catalog property FE no longer emits: its fe-core twin was deleted with apache#60498, which moved the s3tables catalog to a hand-built client, and FE now names only the AWS SDK's own providers there. Nothing in FE, BE, or the regression suites mentions it. The scanner keeps its factory named sys-table rather than the reader every other connector publishes, because it is not iceberg's reader: data files go through BE's native readers, and this serves the metadata tables alone.
…lugin The iceberg system tables ($snapshots, $files, ...) are the one part of iceberg BE reads through Java: FE plans the scan and Java-serializes each FileScanTask across, and only iceberg's own library can turn one back into rows. That makes this module a plugin like any other - its own directory, its own classloader, its own copy of everything it needs. What it needs turns out to be less than the shared preload layer was handing it. A plugin only ever deserializes a task and the FileIO inside it; it never builds a catalog, so the AWS catalog clients (glue, dynamodb, s3tables) stay in FE and the plugin carries only the FileIO half of the SDK. kms stays despite nothing here encrypting anything: AwsClientFactories resolves a kms class in its static initializer, so without it the S3 path dies before it starts. Hadoop, by contrast, has to be added - it used to arrive from lib/hadoop_hdfs on BE's system classpath, which a plugin cannot see. Logging needed one bridge rather than three. Dropping hadoop's reload4j - a second logging implementation, and reachable only from a servlet and a daemon banner - also takes commons-logging's discovery down to its JDK14 backend, so httpclient's output on the S3 path lands in jni.log by itself. Only fe-kerberos, which speaks log4j2, needs a bridge. CustomAwsCredentialsProvider goes. It was reached by class name from a catalog property FE no longer emits: its fe-core twin was deleted with apache#60498, which moved the s3tables catalog to a hand-built client, and FE now names only the AWS SDK's own providers there. Nothing in FE, BE, or the regression suites mentions it. The scanner keeps its factory named sys-table rather than the reader every other connector publishes, because it is not iceberg's reader: data files go through BE's native readers, and this serves the metadata tables alone.
…lugin The iceberg system tables ($snapshots, $files, ...) are the one part of iceberg BE reads through Java: FE plans the scan and Java-serializes each FileScanTask across, and only iceberg's own library can turn one back into rows. That makes this module a plugin like any other - its own directory, its own classloader, its own copy of everything it needs. What it needs turns out to be less than the shared preload layer was handing it. A plugin only ever deserializes a task and the FileIO inside it; it never builds a catalog, so the AWS catalog clients (glue, dynamodb, s3tables) stay in FE and the plugin carries only the FileIO half of the SDK. kms stays despite nothing here encrypting anything: AwsClientFactories resolves a kms class in its static initializer, so without it the S3 path dies before it starts. Hadoop, by contrast, has to be added - it used to arrive from lib/hadoop_hdfs on BE's system classpath, which a plugin cannot see. Logging needed one bridge rather than three. Dropping hadoop's reload4j - a second logging implementation, and reachable only from a servlet and a daemon banner - also takes commons-logging's discovery down to its JDK14 backend, so httpclient's output on the S3 path lands in jni.log by itself. Only fe-kerberos, which speaks log4j2, needs a bridge. CustomAwsCredentialsProvider goes. It was reached by class name from a catalog property FE no longer emits: its fe-core twin was deleted with apache#60498, which moved the s3tables catalog to a hand-built client, and FE now names only the AWS SDK's own providers there. Nothing in FE, BE, or the regression suites mentions it. The scanner keeps its factory named sys-table rather than the reader every other connector publishes, because it is not iceberg's reader: data files go through BE's native readers, and this serves the metadata tables alone.
…lugin The iceberg system tables ($snapshots, $files, ...) are the one part of iceberg BE reads through Java: FE plans the scan and Java-serializes each FileScanTask across, and only iceberg's own library can turn one back into rows. That makes this module a plugin like any other - its own directory, its own classloader, its own copy of everything it needs. What it needs turns out to be less than the shared preload layer was handing it. A plugin only ever deserializes a task and the FileIO inside it; it never builds a catalog, so the AWS catalog clients (glue, dynamodb, s3tables) stay in FE and the plugin carries only the FileIO half of the SDK. kms stays despite nothing here encrypting anything: AwsClientFactories resolves a kms class in its static initializer, so without it the S3 path dies before it starts. Hadoop, by contrast, has to be added - it used to arrive from lib/hadoop_hdfs on BE's system classpath, which a plugin cannot see. Logging needed one bridge rather than three. Dropping hadoop's reload4j - a second logging implementation, and reachable only from a servlet and a daemon banner - also takes commons-logging's discovery down to its JDK14 backend, so httpclient's output on the S3 path lands in jni.log by itself. Only fe-kerberos, which speaks log4j2, needs a bridge. CustomAwsCredentialsProvider goes. It was reached by class name from a catalog property FE no longer emits: its fe-core twin was deleted with apache#60498, which moved the s3tables catalog to a hand-built client, and FE now names only the AWS SDK's own providers there. Nothing in FE, BE, or the regression suites mentions it. The scanner keeps its factory named sys-table rather than the reader every other connector publishes, because it is not iceberg's reader: data files go through BE's native readers, and this serves the metadata tables alone.
…lugin The iceberg system tables ($snapshots, $files, ...) are the one part of iceberg BE reads through Java: FE plans the scan and Java-serializes each FileScanTask across, and only iceberg's own library can turn one back into rows. That makes this module a plugin like any other - its own directory, its own classloader, its own copy of everything it needs. What it needs turns out to be less than the shared preload layer was handing it. A plugin only ever deserializes a task and the FileIO inside it; it never builds a catalog, so the AWS catalog clients (glue, dynamodb, s3tables) stay in FE and the plugin carries only the FileIO half of the SDK. kms stays despite nothing here encrypting anything: AwsClientFactories resolves a kms class in its static initializer, so without it the S3 path dies before it starts. Hadoop, by contrast, has to be added - it used to arrive from lib/hadoop_hdfs on BE's system classpath, which a plugin cannot see. Logging needed one bridge rather than three. Dropping hadoop's reload4j - a second logging implementation, and reachable only from a servlet and a daemon banner - also takes commons-logging's discovery down to its JDK14 backend, so httpclient's output on the S3 path lands in jni.log by itself. Only fe-kerberos, which speaks log4j2, needs a bridge. CustomAwsCredentialsProvider goes. It was reached by class name from a catalog property FE no longer emits: its fe-core twin was deleted with apache#60498, which moved the s3tables catalog to a hand-built client, and FE now names only the AWS SDK's own providers there. Nothing in FE, BE, or the regression suites mentions it. The scanner keeps its factory named sys-table rather than the reader every other connector publishes, because it is not iceberg's reader: data files go through BE's native readers, and this serves the metadata tables alone.
…lugin The iceberg system tables ($snapshots, $files, ...) are the one part of iceberg BE reads through Java: FE plans the scan and Java-serializes each FileScanTask across, and only iceberg's own library can turn one back into rows. That makes this module a plugin like any other - its own directory, its own classloader, its own copy of everything it needs. What it needs turns out to be less than the shared preload layer was handing it. A plugin only ever deserializes a task and the FileIO inside it; it never builds a catalog, so the AWS catalog clients (glue, dynamodb, s3tables) stay in FE and the plugin carries only the FileIO half of the SDK. kms stays despite nothing here encrypting anything: AwsClientFactories resolves a kms class in its static initializer, so without it the S3 path dies before it starts. Hadoop, by contrast, has to be added - it used to arrive from lib/hadoop_hdfs on BE's system classpath, which a plugin cannot see. Logging needed one bridge rather than three. Dropping hadoop's reload4j - a second logging implementation, and reachable only from a servlet and a daemon banner - also takes commons-logging's discovery down to its JDK14 backend, so httpclient's output on the S3 path lands in jni.log by itself. Only fe-kerberos, which speaks log4j2, needs a bridge. CustomAwsCredentialsProvider goes. It was reached by class name from a catalog property FE no longer emits: its fe-core twin was deleted with apache#60498, which moved the s3tables catalog to a hand-built client, and FE now names only the AWS SDK's own providers there. Nothing in FE, BE, or the regression suites mentions it. The scanner keeps its factory named sys-table rather than the reader every other connector publishes, because it is not iceberg's reader: data files go through BE's native readers, and this serves the metadata tables alone.
FYI #59893
The initial implementation was contributed by https://github.com/Sbaia . Many thanks for his contributions.