Robustify boto3 session handling (DynamoDB, RestCatalog) by jayceslesar · Pull Request #2071 · apache/iceberg-python

jayceslesar · 2025-06-07T14:46:57Z

3 changes here...

Basically a mirror of Adds support for creating a GlueCatalog with own client #1920 but for dynamo
Allow users to pass in an existing client/pass initialize client with properties in RestCatalog with sigv4 enabled. (closesRestCatalog with sigv4 enabled doesn't pick up creds provided as arguments #2070, closesAllow to pass botocore session to REST Catalog (S3 Tables) like Glue Catalog does #2008)
Re-use of the client in add_headers in RestCatalog with sigv4. (closesRestCatalog with sigv4 enabled instantiates a boto session per each client call #2069 )

jayceslesar · 2025-06-08T14:39:05Z

 def __init__(self, **properties: str):
 super().__init__()
 self._properties = properties
+ self._boto_session = boto3.Session(
+ region_name=get_first_property_value(self._properties, AWS_REGION),
+ botocore_session=self._properties.get(BOTOCORE_SESSION),
+ aws_access_key_id=get_first_property_value(self._properties, AWS_ACCESS_KEY_ID),
+ aws_secret_access_key=get_first_property_value(self._properties, AWS_SECRET_ACCESS_KEY),
+ aws_session_toxken=get_first_property_value(self._properties, AWS_SESSION_TOKEN),
+ )

 def add_headers(self, request: PreparedRequest, **kwargs: Any) -> None: # pylint: disable=W0613
- boto_session = boto3.Session()
- credentials = boto_session.get_credentials().get_frozen_credentials()
- region = self._properties.get(SIGV4_REGION, boto_session.region_name)


@Fokko @kevinjqliu probably worth refactoring this class out to get some proper test coverage?

Hey @jayceslesar yes, I think that makes sense. Let's do that in a separate PR 👍

rudolfix · 2025-06-09T18:50:23Z

thx for working on this :)

Fokko · 2025-06-10T09:28:53Z

 glue = ["boto3", "mypy-boto3-glue"]
 adlfs = ["adlfs"]
-dynamodb = ["boto3"]
+dynamodb = ["boto3", "mypy-boto3-dynamodb"]


Shouldn't this be a dev-dependency?

probably -- the glue one isnt but I will move both over

Fokko · 2025-06-10T09:30:32Z

+ )
+ self.dynamodb = session.client(DYNAMODB_CLIENT)
+ self.dynamodb_table_name = self.properties.get(DYNAMODB_TABLE_NAME, DYNAMODB_TABLE_NAME_DEFAULT)
+ self._ensure_catalog_table_exists_or_create()


Do we want to execute this when passing in an existing client?

Fokko · 2025-06-10T09:31:28Z

+ aws_session_token=get_first_property_value(properties, DYNAMODB_SESSION_TOKEN, AWS_SESSION_TOKEN),
+ )
+ self.dynamodb = session.client(DYNAMODB_CLIENT)
+ self.dynamodb_table_name = self.properties.get(DYNAMODB_TABLE_NAME, DYNAMODB_TABLE_NAME_DEFAULT)


I think we also want to assign this one when passing in an existing client. Adding the methods to the class for completeness may also be a good thing to do as part of this PR.

I think they do exist (in the MetastoreCatalog) and were just never updated into this class. I will take a look

After a second look I dont know if we need to make a change _ensure_catalog_table_exists_or_create makes the dynamo table which is needed to make + interact with iceberg tables

Co-authored-by: Marko Grujic <markoog@gmail.com>

CharlesAttend · 2025-06-10T12:17:35Z

Thank you for fixing this!

Fokko

Thanks again @jayceslesar for working on this, one minor comment that needs to be addressed, but apart from that, this looks great 🥳

jayceslesar · 2025-06-11T14:19:10Z

-version = "2.44.0"
+version = "2.46.0"


@Fokko possible causes of CI failing?

Maybe need to add
"ignore::ResourceWarning", "ignore::pytest.PytestUnraisableExceptionWarning",
To filterwarnings? idrk
ray-project/ray#10279

Ideally, we don't want to bump Ray as part of this PR. Maybe add the <=2.44.0 constraint to Ray for now?

Fokko · 2025-06-11T19:15:52Z

Finally green, thanks @jayceslesar for working on this 🙌

3 changes here... 1. Basically a mirror of apache#1920 but for dynamo 2. Allow users to pass in an existing client/pass initialize client with properties in `RestCatalog` with sigv4 enabled. (closesapache#2070, closesapache#2008) 3. Re-use of the client in `add_headers` in `RestCatalog` with sigv4. (closesapache#2069 ) --------- Co-authored-by: Marko Grujic <markoog@gmail.com>

jayceslesar added 2 commits June 7, 2025 10:46

Adds support for creating a DynamoDB with own client
e81cd9c

allow custom session, re-use session if it exists
51f1e0b

jayceslesar changed the title ~~Adds support for creating a DynamoDB with own client~~Robustify boto3 session handling (DynamoDB, RestCatalog)Jun 8, 2025

jayceslesar commented Jun 8, 2025
View reviewed changes

Merge branch 'main' into dynamodb_client
eb0e1f9

rudolfix mentioned this pull request Jun 9, 2025
Allow to pass botocore session to REST Catalog (S3 Tables) like Glue Catalog does #2008
Closed

gruuya reviewed Jun 10, 2025
View reviewed changes

Comment threadpyiceberg/catalog/rest/__init__.py Outdated

Fokko reviewed Jun 10, 2025
View reviewed changes

jayceslesarand others added 3 commits June 10, 2025 07:13

Update pyiceberg/catalog/rest/__init__.py
b7ac8ae
Co-authored-by: Marko Grujic <markoog@gmail.com>

make dev deps
457bc4a

fix init
cc28012

Fokko approved these changes Jun 10, 2025
View reviewed changes

Comment threadpyiceberg/catalog/dynamodb.py Outdated

jayceslesar added 2 commits June 10, 2025 16:58

clean up type imports for glue and dynamo catalogs
9819074

Merge branch 'main' into dynamodb_client
26f0767

Fokko reviewed Jun 10, 2025
View reviewed changes

Comment threadpyiceberg/catalog/dynamodb.py Outdated

quote optional type imports
6584f25

Fokko reviewed Jun 10, 2025
View reviewed changes

Comment threadpyiceberg/catalog/glue.py Outdated

last quote I was missing oops
81ed8c6

jayceslesar commented Jun 11, 2025
View reviewed changes

restrict ray
d89610a

Fokko merged commit 38ebb19 into apache:mainJun 11, 2025

Flogue mentioned this pull request Oct 23, 2025
REST catalog, S3tables with botocore session #2657
Open
3 tasks

geruh mentioned this pull request Oct 25, 2025
Add python 3.13 support #2658
Closed

		version = "2.44.0"
		version = "2.46.0"

Uh oh!

Conversation

jayceslesar commented Jun 7, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rudolfix commented Jun 9, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jayceslesarJun 10, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

CharlesAttend commented Jun 10, 2025

Uh oh!

Fokko left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Fokko commented Jun 11, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

jayceslesar commented Jun 7, 2025•
edited
Loading

jayceslesarJun 10, 2025•
edited
Loading