Feature Request / Improvement
I've noticed that boto3 opens a new connection on each client call with rest.sigv4-enabled set to true, which is wasteful.
This happens because a boto3 session is instantiated inside the adapter's add_headers hook (which in turn is called for each HTTP call)
| defadd_headers(self, request: PreparedRequest, **kwargs: Any) ->None: # pylint: disable=W0613 |
| boto_session=boto3.Session() |
| credentials=boto_session.get_credentials().get_frozen_credentials() |
| region=self._properties.get(SIGV4_REGION, boto_session.region_name) |
| service=self._properties.get(SIGV4_SERVICE, "execute-api") |
The boto3 session should be re-used throughout the lifetime of the catalog, like the requests session, module any creds expiration/invalidation.
Feature Request / Improvement
I've noticed that
boto3opens a new connection on each client call withrest.sigv4-enabledset totrue, which is wasteful.This happens because a
boto3session is instantiated inside the adapter'sadd_headershook (which in turn is called for each HTTP call)iceberg-python/pyiceberg/catalog/rest/__init__.py
Lines 394 to 398 in a67c559
The
boto3session should be re-used throughout the lifetime of the catalog, like therequestssession, module any creds expiration/invalidation.