Skip to content
This repository was archived by the owner on Mar 31, 2026. It is now read-only.

feat: Enable custom predicates for media operations - #1385

Merged
andrewsg merged 5 commits into
3.0-develfrom
3.0-retry-unification
Dec 3, 2024
Merged

feat: Enable custom predicates for media operations#1385
andrewsg merged 5 commits into
3.0-develfrom
3.0-retry-unification

Conversation

@andrewsg

@andrewsgandrewsg commented Nov 21, 2024

Copy link
Copy Markdown
Contributor

This refactor unifies the old resumable media retry code with the newer google.api_core retry code, enabling custom predicates for media operations.

Fixes#1361

@andrewsgandrewsg added the do not merge Indicates a pull request not ready for merge, due to either quality or timing. label Nov 21, 2024
@andrewsg
andrewsg requested review from a teamNovember 21, 2024 18:09
@product-auto-labelproduct-auto-labelBot added size: xl Pull request size is extra large. api: storage Issues related to the googleapis/python-storage API. labels Nov 21, 2024
@andrewsgandrewsg removed the do not merge Indicates a pull request not ready for merge, due to either quality or timing. label Nov 21, 2024

@cojencocojenco left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good, very excited about retry unification!

Comment threadtests/resumable_media/unit/requests/test__helpers.py
requests.ConnectionError,
requests_exceptions.ChunkedEncodingError,
requests_exceptions.Timeout,
http.client.BadStatusLine,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for consolidating the retryable errors. Looking at the requests docs requests.ConnectionError should be equivalent to requests.exceptions.ConnectionError so we're good!

Comment threadgoogle/cloud/storage/_media/common.py
)
return _request_helpers.wait_and_retry(retriable_request, self._retry_strategy)


Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: to match download, we probably want to add the retry arg docstrings for MultipartUpload, ResumableUpload etc

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, done

Comment threadgoogle/cloud/storage/blob.py
and the object will configure backoff and timeout options. Custom
predicates (customizable error codes) are not supported for media
operations such as this one.
and the object will configure backoff and timeout options.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Do we want to call out the default retry? I recall we received some feedback around stating the default value in our docstrings

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would like to incorporate DEFAULT_RETRY into the docs somehow without copy/pasting it into every method. Not sure how to do that yet. I'd prefer not to include it in this change as it's already quite large but it's a good idea.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good, let's update that in a separate PR

@andrewsg
andrewsg merged commit a2e9b57 into 3.0-develDec 3, 2024
@andrewsg
andrewsg deleted the 3.0-retry-unification branch December 3, 2024 00:00
@release-pleaserelease-pleaseBot mentioned this pull request Dec 11, 2024
@andrewsgandrewsg mentioned this pull request Dec 12, 2024
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

api: storageIssues related to the googleapis/python-storage API.size: xlPull request size is extra large.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@andrewsg@cojenco