Uh oh!
There was an error while loading. Please reload this page.
fix: replace default retry for upload operations - #480
Conversation
andrewsg
left a comment
There was a problem hiding this comment.
Good work here. Please amend the PR title to include that create_resumable_upload_session is being amended in the same PR (or split it off into a separate PR) so that the change is included in the release notes.
| This private method does not accept ConditionalRetryPolicy values | ||
| because the information necessary to evaluate the policy is instead | ||
| evaluated in client.download_blob_to_file(). | ||
| evaluated in blob._do_upload(). |
There was a problem hiding this comment.
Oh, it looks like we're evaluating it BOTH in blob._do_upload() and in client.download_blob_to_file(). This is harmless, which is why the tests didn't catch it, but we only need one of those.
I don't have a strong opinion on whether we do this in _do_upload or download_blob_to_file. Let's keep this docstring change and additionally we can remove the conditional retry policy code in client.download_blob_to_file, either in another PR, or in this one if you prefer.
There was a problem hiding this comment.
IIUC, I believe we'll need to remain both ConditionalRetryPolicy evaluations as one is for upload operations and the other is for downloads. I revised the docstrings here to match particularly where we're doing the evaluations for upload operations, blob._do_upload() .
tritone
left a comment
There was a problem hiding this comment.
Nice catch on this! A couple questions
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
* fix: revise upload operations preconditions to if_generation_match * fix docstrings * add retry configuration to blob.create_resumable_upload_session * align var values in test * test coverage * Revert "test coverage" This reverts commit e91916f. * Revert "align var values in test" This reverts commit aec585b. * Revert "add retry configuration to blob.create_resumable_upload_session" This reverts commit 8c1ae3c. * revise tests after reverting
* fix: revise upload operations preconditions to if_generation_match * fix docstrings * add retry configuration to blob.create_resumable_upload_session * align var values in test * test coverage * Revert "test coverage" This reverts commit e91916f. * Revert "align var values in test" This reverts commit aec585b. * Revert "add retry configuration to blob.create_resumable_upload_session" This reverts commit 8c1ae3c. * revise tests after reverting
This PR replaces upload operations default retries with
storage.retry.DEFAULT_RETRY_IF_GENERATION_SPECIFIEDand aligns with the retry strategy conditional idempotency.With default retries set to
storage.retry.DEFAULT_RETRY_IF_GENERATION_SPECIFIED, passing inifGenerationMatchmakes the upload operation (1) conditional and (2) only retried by default on whether the object's current generation matches the given value. Settingif_generation_match=0makes the upload operation succeed only if there are no live versions of the object (fresh uploads) and supports retries.blob.create_resumable_upload_session()Fixes#477 🦕