Playing with retries. by daspecster · Pull Request #2040 · googleapis/google-cloud-python

daspecster · 2016-08-01T17:55:47Z

So I was playing with getting the retries working on the bigquery system3 tests.

I'm not sure what fixtures need to be created to get all the tests passing but this seemed to get the bigquery tests to pass.

Also, I have the retry set pretty long, looks like it takes about an average of 20 seconds for it to eventually be consistent.

Sign in to view

+from functools import wraps
+
+
+def retry(ExceptionToCheck, tries=4, delay=3, backoff=2, logger=None):


daspecster · 2016-08-01T18:52:40Z

Sorry, I should have put more in the description I think.

So my point with this, was that there are a lot more errors after the main blocking bigquery error that we keep seeing. I'm sure this isn't the best solution.

When I run system-tests I get a bunch of datastore failures/errors. Are there some fixtures I need to setup to get those to work?

I can fix this up so we can merge and go from there though.

dhermes · 2016-08-01T18:57:47Z

There is nothing wrong with the retry method, it just looks copy-pasted. You'd probably rather use it inside the test case than on the test case:

deftest_case_for_thing(self):
# do some set-up@retry(...)defthing_that_is_eventually_consistent():
global_object.some_method(args)
thing_that_is_eventually_consistent()

daspecster · 2016-08-01T20:31:01Z

@dhermes LMKWYT.

Sign in to view

+
+ @retry(Forbidden, tries=3, delay=30)
+ def update_dataset():
+ dataset.update()


dhermes · 2016-08-01T21:34:09Z

Thanks for taking the initiative here.

Searching
https://github.com/GoogleCloudPlatform/gcloud-python/search?q=eventual+consistency&type=Issues
https://github.com/GoogleCloudPlatform/gcloud-python/search?q=eventually+consistent&type=Issues
we should unify all the issues / code that have consistency implications.

In particular, we should discuss this in #1619

Sign in to view

+
+ # We should try and keep `delay` and tries as low as possible to
+ # reduce test running time. tries=3 and delay=30 worked consistenly
+ # when updating a dataset.


daspecster · 2016-08-02T19:47:54Z

@dhermes here's the class version of the retry decorator. LMWYT.

Sign in to view

+ exception = None
+ tries = None
+ delay = None
+ backoff = None


Sign in to view

+
+ time.sleep(self.delay)
+ tries_counter -= 1
+ self.delay *= self.backoff


Sign in to view

+
+ # We need to wait for the changes in the dataset to propgate and be
+ # eventually consistent. The alternative outcome is a 403 Forbidden
+ # response from upstream.


Sign in to view

+ tries_counter = self.tries
+ exception = self.exception
+ delay = self.delay
+ backoff = self.backoff


dhermes · 2016-08-02T22:29:35Z

LGTM pending Travis

Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com> Co-authored-by: Chalmer Lowe <chalmerlowe@google.com>

daspecster added testing api: bigquery Issues related to the BigQuery API. do not merge Indicates a pull request not ready for merge, due to either quality or timing. labels Aug 1, 2016

daspecster assigned tseaver and dhermes Aug 1, 2016

googlebot added the cla: yes This human has signed the Contributor License Agreement. label Aug 1, 2016

dhermes reviewed Aug 1, 2016
View reviewed changes

Comment threadsystem_tests/retry.py Outdated
from functools import wraps

def retry(ExceptionToCheck, tries=4, delay=3, backoff=2, logger=None):
This comment was marked as spam.
Sign in to view

Playing with retries.
2dc08cf

daspecster force-pushed the retry-system3-tests branch from 067e147 to 2dc08cfCompare August 1, 2016 20:30

daspecster removed the do not merge Indicates a pull request not ready for merge, due to either quality or timing. label Aug 1, 2016

dhermes reviewed Aug 1, 2016
View reviewed changes

Comment threadsystem_tests/bigquery.py

@retry(Forbidden, tries=3, delay=30)
def update_dataset():
dataset.update()
This comment was marked as spam.
Sign in to view

Feedback updates.
1de7a5c

dhermes reviewed Aug 1, 2016
View reviewed changes

This was referenced Aug 1, 2016
Add Group support to the Monitoring API #2035
Merged
Consolidate backoff code in system tests #2042
Closed

Switch retry to Class style decorator.
49b3027

dhermes reviewed Aug 2, 2016
View reviewed changes

Comment threadsystem_tests/retry.py Outdated
exception = None
tries = None
delay = None
backoff = None
This comment was marked as spam.
Sign in to view

Fix tries counter and change comments.
c532444

dhermes reviewed Aug 2, 2016
View reviewed changes

Comment threadsystem_tests/retry.py Outdated

time.sleep(self.delay)
tries_counter -= 1
self.delay *= self.backoff
This comment was marked as spam.
Sign in to view
This comment was marked as spam.
Sign in to view

daspecster force-pushed the retry-system3-tests branch from 874cf22 to c532444Compare August 2, 2016 20:36

dhermes reviewed Aug 2, 2016
View reviewed changes

Comment threadsystem_tests/bigquery.py Outdated

# We need to wait for the changes in the dataset to propgate and be
# eventually consistent. The alternative outcome is a 403 Forbidden
# response from upstream.
This comment was marked as spam.
Sign in to view

Scoping the settings.
f33b2ea

daspecster force-pushed the retry-system3-tests branch from 090b734 to f33b2eaCompare August 2, 2016 20:46

dhermes reviewed Aug 2, 2016
View reviewed changes

Change comments, Retry class properties.
e06815d

daspecster merged commit 2724805 into googleapis:masterAug 2, 2016

This was referenced Aug 3, 2016
Prepare 0.18.0 release #2044
Closed
Prepare 0.18.0 release. #2048
Merged

dhermes mentioned this pull request Aug 3, 2016
Consolidate retry logic #2050
Merged

daspecster deleted the retry-system3-tests branch January 24, 2017 15:18

theacodes unassigned dhermes Sep 28, 2018

parthea pushed a commit that referenced this pull request Mar 6, 2026
chore(main): release 3.27.0 (#2040)
8f2abe3
Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com> Co-authored-by: Chalmer Lowe <chalmerlowe@google.com>

parthea pushed a commit that referenced this pull request Apr 1, 2026
feat: Can pivot unordered, unindexed dataframe (#2040)
33845f8

		from functools import wraps


		def retry(ExceptionToCheck, tries=4, delay=3, backoff=2, logger=None):

Uh oh!

Conversation

daspecster commented Aug 1, 2016

Uh oh!

This comment was marked as spam.

Uh oh!

daspecster commented Aug 1, 2016

Uh oh!

dhermes commented Aug 1, 2016

Uh oh!

daspecster commented Aug 1, 2016

Uh oh!

This comment was marked as spam.

Uh oh!

dhermes commented Aug 1, 2016

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

daspecster commented Aug 2, 2016

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

dhermes commented Aug 2, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants