fix: retry 'job exceeded rate limits' for DDL queries

[kiraksi]

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

• Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
• Ensure the tests and linter pass
• Code coverage does not decrease (if any source code was changed)
• Appropriate docs were updated (if necessary)

Fixes#1790 🦕

[tswast]

Thanks!

I have some feedback regarding testing. I think it may make more sense to follow the pattern I use here, instead:

python-bigquery/tests/unit/test__job_helpers.py

Lines 388 to 447 in 1f96439

	deftest_query_and_wait_retries_job():
	freezegun.freeze_time(auto_tick_seconds=100)
	client=mock.create_autospec(Client)
	client._call_api.__name__="_call_api"
	client._call_api.__qualname__="Client._call_api"
	client._call_api.__annotations__= {}
	client._call_api.__type_params__= ()
	client._call_api.side_effect= (
	google.api_core.exceptions.BadGateway("retry me"),
	google.api_core.exceptions.InternalServerError("job_retry me"),
	google.api_core.exceptions.BadGateway("retry me"),
	{
	"jobReference": {
	"projectId": "response-project",
	"jobId": "abc",
	"location": "response-location",
	},
	"jobComplete": True,
	"schema": {
	"fields": [
	{"name": "full_name", "type": "STRING", "mode": "REQUIRED"},
	{"name": "age", "type": "INT64", "mode": "NULLABLE"},
	],
	},
	"rows": [
	{"f": [{"v": "Whillma Phlyntstone"}, {"v": "27"}]},
	{"f": [{"v": "Bhetty Rhubble"}, {"v": "28"}]},
	{"f": [{"v": "Phred Phlyntstone"}, {"v": "32"}]},
	{"f": [{"v": "Bharney Rhubble"}, {"v": "33"}]},
	],
	},
	)
	rows=_job_helpers.query_and_wait(
	client,
	query="SELECT 1",
	location="request-location",
	project="request-project",
	job_config=None,
	page_size=None,
	max_results=None,
	retry=retries.Retry(
	lambdaexc: isinstance(exc, google.api_core.exceptions.BadGateway),
	multiplier=1.0,
	).with_deadline(
	200.0
	), # Since auto_tick_seconds is 100, we should get at least 1 retry.
	job_retry=retries.Retry(
	lambdaexc: isinstance(exc, google.api_core.exceptions.InternalServerError),
	multiplier=1.0,
	).with_deadline(600.0),
	)
	assertlen(list(rows)) ==4
	# For this code path, where the query has finished immediately, we should
	# only be calling the jobs.query API and no other request path.
	request_path="/projects/request-project/queries"
	forcallinclient._call_api.call_args_list:
	_, kwargs=call
	assertkwargs["method"] =="POST"
	assertkwargs["path"] ==request_path

It may make sense to also add a test for just the predicate itself:

python-bigquery/tests/unit/test_retry.py

Line 109 in 1f96439

deftest_DEFAULT_JOB_RETRY_predicate():

[tswast]

This is an interesting one. Not sure why we'd be doing a POST to this path. That'd indicate that we are doing a jobs insert call, which really shouldn't be happening.

[kiraksi]

Added a todo statement for the fix in #1797

[tswast]

Thanks!

[nlenepveu]

Thanks @kiraksi!

[tswast]

Import modules not classes / methods.

https://google.github.io/styleguide/pyguide.html#22-imports