fix: Stop using api_core default timeouts in publish since they are broken

[mukund-ananthu]

Change in behavior

Default publish timeout behavior:

Pre-change: 60s timeout for initial RPC, 0s timeout per retry, 600s total timeout
Post-change:60s timeout for initial RPC, 60s timeout per retry, 600s total timeout

Change

Only non-test / non sample(i.e functional) usage found for PublisherOptions.timeout is here:

python-pubsub/google/cloud/pubsub_v1/publisher/client.py

Lines 440 to 441 in f648f65

	iftimeoutisgapic_v1.method.DEFAULT: # if custom timeout not passed in
	timeout=self.publisher_options.timeout

This change in usage is intended.

Call stack:

python-pubsub/google/cloud/pubsub_v1/publisher/client.py

Line 299 in f648f65

timeout: "types.OptionalTimeout"=gapic_v1.method.DEFAULT,

invokes one of the sequencers(ordered or unordered)

python-pubsub/google/cloud/pubsub_v1/publisher/client.py

Lines 477 to 479 in f648f65

	future=sequencer.publish(
	wrapper=wrapper, retry=retry, timeout=timeout
	)

which then create a batch with this timeout

python-pubsub/google/cloud/pubsub_v1/publisher/_sequencer/ordered_sequencer.py

Lines 255 to 263 in f648f65

	returnself._client._batch_class(
	client=self._client,
	topic=self._topic,
	settings=self._client.batch_settings,
	batch_done_callback=self._batch_done_callback,
	commit_when_full=False,
	commit_retry=commit_retry,
	commit_timeout=commit_timeout,
	)

which uses it when committing the batch / publishing the messages:

python-pubsub/google/cloud/pubsub_v1/publisher/_batch/thread.py

Lines 323 to 328 in f648f65

	response=self._client._gapic_publish(
	topic=self._topic,
	messages=[wrapper.messageforwrapperinself._message_wrappers],
	retry=self._commit_retry,
	timeout=self._commit_timeout,
	)

and invokes gapic_publish:

python-pubsub/google/cloud/pubsub_v1/publisher/client.py

Lines 289 to 291 in f648f65

	def_gapic_publish(self, *args, **kwargs) ->"pubsub_types.PublishResponse":
	"""Call the GAPIC public API directly."""
	returnsuper().publish(*args, **kwargs)

which then uses it in the RPC call:

python-pubsub/google/pubsub_v1/services/publisher/client.py

Lines 1067 to 1072 in f648f65

	response=rpc(
	request,
	retry=retry,
	timeout=timeout,
	metadata=metadata,
	)

that is obtained by wrapping the transport base publish call:

python-pubsub/google/pubsub_v1/services/publisher/client.py

Line 1055 in f648f65

rpc=self._transport._wrapped_methods[self._transport.publish]

The wrapper uses the timeout when invoking the GapicCallable:

https://github.com/googleapis/python-api-core/blob/b1fae31c8c71ed65ad22a415dab2b54720c3d4ba/google/api_core/gapic_v1/method.py#L245-L252

which currently overrides float and int timeouts to use TimeToDeadlineTimeout:

https://github.com/googleapis/python-api-core/blob/b1fae31c8c71ed65ad22a415dab2b54720c3d4ba/google/api_core/gapic_v1/method.py#L112-L113

which is incorrect(why its incorrect is mentioned in the attached bug)

Fixes#1325 🦕

[mukund-ananthu]

@beltran PTAL

[kamalaboulhosn]

I'm a little nervous about changing the default behavior from 60s to 5s for the initial request. Prior to the change that resulted in unexpected backoff behavior, what was the initial value? If it was 5, this could be considered a return to the "correct" value, but if this is the first time we are introducing it as 5, I fear we could break behavior for people.

[mukund-ananthu]

Prior to the change that resulted in unexpected backoff behavior, what was the initial value?

Initial value = 60s

Pre api_core change:

initial RPC value = 60s
retry RPC value = 60s
total timeout across retries = 600s

Post api_core change:

initial RPC value = 60s
retry RPC value = 0s
total timeout across retries = 600s

Post this PR:

Initial RPC value = 5s
retry RPC value = [5, 60] with multiplier=1.3
total timeout across retries = 600s

but if this is the first time we are introducing it as 5, I fear we could break behavior for people.

• This is the first time we have 5s as the initial RPC value and yes, this is a breaking change.
• Hypothetically, even if the initial value was 5s(it isn't), then changed to 60s in 2022, I'm not sure how much weight we must give to the fact that it was 5s initially, since its been a few years with this new behavior, i.e, this would still be a breaking change for those who onboarded between 2022 and now.

Code / Links for easy reference

• Both pre and post api_core change, we passed indefault_timeout=60 from Pub/Sub to api_core library.
• Pre api_core change, this was interpreted as ConstantTimeout(60)
• Post api_core change, this is interpreted as TimeToDeadlineTimeout(60)
• This interpretation of default_timeout=60.0 in api_core happens here
• Post PR change, timeout decorator applied would be ExponentialTimeout instead of TimeToDeadlineTimeout(60)
• Definition of different timeout options can be found here