pubsub - modify_ack_deadline does not work, message gets redelivered after 10 minutes always

[sachin-shetty]

1. Specify the API at the beginning of the title (for example, "BigQuery: ...")
   General, Core, and Other are also allowed as types
   google-cloud-pubsub - 0.29.2
2. OS type and version
   Linux 4.4.0-104-generic Set up local machine test automation with tox. #127-Ubuntu SMP Mon Dec 11 12:16:42 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
3. Python version and virtual environment information python --version
   Python 2.7.12
4. google-cloud-python version pip show google-cloud, pip show google-<service> or pip freeze
   google-cloud: 0.30.0
   google-cloud-pubsub - 0.29.2
5. Stacktrace if available
6. Steps to reproduce
   I need to have very long ack deadlines since our consumer process could be running for long time per message. No matter what I set as ack_deadline_seconds in subscription definition, or call subscriber.modify_ack_deadline every 1 minute, the message gets redelivered in exact ten minutes.
7. Code example

I have a test case with a simple subscriber::

subscriber.create_subscription(sname, tname, ack_deadline_seconds=120)
subscription = subscriber.subscribe(
 sname #, flow_control=flow_control
 )
future = subscription.open(callback)

def callback(pubsub_message):
 logging.warn("Received message: %s %s at %s", pubsub_message.message_id, pubsub_message.data, time.strftime('%Y-%m-%d %H:%M:%S', time.localtime(time.time())))
 i = 0
 while i < 480:
 i = i + 1
 time.sleep(60)
 subscriber.modify_ack_deadline(sname, [pubsub_message._ack_id], 120)
 logging.warn("Renewed: %s %s %s at %s", i, pubsub_message.message_id, pubsub_message.data, time.strftime('%Y-%m-%d %H:%M:%S', time.localtime(time.time())))

 logging.warn("Ack: %s %s at %s", pubsub_message.message_id, pubsub_message.data, time.strftime('%Y-%m-%d %H:%M:%S', time.localtime(time.time())))
 pubsub_message.ack()

enabling DEBUG logging, I see ack messages every few seconds
DEBUG:google.cloud.pubsub_v1.subscriber.policy.base:The current p99 value is 10 seconds.
DEBUG:google.cloud.pubsub_v1.subscriber.policy.base:Renewing lease for 5 ack IDs.
DEBUG:google.cloud.pubsub_v1.subscriber.policy.base:Snoozing lease management for 7.362112 seconds.
DEBUG:google.cloud.pubsub_v1.subscriber._consumer:Sending request:

but message gets redelivered in 10 minutes.

Using GitHub flavored markdown can help make your request clearer.
See: https://guides.github.com/features/mastering-markdown/

[dhermes]

@sachin-shetty Could you try with google-cloud-python==0.30.0. There were a few bugs in flow control fixed between 0.29.2 and 0.29.4 (though it doesn't seem like that would be causing the issue you describe.

[dhermes]

Also, I'm not 100% sure how the line

subscriber.modify_ack_deadline(sname, [pubsub_message._ack_id], 120)

will interact with the lease management thread (i.e. the one that is logging "Renewing lease for ...") but that seems to be the problem.

The "expected" usage pattern would be that you assume your callback only need the state of the Message it gets:

pubsub_message.modify_ack_deadline(120.0)

[sachin-shetty]

Hi @dhermes

Tried new version as well as pubsub_message.modify_ack_deadline. Same result. Message gets re-delivered in exactly 10 minutes even after calling pubsub_message.modify_ack_deadline(120) every minute.

and even though pubsub_message.modify_ack_deadline(120) is called, debug logs still show 10s

DEBUG:google.cloud.pubsub_v1.subscriber.policy.base:The current p99 value is 10 seconds.
DEBUG:google.cloud.pubsub_v1.subscriber.policy.base:Renewing lease for 4 ack IDs.
DEBUG:google.cloud.pubsub_v1.subscriber.policy.base:Snoozing lease management for 8.668086 seconds.
DEBUG:google.cloud.pubsub_v1.subscriber._consumer:Sending request:
modify_deadline_seconds: 10
modify_deadline_seconds: 10
modify_deadline_seconds: 10
modify_deadline_seconds: 10
modify_deadline_ack_ids: "fjY5RUFeQBJMPQxESVMrQwsqWBFOBCEhPjA-RVNEUAYWLF1GSFE3GQhoUQ5PXiM_NSAoRRIAIG8QLUJaGmJoXFx1B1ALGXV6MCFrU0cACERZfndrOTNpWF9xAFQEHnR7YHRjWjvkncSLwfJoZh89WxJLLD4"
modify_deadline_ack_ids: "fjY5RUFeQBJMPQxESVMrQwsqWBFOBCEhPjA-RVNEUAYWLF1GSFE3GQhoUQ5PXiM_NSAoRRIAIG8QLUJaGmJoXFx1B1ALGXV6MCFrU0cACEVZfndrOTNpWF9xDlAEGnV9aXxpUjvkncSLwfJoZh89WxJLLD4"
modify_deadline_ack_ids: "fjY5RUFeQBJMPQxESVMrQwsqWBFOBCEhPjA-RVNEUAYWLF1GSFE3GQhoUQ5PXiM_NSAoRRIAIG8QLUJaGGJoXFx1B1ALGXV6MHdjDhUBCERZfndrOTNpWF9wA1UKHXt8Yn1tXzvkncSLwfJoZh89WxJLLD4"
modify_deadline_ack_ids: "fjY5RUFeQBJMPQxESVMrQwsqWBFOBCEhPjA-RVNEUAYWLF1GSFE3GQhoUQ5PXiM_NSAoRRIAIG8QLUJaGGJoXFx1B1ALGXV6MHdjDhUBCEVZfndrOTNpWF9xAVELGXB1Z3RvUjvkncSLwfJoZh89WxJLLD4"

[sachin-shetty]

I tried commenting _start_lease_worker() in subscriber/policy/thread.py, that stopped the lease worker from forcing a 10s ack deadline.

but the message sent to server by the subscriber.modify_ack_deadline(sname, [pubsub_message._ack_id], 120) still shows as 10s in the debug logs. I see that 10s is hardcoded in base.py

Is there a version I can downgrade to to get this to work, I need longer message ack time.

[dhermes]

I'm fairly certain this is a bug. The lease maintenance thread is trying to make sure a deadline gets extended for any messages that the subscriber is still responsible for, but in doing this extension it actually reduces the lease time for your message (this is a mistake in your and my opinion).

There is no equivalent high-throughput / highly concurrent implementation you can downgrade to. Here is every Pub / Sub release. Version 0.28.0 was when the rewrite happened, so downgrading to 0.27.0 or earlier you won't be subject to this bug, but the library was totally different at that point. (We don't have any published docs for that version of the library, but I'd be happy to work with you to show you how to build the docs locally.)

[sachin-shetty]

Thankyou @dhermes. Do you think a potential fix would be available anytime soon? I am trying to figure out if I should build application level logic to handle this or wait for a fix.

[MaxDesiatov]

actually 0.27.0 docs are still available at http://temp.theadora.io/google-cloud-python-pubsub-0-27/pubsub/usage.html

[dhermes]

@explicitcall Thanks for the link, but be aware that the temp is not a typo in that domain (it is a staging domain used by @jonparrott).

[MaxDesiatov]

@dhermes are there any plans to host documentation for multiple versions on the official docs site? That would be greatly appreciated.

[dhermes]

@explicitcall No concrete plans, but I have ideas in my head once we can split google-cloud-python into N repositories (one for each subpackage). It's a difficult task right now that would require a HUGE investment of time making modifications to the RTD Sphinx theme.