Uh oh!
There was an error while loading. Please reload this page.
feat: add flow control for message publishing - #96
Conversation
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
I added support for multithreaded Things left:
|
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
plamut
commented
May 28, 2020
@pradn Do you have anything to add here, or are we good to merge? |
| error_msg = "Flow control limits too low for the message - {}.".format( | ||
| load_info | ||
| ) | ||
| raise exceptions.PermanentlyBlockedError(error_msg) |
There was a problem hiding this comment.
"WouldPermanentlyBlockError" makes more sense, because it's not already permanently blocked.
Even better, I think we can simply omit the details about blocking and say something more generic: FlowControlMessageTooLargeError
There was a problem hiding this comment.
Makes sense, I will rename the exception.
There was a problem hiding this comment.
I eventually opted for the generic FlowControlLimitError for a few reasons:
FlowControlMessageTooLargeErrorcould be confusing, as there already exists a similarly namedMessageTooLargeErrorfor the messages that would exceed the limits on the backend.- A message could block permanently for reasons other than its size, e.g. if flow control's max messages is set to 0 or some other non-sensible value. Mentioning the size would actually be misleading in such case, but a generic
FlowControlLimitsErrorwould semantically still be correct. - One less exception type to keep track of.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
@pradn If there's nothing else left, please just give an approval to unblock the merge, thanks! :) |
## [1.6.0](https://www.github.com/googleapis/python-pubsub/compare/v1.5.0...v1.6.0) (2020-06-09) ### Features * Add flow control for message publishing ([#96](https://www.github.com/googleapis/python-pubsub/issues/96)) ([06085c4](https://www.github.com/googleapis/python-pubsub/commit/06085c4083b9dccdd50383257799904510bbf3a0)) ### Bug Fixes * Fix PubSub incompatibility with api-core 1.17.0+ ([#103](https://www.github.com/googleapis/python-pubsub/issues/103)) ([c02060f](https://www.github.com/googleapis/python-pubsub/commit/c02060fbbe6e2ca4664bee08d2de10665d41dc0b)) ### Documentation - Clarify that Schedulers shouldn't be used with multiple SubscriberClients ([#100](#100)) ([cf9e87c](cf9e87c)) - Fix update subscription/snapshot/topic samples ([#113](#113)) ([e62c38b](e62c38b)) ### Internal / Testing Changes - Re-generated service implementaton using synth: removed experimental notes from the RetryPolicy and filtering features in anticipation of GA, added DetachSubscription (experimental) ([#114](#114)) ([0132a46](0132a46)) - Incorporate will_accept() checks into publish() ([#108](#108)) ([6c7677e](6c7677e))
Closes#16.
Closes#72.
This PR adds flow control settings to publisher client, similar to what was added to the Java client recently.
The PR does not interfere with existing publish logic (sequencers, ordering keys, batching...). Instead, it acts like a valve placed in front of the publish pipeline, and taking the configured action if it detects excessive message flow.
Things left to do / discuss
FlowControlsettings are placed underPublisherOptions, and notBatchSettings. They fit there much more naturally IMO, but I can change this.FlowControlvalues (the thresholds, specifically)? I set the message count and byte size to 10x the defaults in BatchSettings, would that be fine for most users?PR checklist