Skip to content

fix(pubsub): include request overhead when computing publish batch size overflow - #9911

Merged
plamut merged 5 commits into
googleapis:masterfrom
plamut:iss-7108
Dec 5, 2019
Merged

fix(pubsub): include request overhead when computing publish batch size overflow#9911
plamut merged 5 commits into
googleapis:masterfrom
plamut:iss-7108

Conversation

@plamut

@plamutplamut commented Dec 3, 2019•
edited
Loading

Copy link
Copy Markdown
Contributor

Fixes#7108.

This PR fixes the logic that computes the publish batch size overflow, taking the total request message size overhead into account. The improved logic prevents the server-side errors simular to the following:

google.api_core.exceptions.InvalidArgument: 400 The value for request_size is too large. You passed 10000096 in the request, but the maximum value is 10000000.

How to test

(see also the system test in this PR)

  • Create a pubisher client with BatchSettings.max_bytes substantially larger than 10_000_000, and BatchSettings.max_latency to one second (so that the publish autocommit does not kick in too soon).
  • Quickly publish a few sizable messages to a topic. Their total size should slightly exceed 10_000_000 bytes.

Actual result (before the fix):
The backend responds with a "400 InvalidArgument" error.

Expected result (after the fix):
All messages are successfully published (the code splits them into multiple publish batches).

PR checklist

  • Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
  • Ensure the tests and linter pass
  • Code coverage does not decrease (if any source code was changed)
  • Appropriate docs were updated (if necessary)

@plamut
Clarify the description of BatchSettings.max_bytes
550b0c6
@plamut
Include overhead in batch overflow calculation
2000193
The maximum allowed size for a PublishRequest on the backend is lower
than a mere sum of the byte sizes of individual messages.
This commit adjusts the batch size overflow calculation to account for
this overhead. It also caps the effective maximum BatchSetting.max_size
value to 10_000_000 bytes (the limit on the backend).
(credit also to GitHub @relud for outlining the main idea first in the
issue description)
@plamutplamut added the api: pubsub Issues related to the Pub/Sub API. label Dec 3, 2019
@plamut
plamut requested a review from pradnDecember 3, 2019 22:35
@googlebotgooglebot added the cla: yes This human has signed the Contributor License Agreement. label Dec 3, 2019
software-dov
software-dov reviewed Dec 3, 2019
View reviewed changes
Comment threadpubsub/google/cloud/pubsub_v1/publisher/_batch/thread.py Outdated
software-dov
software-dov reviewed Dec 3, 2019
View reviewed changes
Comment threadpubsub/tests/system.py Outdated
Comment threadpubsub/tests/unit/pubsub_v1/publisher/batch/test_thread.py Outdated
Comment threadpubsub/tests/unit/pubsub_v1/publisher/batch/test_thread.py Outdated
Comment threadpubsub/google/cloud/pubsub_v1/publisher/_batch/thread.py Outdated
Comment threadpubsub/tests/unit/pubsub_v1/publisher/batch/test_thread.py Outdated
Comment threadpubsub/tests/unit/pubsub_v1/publisher/batch/test_thread.py
software-dov
software-dov reviewed Dec 4, 2019
View reviewed changes

@software-dovsoftware-dov left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Judgement call on your part to make a distinct exception class for test_publish_single_message_size_exceeeds_server_size_limit or leave as is, otherwise LGTM.

@plamut

plamut commented Dec 4, 2019

Copy link
Copy Markdown
ContributorAuthor

@software-dov Thanks for the quick review, addressed the last outstanding comment.

software-dov
software-dov approved these changes Dec 4, 2019
View reviewed changes

@software-dovsoftware-dov left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@plamut
plamut merged commit 0699ba6 into googleapis:masterDec 5, 2019
@plamut
plamut deleted the iss-7108 branch December 5, 2019 07:05
This was referenced Jan 29, 2020
Merged
Closed
Closed
parthea pushed a commit that referenced this pull request Mar 2, 2026
@plamut
fix(pubsub): include request overhead when computing publish batch si…
806fdd1
…ze overflow (#9911)
* Clarify the description of BatchSettings.max_bytes
* Include overhead in batch overflow calculation
The maximum allowed size for a PublishRequest on the backend is lower
than a mere sum of the byte sizes of individual messages.
This commit adjusts the batch size overflow calculation to account for
this overhead. It also caps the effective maximum BatchSetting.max_size
value to 10_000_000 bytes (the limit on the backend).
(credit also to GitHub @relud for outlining the main idea first in the
issue description)
* Access settings inside Batch in a consistent way.
* Cleanup and refactor a few code snippets
* Raise more specific error if message too large
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pradnpradnAwaiting requested review from pradn
@anguillanneufanguillanneufAwaiting requested review from anguillanneuf
1 more reviewer
@software-dovsoftware-dovsoftware-dov approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

api: pubsubIssues related to the Pub/Sub API.cla: yesThis human has signed the Contributor License Agreement.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

PubSub: check batch max bytes against request byte size

3 participants

@plamut@software-dov@googlebot