Skip to content

feat(pubsub): add message and acknowledgment batching - #338

Closed
sigurdm wants to merge 2 commits into
pubsub-retry-batchingfrom
pubsub-topic-batching
Closed

feat(pubsub): add message and acknowledgment batching#338
sigurdm wants to merge 2 commits into
pubsub-retry-batchingfrom
pubsub-topic-batching

Conversation

@sigurdm

@sigurdm sigurdm commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

This is Part 2 of 3 in a stacked set of PRs:

  1. feat(pubsub): add message and acknowledgment batching with retry infrastructure #292: Core retry and batching infrastructure (RetrySettings, Batcher, BatchingSettings)
  2. feat(pubsub): add message and acknowledgment batching #338 (this PR): Topic publishing batching & lifecycle management (PublishSettings, Topic.publish, Topic.close)
  3. feat(pubsub): add resilient parallel streaming pull #339: Subscription batching, resilient parallel streaming pull, and lifecycle management (AckSettings, Subscription.streamingPull, Subscription.close)

Key Features in Part 2

  • Topic Batching: Added PublishSettings controlling background batching (BatchingSettings) and unary retry configuration (RetrySettings) for Topic.publish.
  • Accurate Byte Counting: Computes payload and attribute key/value byte overhead when batching messages against byte limits.
  • Completer Safety: Ensures all completers in a batch complete or fail exactly once even on mismatched server responses or backend errors.
  • Lifecycle Management: Async Topic.close() that flushes pending batches and awaits in-flight RPCs, preventing subsequent publishes with StateError.
  • Instance Settings Preservation: Topic.create() returns this rather than constructing a new default instance, preserving configured publishSettings.
  • String Representations: Informative Message.toString() and ReceivedMessage.toString().

TAG=agy
CONV=bc7c6164-e9cb-4a0f-b326-7752c814e1ce


Stack created with GitHub Stacks CLIGive Feedback 💬

@sigurdm

sigurdm commented Sep 7, 2026

Copy link
Copy Markdown
Contributor Author

/gcbrun

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces message batching and retry settings (PublishSettings) for Topic publishing, along with lifecycle improvements such as an asynchronous close() method and safer batch completion handling. Review feedback suggests optimizing the message size calculation by avoiding utf8.encode() allocations to reduce memory churn, and simplifying a redundant Future.value return in an async method.

Comment thread pkgs/google_cloud_pubsub/lib/src/topic.dart
String topic,
List<Message> messages,
) async {
if (messages.isEmpty) return Future.value(<String>[]);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

In an 'async' function, returning 'Future.value(...)' is redundant because any returned value is automatically wrapped in a 'Future'. You can simplify this by returning the list literal directly.

Suggested change
if (messages.isEmpty) return Future.value(<String>[]);
if (messages.isEmpty) return <String>[];

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed: Removed the redundant Future.value(...) wrapping in publishMessages (and similarly in acknowledge and modifyAckDeadline).

@sigurdm
sigurdm force-pushed the pubsub-topic-batching branch from e907d4a to 9e88ae0 Compare September 8, 2026 08:23
@sigurdm
sigurdm force-pushed the pubsub-retry-batching branch from 5f98a79 to 791b1c7 Compare September 8, 2026 08:23
@sigurdm

sigurdm commented Sep 8, 2026

Copy link
Copy Markdown
Contributor Author

/gcbrun

@sigurdm
sigurdm force-pushed the pubsub-topic-batching branch from 9e88ae0 to 3159630 Compare September 8, 2026 09:32
@sigurdm

sigurdm commented Sep 8, 2026

Copy link
Copy Markdown
Contributor Author

/gcbrun

Add exponential backoff retry settings (RetrySettings) with jitter,
totalTimeout support, and transient error classification (isRetryable)
to package:google_cloud_pubsub.

TAG=agy
CONV=bc7c6164-e9cb-4a0f-b326-7752c814e1ce
@sigurdm
sigurdm force-pushed the pubsub-topic-batching branch from 3159630 to b6398bc Compare September 8, 2026 11:05
@sigurdm
sigurdm force-pushed the pubsub-retry-batching branch from 791b1c7 to 5ad97bc Compare September 8, 2026 11:05
@sigurdm sigurdm changed the title feat(pubsub): add background batching and lifecycle management for Topic publishing feat(pubsub): add message and acknowledgment batching Sep 8, 2026
@sigurdm

sigurdm commented Sep 8, 2026

Copy link
Copy Markdown
Contributor Author

/gcbrun

Add request batching infrastructure (Batcher, BatchingSettings),
background message batching and lifecycle management for Topic
publishing (PublishSettings), background acknowledgment and deadline
modification batching for Subscriptions (AckSettings), and batch
publishing via PubSub.publishMessages.

TAG=agy
CONV=bc7c6164-e9cb-4a0f-b326-7752c814e1ce
@sigurdm
sigurdm force-pushed the pubsub-topic-batching branch from b6398bc to 1379642 Compare September 8, 2026 11:22
@sigurdm
sigurdm force-pushed the pubsub-retry-batching branch from 5ad97bc to 64e494c Compare September 8, 2026 11:58
@sigurdm

sigurdm commented Sep 8, 2026

Copy link
Copy Markdown
Contributor Author

Collapsed into PR #292 (retries + batching combined).

@sigurdm sigurdm closed this Sep 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant