PubSub Publisher by lukesneeringer · Pull Request #3463 · googleapis/google-cloud-python

lukesneeringer · 2017-06-01T14:40:00Z

This is what I assert to be a mostly complete publisher for Pub/Sub (minus error checking, which is a pretty big caveat, and tests, which is also a pretty big caveat). I would like feedback on this much before I move forward, since (as we all know) multiprocessing is tricky.

I am about to add some notes inline at specific places, but obviously feel free to comment on any of it.

…python into pubsub

Sign in to view

+ def __init__(self, client, topic, settings, autocommit=True):
+ self._client = client
+
+ # Create a namespace that is owned by the client manager; this


Sign in to view

+ raise Exception('Empty queue')
+ response = self._client.api.publish(self._.topic, self._.messages)
+
+ # FIXME (lukesneeringer): Check for failures; retry.


Sign in to view

@@ -1,4 +1,4 @@
-# Copyright 2016 Google Inc.


Sign in to view

+ Args:
+ message_id (str): The message ID, as a string.
+ """
+ for callback in self._callbacks:


Sign in to view

+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+def retry(func, delay=0, count=0, err=None, **kwargs):


Sign in to view

+__VERSION__ = pkg_resources.get_distribution('google-cloud-pubsub').version
+
+
+class SubscriberClient(subscriber_client.SubscriberClient):


Sign in to view

+
+__all__ = (
+ 'PublisherClient',
+ 'SubscriberClient',


Sign in to view

+ :class:`google.cloud.pubsub_v1.PublisherClient`.
+
+ Args:
+ client (:class:`google.cloud.pubsub_v1.PublisherClient`): The


Sign in to view

+ inferred from this.
+ topic (str): The topic. The format for this is
+ ``projects/{project}/topics/{topic}``.
+ settings (:class:`google.cloud.pubsub_v1.types.Batching`): The


Sign in to view

+ # Create a namespace that is owned by the client manager; this
+ # is necessary to be able to have these values be communicable between
+ # processes.
+ self._ = self.manager.Namespace()


Sign in to view

+ self._.messages = self.manager.list()
+ self._.message_ids = self.manager.dict()
+ self._.settings = settings
+ self._.status = 'accepting messages'


Sign in to view

+ Future: An object conforming to the ``concurrent.futures.Future``
+ interface.
+ """
+ # Sanity check: Is the data being sent as a bytestring?


Sign in to view

+ return f
+
+
+# Make a fake batch. This is used by the client to do single-op checks


Sign in to view

+ def commit(self):
+ """Actually publish all of the messages on the active batch.
+
+ This moves the batch out from being the active batch to an in-flight


Sign in to view

+ Args:
+ message_id (str): The message ID, as a string.
+ """
+ for callback in self._callbacks:


Sign in to view

+import functools
+
+
+def add_methods(SourceClass, blacklist=()):


Sign in to view

+ :class:`google.cloud.pubsub_v1.PublisherClient`.
+
+ Args:
+ client (:class:`google.cloud.pubsub_v1.PublisherClient`): The


Sign in to view

+ message to be published is received; subsequent messages are added to
+ that batch until the process of actual publishing _starts_.
+
+ Once this occurs, any new messages sent to ``publish`` open a new batch.


Sign in to view

+ Future: An object conforming to the ``concurrent.futures.Future``
+ interface.
+ """
+ # Sanity check: Is the data being sent as a bytestring?


Sign in to view

+ :class:`~gapic.pubsub.v1.publisher_client.PublisherClient`.
+ Generally, you should not need to set additional keyword arguments.
+ """
+ _gapic_class = publisher_client.PublisherClient


Sign in to view

+ """
+ # If there is no matching batch yet, then potentially create one
+ # and place it on the batches dictionary.
+ if self._batches.get(topic, FAKE).status != 'accepting messages':


Sign in to view

+
+import queue
+import uuid
+import time


Also rename base.BaseBatch to base.Batch.

theacodes

Looks mostly good, some small things and nits.

Sign in to view

+ """
+ raise NotImplementedError
+
+ class Status(object):


Sign in to view

@@ -0,0 +1,130 @@
+# Copyright 2017, Google Inc. All rights reserved.


Sign in to view

+ """
+ raise NotImplementedError
+
+ @property


Sign in to view

+ # commit when the max latency is reached.
+ self._thread = None
+ if autocommit and self._settings.max_latency < float('inf'):
+ self._thread = threading.Thread(target=self.monitor)


Sign in to view

+ version, which calls this one.
+ """
+ # Update the status.
+ self._status = 'in-flight'


Sign in to view

+ if isinstance(v, six.binary_type):
+ attrs[k] = v.decode('utf-8')
+ continue
+ raise TypeError('All attributes being published to Pub/Sub must '


Sign in to view

+ pass
+
+
+class TimeoutError(GoogleAPIError):


Sign in to view

+ return False
+
+ def running(self):
+ """Publishes in Pub/Sub currently may not be canceled.


Sign in to view

+
+ # Wait a little while and try again.
+ time.sleep(_wait)
+ return self.exception(


Sign in to view

+)
+
+
+names = ['BatchSettings', 'FlowControl']


#3463 (comment)

lukesneeringer · 2017-08-22T20:16:28Z

@jonparrott I think this is good other than the inner Status class that I asked about.

theacodes · 2017-08-22T20:19:03Z

@jonparrott I think this is good other than the inner Status class that I asked about.

I answered

lukesneeringer · 2017-08-22T23:11:21Z

Got it. GitHub obscured that and I missed it.

lukesneeringer · 2017-08-22T23:11:36Z

Also, I think this knocks out everything on this PR. Is that accurate?

theacodes · 2017-08-22T23:21:51Z

Two outstanding items:

lukesneeringer · 2017-08-24T03:48:55Z

Intentionally bypassing CircleCI because:

They already ran on pubsub-subscriber, which was branched off of this branch.
Same for pubsub-docs.
I am setting up a final check to merge pubsub to master, and CI can and will run from that tonight to hopefully merge tomorrow morning.

plamut · 2021-12-22T14:37:15Z

+def add_methods(source_class, blacklist=()):
+ """Add wrapped versions of the `api` member's methods to the class.


@lukesneeringer @theacodes @dhermes
Hey folks, happy 2022!
This is a long shot, as the PR is more than 4 years old, but does any of you remember what problem this decorator tried to solve? Why the generated client classes were not directly subclassed by the hand-written classes and instead kept an instance of the generated class in the .api attribute?
The thing is that type checkers issue a lot of complaints about non-existing methods, because they simply cannot know that a lot of these methods are dynamically injected at import time.
We would thus like to subclass the generated classes directly, already have a POC working, but we would like to know if there are any significant downsides to it, if anyone remembers?
Thanks!

…eCloudPlatform/python-docs-samples#3463)

Luke Sneeringerand others added 14 commits May 25, 2017 07:03

[WIP] Getting started on Pub/Sub.
f784121

stuff
c8a9fd6

WIP
9b09b8f

wip
8f0832e

WIP
02d7658

Merge branch 'pubsub' of github.com:GoogleCloudPlatform/google-cloud-…
3a3ebe9
…python into pubsub

WIP, fixing bugs.
503d11f

WIP
3b21b93

wip
1e879a1

wip
4bf0552

wip
c6dc098

wip
13821a7

wip
c2d7af8

Merge branch 'public-master' into pubsub
4c0bf84

lukesneeringer added api: pubsub Issues related to the Pub/Sub API. priority: p1 Important issue which blocks shipping the next release. Will be fixed prior to next release. do not merge Indicates a pull request not ready for merge, due to either quality or timing. labels Jun 1, 2017

lukesneeringer self-assigned this Jun 1, 2017

lukesneeringer requested review from dhermes and theacodes June 1, 2017 14:40

googlebot added the cla: yes This human has signed the Contributor License Agreement. label Jun 1, 2017