Add Async Background Thread transport by waprin · Pull Request #1942 · googleapis/google-cloud-python

waprin · 2016-06-29T21:52:59Z

Next step: asynchronous transport using a background thread.

Some notes:

This includes a bugfix where Batch entries were writing logNames incorrectly
Background thread just writes to an entry and keeps re-using it. Seemed ok?
I didn't add a base Transport class yet, though possibly I should? With just a send message that raises NotImpementedException
Background thread will pretty much write an entry as soon as it shows up, although it's possible for a few entries to be written before it's scheduled. I want to run some performance tests on all these handlers and at that point I will start seeing if it would make a difference to wait for more entries to be written before committing them
Docs are getting kind of messy, feel like I might need more nesting. If other people agree I can try to spend more time on it
We should be using Stackdriver Logging everywhere, not CloudLogging. Will fix that all in one go in future PR.

Next handler up will be fluentd.

waprin · 2016-06-29T21:54:15Z

/cc @sharbison3 @filipjs @jonparrott

waprin · 2016-06-29T21:55:40Z

Actually I think I should add the Base transport class, will add that quickly first...

waprin · 2016-06-30T21:02:09Z

Added base Transport class, Travis passes, review/merge at your convenience.

Sign in to view

+ entries on a background :class:`python.threading.Thread`.
+
+ 1. :class:`gcloud.logging.handlers.SyncTransport` this handler does a direct API call on each
+ logging statement to writ the entry.


googlebot · 2016-07-06T22:16:53Z

We found a Contributor License Agreement for you (the sender of this pull request) and all commit authors, but as best as we can tell these commits were authored by someone else. If that's the case, please add them to this pull request and have them confirm that they're okay with these commits being contributed to Google. If we're mistaken and you did author these commits, just reply here to confirm.

waprin · 2016-07-07T00:23:25Z

@tseaver was waiting for review/merge, I just rebased it onto origin/master but now it thinks all those commits are mine. Not sure how to fix?

tseaver · 2016-07-08T18:05:13Z

@waprin is 9e30584 the only "real" commit for this PR? I think you should've rebased it against the logging-stdlib-handler-feature branch, rather than master.

googlebot · 2016-07-08T18:50:48Z

CLAs look good, thanks!

waprin · 2016-07-08T20:50:07Z

@tseaver ok i rebased it, i was trying to merge upstream but I think I did that wrong and I'll just hold off on it for now.

Travis is now failing the lint but looks like origin/master is too so I'm guessing it's not related to this PR.

daspecster · 2016-07-14T14:50:31Z

@waprin, we've been battling with pylint and a few things. I think if you rebase again, it will fix this lint error in travis.

googlebot · 2016-07-14T23:50:43Z

We found a Contributor License Agreement for you (the sender of this pull request) and all commit authors, but as best as we can tell these commits were authored by someone else. If that's the case, please add them to this pull request and have them confirm that they're okay with these commits being contributed to Google. If we're mistaken and you did author these commits, just reply here to confirm.

Sign in to view

+ http)
+ logger = self.client.logger(name)
+ self.worker = _Worker(logger)
+


waprin · 2016-07-15T22:10:54Z

@daspecster ready for another review at your convenience, just mentioned yo in another comment about the need for the class to copy the client.

Sign in to view

+ while not self.stopping:
+ if len(self.batch.entries) == 0:
+ # branch coverage of this code extremely flaky
+ self._entries_condition.wait() # pragma: NO COVER


waprin · 2016-07-25T23:11:30Z

So where are we leaving off with this?

daspecster · 2016-07-25T23:13:03Z

@tseaver @dhermes WDYT?

dhermes · 2016-07-26T16:50:57Z

I'm too out of the loop to comment

waprin · 2016-07-27T22:51:12Z

@tseaver ping

waprin · 2016-07-28T15:39:57Z

@tseaver @dhermes @daspecster Been almost a month, would like this to get merged and the feature branch to get merged into master, what's going on?

jgeewax · 2016-07-28T15:41:11Z

@daspecster : Can you take a look?

daspecster · 2016-07-28T15:44:17Z

@jgeewax sure, I have already a little bit and @waprin addressed my questions. So, LGTM at the moment!

daspecster · 2016-07-28T15:46:15Z

I'll merge if we're good to go?

Refactors handlers into separate package Adds background threaded transport Adds fix to Batch commit to properly set log name

waprin · 2016-07-28T17:23:35Z

Had to kill shebang (since realized it's a bad thing to have in default template), but after Travis passes we can merge.

This is still just into a feature branch so we can still have Tres take a final pass before merge into master.

waprin · 2016-07-28T18:05:25Z

Traceback (most recent call last):
File "/home/travis/build/GoogleCloudPlatform/gcloud-python/.tox/py26/lib/python2.6/site-packages/nose/loader.py", line 418, in loadTestsFromName
addr.filename, addr.module)
File "/home/travis/build/GoogleCloudPlatform/gcloud-python/.tox/py26/lib/python2.6/site-packages/nose/importer.py", line 47, in importFromPath
return self.importFromDir(dir_path, fqname)
File "/home/travis/build/GoogleCloudPlatform/gcloud-python/.tox/py26/lib/python2.6/site-packages/nose/importer.py", line 94, in importFromDir
mod = load_module(part_fqname, fh, filename, desc)
File "/home/travis/build/GoogleCloudPlatform/gcloud-python/gcloud/logging/init.py", line 17, in
from gcloud.logging.client import Client
File "/home/travis/build/GoogleCloudPlatform/gcloud-python/gcloud/logging/client.py", line 42, in
from gcloud.logging.entries import ProtobufEntry
File "/home/travis/build/GoogleCloudPlatform/gcloud-python/gcloud/logging/entries.py", line 20, in
from google.protobuf.json_format import Parse
File "/home/travis/build/GoogleCloudPlatform/gcloud-python/.tox/py26/lib/python2.6/site-packages/google/protobuf/json_format.py", line 48, in
from ordereddict import OrderedDict #PY26
ImportError: No module named ordereddict

😢

tseaver · 2016-07-28T18:07:28Z

Ugh. That dependency is restored on master. Let me look at fixing the target branch.

tseaver · 2016-07-28T19:20:23Z

I merged master to the logging-stdlib-handler-feature branch, and restarted Travis. Fingers crossed.

tseaver · 2016-07-28T19:29:19Z

... and the bulid is green. Whew!

Sign in to view

@@ -0,0 +1,7 @@
+Python Logging Handler Sync Transport
+======================================


waprin · 2016-07-28T20:17:07Z

Thanks, will fix those issues in final commit before I aim for a master merge (fluentd handler).

We have a couple circular imports in the library: - _mtls_helper and _agent_identity_utils both require each other - credentials and _default form a cycle This makes it difficult to import the library into google3 This PR resolved the import issues

PR created by the Librarian CLI to initialize a release. Merging this PR will auto trigger a release. Librarian Version: v1.0.1 Language Image: us-central1-docker.pkg.dev/cloud-sdk-librarian-prod/images-prod/python-librarian-generator@sha256:b8058df4c45e9a6e07f6b4d65b458d0d059241dd34c814f151c8bf6b89211209 <details><summary>google-auth: 2.48.0</summary> ## [2.48.0](googleapis/google-auth-library-python@v2.47.0...v2.48.0) (2026-01-21) ### Features * honor `NO_GCE_CHECK` environment variable (#1610) ([383c9827](googleapis/google-auth-library-python@383c9827)) * add configurable GCE Metadata Server retries (#1488) ([454b441b](googleapis/google-auth-library-python@454b441b)) * add cryptography as required dependency (#1929) ([52558ae2](googleapis/google-auth-library-python@52558ae2)) * Support the mTLS IAM domain for Certificate based Access (#1938) ([8dcf91a1](googleapis/google-auth-library-python@8dcf91a1)) ### Bug Fixes * resolve circular imports (#1942) ([25c1b064](googleapis/google-auth-library-python@25c1b064)) * Use `user_verification=preferred` for ReAuth WebAuthn challenge (#1798) ([3f88a240](googleapis/google-auth-library-python@3f88a240)) * removes `content-header` from AWS IMDS get request (#1934) ([97bfea9e](googleapis/google-auth-library-python@97bfea9e)) * detect correct auth when ADC env var is set but empty (#1374) ([bfc07e10](googleapis/google-auth-library-python@bfc07e10)) * replace deprecated utcfromtimestamp (#1799) ([e431f20c](googleapis/google-auth-library-python@e431f20c)) </details>

…use to download first page of results (#1942) * perf: if `page_size` or `max_results` is set on `QueryJob.result()`, use to download first page of results * add unit tests for query_and_wait * populate maxResults on page 2 * fix maxResults * fix coverage --------- Co-authored-by: Lingqing Gan <lingqing.gan@gmail.com>

Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com>

googlebot added the cla: yes This human has signed the Contributor License Agreement. label Jun 29, 2016

waprin force-pushed the master branch 4 times, most recently from d0e8da2 to f5b9da5Compare June 30, 2016 20:40

theacodes reviewed Jun 30, 2016
View reviewed changes

Comment threaddocs/logging-usage.rst Outdated
entries on a background :class:`python.threading.Thread`.

1. :class:`gcloud.logging.handlers.SyncTransport` this handler does a direct API call on each
logging statement to writ the entry.
This comment was marked as spam.
Sign in to view

waprin force-pushed the master branch 2 times, most recently from d6b9658 to 9e30584Compare July 6, 2016 22:15

googlebot added cla: no This human has *not* signed the Contributor License Agreement. and removed cla: yes This human has signed the Contributor License Agreement. labels Jul 6, 2016

waprin force-pushed the master branch from 9e30584 to 7c27e32Compare July 8, 2016 18:50

googlebot added cla: yes This human has signed the Contributor License Agreement. and removed cla: no This human has *not* signed the Contributor License Agreement. labels Jul 8, 2016

waprin force-pushed the master branch 2 times, most recently from a6bc526 to 61c88a6Compare July 8, 2016 19:58

googlebot added cla: no This human has *not* signed the Contributor License Agreement. and removed cla: yes This human has signed the Contributor License Agreement. labels Jul 14, 2016

waprin force-pushed the master branch from 0576f1d to a62781cCompare July 15, 2016 21:46

waprin reviewed Jul 15, 2016
View reviewed changes

waprin mentioned this pull request Jul 22, 2016
Add Error Reporting Client #1946
Merged

daspecster reviewed Jul 22, 2016
View reviewed changes

Add Async Background Thread transport
4456c70
Refactors handlers into separate package Adds background threaded transport Adds fix to Batch commit to properly set log name

waprin force-pushed the master branch from a62781c to 4456c70Compare July 28, 2016 17:15

tseaver reviewed Jul 28, 2016
View reviewed changes

Comment threaddocs/logging-transports-base.rst
@@ -0,0 +1,7 @@
Python Logging Handler Sync Transport
======================================
This comment was marked as spam.
Sign in to view

tseaver merged commit 930342a into googleapis:logging-stdlib-handler-featureJul 28, 2016

waprin mentioned this pull request Aug 5, 2016
Asynchronous code in this library #2059
Closed

parthea added a commit that referenced this pull request Nov 24, 2025
fix: require google-api-core 1.34.1 (#1942)
14e95e9

parthea pushed a commit that referenced this pull request Apr 1, 2026
chore: usebigframes.pandas._read_gbq_colab()in benchmarks (#1942)
b41cea8
Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com>

		@@ -0,0 +1,7 @@
		Python Logging Handler Sync Transport
		======================================

Uh oh!

Conversation

waprin commented Jun 29, 2016

Uh oh!

waprin commented Jun 29, 2016• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

waprin commented Jun 29, 2016

Uh oh!

waprin commented Jun 30, 2016

Uh oh!

This comment was marked as spam.

Uh oh!

googlebot commented Jul 6, 2016

Uh oh!

waprin commented Jul 7, 2016

Uh oh!

tseaver commented Jul 8, 2016

Uh oh!

googlebot commented Jul 8, 2016

Uh oh!

waprin commented Jul 8, 2016

Uh oh!

daspecster commented Jul 14, 2016• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

googlebot commented Jul 14, 2016

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

waprin commented Jul 15, 2016

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

waprin commented Jul 25, 2016

Uh oh!

daspecster commented Jul 25, 2016

Uh oh!

dhermes commented Jul 26, 2016

Uh oh!

waprin commented Jul 27, 2016

Uh oh!

waprin commented Jul 28, 2016

Uh oh!

jgeewax commented Jul 28, 2016

Uh oh!

daspecster commented Jul 28, 2016

Uh oh!

daspecster commented Jul 28, 2016

Uh oh!

waprin commented Jul 28, 2016

Uh oh!

waprin commented Jul 28, 2016

Uh oh!

tseaver commented Jul 28, 2016

Uh oh!

tseaver commented Jul 28, 2016

Uh oh!

tseaver commented Jul 28, 2016

Uh oh!

waprin commented Jun 29, 2016•
edited
Loading

daspecster commented Jul 14, 2016•
edited
Loading