Make connection return a thread-local instance of http. by theacodes · Pull Request #1274 · googleapis/google-cloud-python

theacodes · 2015-12-09T03:26:48Z

Fixes#926, and opens up the possibility of using an object pool later.

theacodes · 2015-12-09T03:28:51Z

@dhermes this is a "basic" fix for #926. It has the drawback that a new httplib2.Http instance will be created and credentials.authorize will be called for every thread. However, I think this is preferable to just exploding when being used with multiple threads.

We can open up a new bug to create a thread-local object pool and use it to provide http objects.

Sign in to view

dhermes · 2015-12-10T06:30:30Z

I think I like it but I'm scared

theacodes · 2015-12-11T03:04:11Z

I think I like it but I'm scared

Agreed. Though, this is the Simplest Thing That Works™. Pooling is a better (more memory/resource friendly) way to do it, but as I said we can take small steps towards that.

dhermes · 2015-12-14T06:01:59Z

@tseaver Can you weigh in? This will have big implications

Sign in to view

Fixesgoogleapis#926, and opens up the possibility of using an object pool later.

theacodes · 2016-01-04T16:56:53Z

@dhermes I see there's a lot of conversation over at #1214 about making httplib2 itself threadsafe. Is this PR obsolete?

dhermes · 2016-01-04T17:46:00Z

No this PR is not obsolete. We won't be fixing httplib2, I just wanted to learn how the handshake occurs.

theacodes · 2016-01-04T20:51:39Z

Gotcha. Let me know if there's anything I need to do to move this along.

dhermes · 2016-01-04T20:54:48Z

Was waiting to hear from @tseaver.

I think I'd prefer us writing a custom http implementation and using that as our default transport instead of httplib2.Http() when one is not passed.

theacodes · 2016-01-04T21:46:22Z

I think I'd prefer us writing a custom http implementation and using that as our default transport instead of httplib2.Http() when one is not passed.

Sounds somewhat reasonable, though what sounds even more reasonable is just using urllib3. Not sure how you feel about that. Either way, can we do it with another bug/PR?

dhermes · 2016-01-04T22:20:08Z

I am all for it, but we need to make oauth2client more flexible first. That's why I wanted to get 100% code coverage there. So we could re-factor it while feeling more secure that we were maintaining functionality. (googleapis/oauth2client#212)

theacodes · 2016-01-04T22:21:01Z

SGTM. In a related effort, I've been trying to make sure urllib3 works on App Engine.

dhermes · 2016-01-04T22:21:47Z

Ha! Very nice.

bendemaree · 2016-01-05T00:29:26Z

@jonparrott It does (mostly) though it will use urlfetch and the sandboxed sockets, IIRC. We vendor requests==2.4.0 in the python27 environment, though that's upstream of urllib3 of course, so a newer version might work as well.

Edit: urllib3==1.10 should work by my same logic. 😉 Hope this gives you some place to start from.

theacodes · 2016-01-05T18:55:39Z

@bendemaree it's now configurable, and there's a PR to requests-toolbelt to enable ongoing support for GAE.

Regardless, I think it's better to do this now, then discuss how we want to support other http libraries in another bug.

theacodes · 2016-01-05T19:20:04Z

Let's continue the discussion around the http library/transport over at #1346.

dhermes · 2016-01-05T20:46:16Z

OK

bendemaree · 2016-01-05T23:04:53Z

@jonparrott Excellent info, thank you. 👍

dhermes · 2016-09-23T22:25:25Z

@jonparrott Shall we close this PR?

theacodes · 2016-09-23T22:33:50Z

Yeah

On Fri, Sep 23, 2016, 3:25 PM Danny Hermes notifications@github.com wrote:

@jonparrott https://github.com/jonparrott Shall we close this PR?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#1274 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAPUc4T3UKXk5TjqR833udKk1YixxCvGks5qtFHVgaJpZM4Gxj3Y
.

saif-freestar · 2018-04-12T23:39:59Z

In my case..i fixed the thing with NOT using the shared connection..and downloading files in chunks async from cloud storage.

…1274) Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com> Co-authored-by: ohmayr <omairn@google.com> Co-authored-by: Anthonios Partheniou <partheniou@google.com>

* docs(samples): add table snapshot sample * docs(samples): fix region tag

* chore: remove deprecated methods in samples and tests * update method

* observability: PDML + some batch write spans This change adds spans for Partitioned DML and making updates for Batch. Carved out from PR #1241. * Add more system tests * Account for lack of OpenTelemetry on Python-3.7 * Update tests * Fix more test assertions * Updates from code review * Update tests with code review suggestions * Remove return per code review nit

googlebot added the cla: yes This human has signed the Contributor License Agreement. label Dec 9, 2015

dhermes reviewed Dec 10, 2015
View reviewed changes

Comment threadgcloud/connection.py Outdated
This comment was marked as spam.
Sign in to view
This comment was marked as spam.
Sign in to view

theacodes mentioned this pull request Dec 11, 2015
httplib2.Http is not thread-safe #1214
Closed

theacodes force-pushed the threadsafe-connection branch 2 times, most recently from 363dcda to 01bf3f5Compare December 11, 2015 03:02

theacodes force-pushed the threadsafe-connection branch 3 times, most recently from 7daeccf to 8417e64Compare December 11, 2015 19:01

dhermes reviewed Dec 14, 2015
View reviewed changes

Comment threadgcloud/test_connection.py Outdated
This comment was marked as spam.
Sign in to view
This comment was marked as spam.
Sign in to view

theacodes force-pushed the threadsafe-connection branch 2 times, most recently from 6836a46 to 99daaafCompare December 14, 2015 18:24

dhermes reviewed Dec 14, 2015
View reviewed changes

Comment threadgcloud/test_connection.py
This comment was marked as spam.
Sign in to view

Make connection return a thread-local instance of http.
7148ef7
Fixesgoogleapis#926, and opens up the possibility of using an object pool later.

theacodes force-pushed the threadsafe-connection branch from 99daaaf to 7148ef7Compare December 14, 2015 18:29

dhermes mentioned this pull request Dec 17, 2015
Adding GC Rule to Bigtable Column Family class. #1302
Merged

dhermes mentioned this pull request Jan 5, 2016
The http object doesn't need to be service specific LeadPages/gcloud_requests#5
Closed

theacodes mentioned this pull request Jan 5, 2016
[Discussion] HTTP library #1346
Closed

dhermes closed this Sep 23, 2016

parthea pushed a commit that referenced this pull request Mar 6, 2026
docs(samples): add table snapshot sample (#1274)
e6a6bca
* docs(samples): add table snapshot sample * docs(samples): fix region tag

parthea pushed a commit that referenced this pull request Mar 9, 2026
fix: remove deprecated methods in samples and tests (#1274)
eb6b3b4
* chore: remove deprecated methods in samples and tests * update method

Uh oh!

Conversation

theacodes commented Dec 9, 2015

Uh oh!

theacodes commented Dec 9, 2015

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

dhermes commented Dec 10, 2015

Uh oh!

theacodes commented Dec 11, 2015

Uh oh!

dhermes commented Dec 14, 2015

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

theacodes commented Jan 4, 2016

Uh oh!

dhermes commented Jan 4, 2016

Uh oh!

theacodes commented Jan 4, 2016

Uh oh!

dhermes commented Jan 4, 2016

Uh oh!

theacodes commented Jan 4, 2016

Uh oh!

dhermes commented Jan 4, 2016

Uh oh!

theacodes commented Jan 4, 2016

Uh oh!

dhermes commented Jan 4, 2016

Uh oh!

bendemaree commented Jan 5, 2016

Uh oh!

theacodes commented Jan 5, 2016

Uh oh!

theacodes commented Jan 5, 2016

Uh oh!

dhermes commented Jan 5, 2016

Uh oh!

bendemaree commented Jan 5, 2016

Uh oh!

dhermes commented Sep 23, 2016

Uh oh!

theacodes commented Sep 23, 2016

Uh oh!

saif-freestar commented Apr 12, 2018• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

saif-freestar commented Apr 12, 2018•
edited
Loading