Uh oh!
There was an error while loading. Please reload this page.
fix(bigquery): Fix bigquery socket leak - #17953
Conversation
There was a problem hiding this comment.
Code Review
This pull request refactors the test_dbapi_connection_does_not_leak_sockets system test to use a more deterministic approach by patching and tracking HTTP sessions and gRPC channels instead of relying on flaky psutil socket counts. The review feedback suggests using sets instead of lists to track these objects, which prevents potential flakiness from duplicate close() calls and simplifies both the assertions and the cleanup logic.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
…or symmetry and isolation
…ecks in Connection.close()
chalmerlowe
commented
Jul 31, 2026
Related to work here. |
parthea
commented
Jul 31, 2026
Googlers see b/540939659 |
Uh oh!
There was an error while loading. Please reload this page.
…tion (#17964) Following the merge of PR #17953 for the socket leak fix, this PR focuses on the remaining orthogonal test and fixture hardening fixes: 1. **System Test Teardown Resilience:** Wrap TagKey and TagValue resource deletions in `try...except NotFound: pass` during `TestBigQuery.tearDown()` to prevent cascaded teardown failures when tag resources were already deleted. 2. **Unit Test ADC Isolation:** Enable `autouse=True` on the `use_local_magics_context` fixture in `test_magics.py` to prevent credentials mutation across test runs in uncredentialed CI environments. 3. **InteractiveShell Fixture & Fallback:** Use `IPython.core.interactiveshell.InteractiveShell` and add fallback unit test coverage for `magics.Context`. Related: b/540939659 🦕 --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
🤖 I have created a release *beep* *boop* --- <details><summary>bigquery-magics: 0.15.1</summary> ## [0.15.1](bigquery-magics-v0.15.0...bigquery-magics-v0.15.1) (2026-08-06) ### Bug Fixes * bump grpcio to 1.59.0; require Python 3.10+ ([#17351](#17351)) ([a53487a](a53487a)) * require Protobuf 6.33.5+ ([#17743](#17743)) ([d267342](d267342)) </details> <details><summary>gapic-generator: 1.38.0</summary> ## [1.38.0](gapic-generator-v1.37.1...gapic-generator-v1.38.0) (2026-08-06) ### Features * **generator:** delegate REST transcoding to google-api-core ([#17766](#17766)) ([7b05aab](7b05aab)) * **generator:** gapic generator centralization routing ([#17816](#17816)) ([32a2442](32a2442)) ### Bug Fixes * add typing for header_params ([#17914](#17914)) ([9e98b93](9e98b93)) * **api-core:** use truthiness check in setup_request_id to support proto-plus messages ([#18000](#18000)) ([ad8f93c](ad8f93c)) * avoid retaining routing parameter instances in cache ([#17961](#17961)) ([f64ada2](f64ada2)) * bump aiohttp from 3.13.5 to 3.14.3 in /packages/gapic-generator ([#17990](#17990)) ([6ff5815](6ff5815)) * bump cryptography from 48.0.1 to 50.0.0 in /packages/gapic-generator ([#17991](#17991)) ([d607f25](d607f25)) * bump grpcio to 1.59.0; require Python 3.10+ ([#17351](#17351)) ([a53487a](a53487a)) * **generator:** use flat_ref_types in test templates and delete remove-unused-imports ([#17900](#17900)) ([395f764](395f764)) * require Protobuf 6.33.5+ ([#17743](#17743)) ([d267342](d267342)) * resolve collision with reserved words in samples ([#17912](#17912)) ([588cda9](588cda9)) * upgrade Protobuf and gRPC in WORKSPACE ([#17882](#17882)) ([5b5ece5](5b5ece5)) </details> <details><summary>gcp-sphinx-docfx-yaml: 3.3.1</summary> ## [3.3.1](gcp-sphinx-docfx-yaml-v3.3.0...gcp-sphinx-docfx-yaml-v3.3.1) (2026-08-06) ### Bug Fixes * bump grpcio to 1.59.0; require Python 3.10+ ([#17351](#17351)) ([a53487a](a53487a)) * require Protobuf 6.33.5+ ([#17743](#17743)) ([d267342](d267342)) </details> <details><summary>google-ads-admanager: 0.10.1</summary> ## [0.10.1](google-ads-admanager-v0.10.0...google-ads-admanager-v0.10.1) (2026-08-06) ### Features * update googleapis and regenerate ([#17893](#17893)) ([e70ab6f](e70ab6f)) </details> <details><summary>google-api-core: 2.34.0</summary> ## [2.34.0](google-api-core-v2.33.0...google-api-core-v2.34.0) (2026-08-06) ### Features * Add Feature Gating configuration helpers. ([#17524](#17524)) ([eceea95](eceea95)) * **api-core:** centralize rest transcoding helpers ([#17765](#17765)) ([4f21b8b](4f21b8b)) ### Bug Fixes * **api-core:** use truthiness check in setup_request_id to support proto-plus messages ([#18000](#18000)) ([ad8f93c](ad8f93c)) * bump grpcio to 1.59.0; require Python 3.10+ ([#17351](#17351)) ([a53487a](a53487a)) * deduplicate x-goog-api-client headers ([#17616](#17616)) ([6167e41](6167e41)) * require Protobuf 6.33.5+ ([#17743](#17743)) ([d267342](d267342)) </details> <details><summary>google-apps-chat: 0.10.4</summary> ## [0.10.4](google-apps-chat-v0.10.3...google-apps-chat-v0.10.4) (2026-08-06) ### Features * update googleapis and regenerate ([#17893](#17893)) ([e70ab6f](e70ab6f)) </details> <details><summary>google-auth: 2.56.3</summary> ## [2.56.3](google-auth-v2.56.2...google-auth-v2.56.3) (2026-08-06) ### Bug Fixes * **auth:** avoid creating mTLS SSL context for custom async transports ([#17825](#17825)) ([fbe33f9](fbe33f9)), refs [#17622](#17622) * **auth:** only trigger mTLS certificate rotation on mTLS endpoints ([#17928](#17928)) ([f7b49ea](f7b49ea)) * **auth:** properly extract stdout from gnubby webauthn plugin failures ([#17885](#17885)) ([744e826](744e826)) * deduplicate x-goog-api-client headers ([#17616](#17616)) ([6167e41](6167e41)) * **oauth2:** avoid redundant JWKS network fetches ([#17891](#17891)) ([de53298](de53298)) ### Performance Improvements * **auth:** use generator expression in any() to allow short-circuiting ([735e565](735e565)) * **auth:** use generator expression in any() to allow short-circuiting ([#17937](#17937)) ([735e565](735e565)) </details> <details><summary>google-auth-httplib2: 0.4.1</summary> ## [0.4.1](google-auth-httplib2-v0.4.0...google-auth-httplib2-v0.4.1) (2026-08-06) ### Bug Fixes * require Protobuf 6.33.5+ ([#17743](#17743)) ([d267342](d267342)) </details> <details><summary>google-cloud-access-context-manager: 0.6.1</summary> ## [0.6.1](google-cloud-access-context-manager-v0.6.0...google-cloud-access-context-manager-v0.6.1) (2026-08-06) ### Bug Fixes * bump grpcio to 1.59.0; require Python 3.10+ ([#17351](#17351)) ([a53487a](a53487a)) * require Protobuf 6.33.5+ ([#17743](#17743)) ([d267342](d267342)) </details> <details><summary>google-cloud-agentidentitycredentials: 0.1.1</summary> ## [0.1.1](google-cloud-agentidentitycredentials-v0.1.0...google-cloud-agentidentitycredentials-v0.1.1) (2026-08-06) ### Features * **google/cloud/agentidentitycredentials/v1beta:** add google-cloud-agentidentitycredentials v1beta ([#17898](#17898)) ([b692dae](b692dae)) * update googleapis and regenerate ([#17893](#17893)) ([e70ab6f](e70ab6f)) </details> <details><summary>google-cloud-audit-log: 0.6.1</summary> ## [0.6.1](google-cloud-audit-log-v0.6.0...google-cloud-audit-log-v0.6.1) (2026-08-06) ### Bug Fixes * bump grpcio to 1.59.0; require Python 3.10+ ([#17351](#17351)) ([a53487a](a53487a)) * require Protobuf 6.33.5+ ([#17743](#17743)) ([d267342](d267342)) </details> <details><summary>google-cloud-backupdr: 0.10.1</summary> ## [0.10.1](google-cloud-backupdr-v0.10.0...google-cloud-backupdr-v0.10.1) (2026-08-06) ### Features * update googleapis and regenerate ([#17893](#17893)) ([e70ab6f](e70ab6f)) </details> <details><summary>google-cloud-batch: 0.22.2</summary> ## [0.22.2](google-cloud-batch-v0.22.1...google-cloud-batch-v0.22.2) (2026-08-06) ### Features * update googleapis and regenerate ([#17893](#17893)) ([e70ab6f](e70ab6f)) </details> <details><summary>google-cloud-biglake: 0.5.1</summary> ## [0.5.1](google-cloud-biglake-v0.5.0...google-cloud-biglake-v0.5.1) (2026-08-06) ### Features * update googleapis and regenerate ([#17893](#17893)) ([e70ab6f](e70ab6f)) </details> <details><summary>google-cloud-biglake-hive: 0.3.2</summary> ## [0.3.2](google-cloud-biglake-hive-v0.3.1...google-cloud-biglake-hive-v0.3.2) (2026-08-06) ### Features * update googleapis and regenerate ([#17893](#17893)) ([e70ab6f](e70ab6f)) </details> <details><summary>google-cloud-bigquery: 3.43.0</summary> ## [3.43.0](google-cloud-bigquery-v3.42.3...google-cloud-bigquery-v3.43.0) (2026-08-06) ### Features * add pandas-gbq capability helper ([#17957](#17957)) ([2207ca6](2207ca6)) ### Bug Fixes * **bigquery:** Fix bigquery socket leak ([#17953](#17953)) ([8c26b38](8c26b38)) * remove stray debug print in RangeQueryParameter constructor ([#17973](#17973)) ([fe7bfd0](fe7bfd0)) ### Documentation * add connector libraries overview table to package README ([#17939](#17939)) ([71bc622](71bc622)) * correct return type in CellDataParser.time_to_py docstring ([#17972](#17972)) ([bd1e224](bd1e224)) </details> <details><summary>google-cloud-bigquery-storage: 2.40.0</summary> ## [2.40.0](google-cloud-bigquery-storage-v2.39.0...google-cloud-bigquery-storage-v2.40.0) (2026-08-06) ### Features * delegate ReadRowsPage.to_arrow to pandas_gbq.arrow ([#17938](#17938)) ([aedc66f](aedc66f)) ### Bug Fixes * bump grpcio to 1.59.0; require Python 3.10+ ([#17351](#17351)) ([a53487a](a53487a)) </details> <details><summary>google-cloud-binary-authorization: 1.19.1</summary> ## [1.19.1](google-cloud-binary-authorization-v1.19.0...google-cloud-binary-authorization-v1.19.1) (2026-08-06) ### Bug Fixes * require Protobuf 6.33.5+ ([#17743](#17743)) ([d267342](d267342)) </details> <details><summary>google-cloud-build: 3.38.1</summary> ## [3.38.1](google-cloud-build-v3.38.0...google-cloud-build-v3.38.1) (2026-08-06) ### Bug Fixes * require Protobuf 6.33.5+ ([#17743](#17743)) ([d267342](d267342)) </details> <details><summary>google-cloud-commerceproducer: 0.1.1</summary> ## [0.1.1](google-cloud-commerceproducer-v0.1.0...google-cloud-commerceproducer-v0.1.1) (2026-08-06) ### Features * update googleapis and regenerate ([#17893](#17893)) ([e70ab6f](e70ab6f)) </details> <details><summary>google-cloud-containeranalysis: 2.22.1</summary> ## [2.22.1](google-cloud-containeranalysis-v2.22.0...google-cloud-containeranalysis-v2.22.1) (2026-08-06) ### Bug Fixes * require Protobuf 6.33.5+ ([#17743](#17743)) ([d267342](d267342)) </details> <details><summary>google-cloud-core: 2.6.1</summary> ## [2.6.1](google-cloud-core-v2.6.0...google-cloud-core-v2.6.1) (2026-08-06) ### Bug Fixes * bump grpcio to 1.59.0; require Python 3.10+ ([#17351](#17351)) ([a53487a](a53487a)) * require Protobuf 6.33.5+ ([#17743](#17743)) ([d267342](d267342)) </details> <details><summary>google-cloud-documentai-toolbox: 0.17.2</summary> ## [0.17.2](google-cloud-documentai-toolbox-v0.17.1...google-cloud-documentai-toolbox-v0.17.2) (2026-08-06) ### Bug Fixes * require Protobuf 6.33.5+ ([#17743](#17743)) ([d267342](d267342)) </details> <details><summary>google-cloud-filestore: 1.17.1</summary> ## [1.17.1](google-cloud-filestore-v1.17.0...google-cloud-filestore-v1.17.1) (2026-08-06) ### Bug Fixes * require Protobuf 6.33.5+ ([#17743](#17743)) ([d267342](d267342)) </details> <details><summary>google-cloud-firestore: 2.28.1</summary> ## [2.28.1](google-cloud-firestore-v2.28.0...google-cloud-firestore-v2.28.1) (2026-08-06) ### Bug Fixes * **firestore:** BulkWriter pop from an empty deque ([#17490](#17490)) ([8e826f0](8e826f0)) * **firestore:** preserve async limit_to_last ordering ([#17879](#17879)) ([caf2fdb](caf2fdb)) </details> <details><summary>google-cloud-gke-hub: 1.25.1</summary> ## [1.25.1](google-cloud-gke-hub-v1.25.0...google-cloud-gke-hub-v1.25.1) (2026-08-06) ### Bug Fixes * require Protobuf 6.33.5+ ([#17743](#17743)) ([d267342](d267342)) </details> <details><summary>google-cloud-iam: 2.24.1</summary> ## [2.24.1](google-cloud-iam-v2.24.0...google-cloud-iam-v2.24.1) (2026-08-06) ### Bug Fixes * require Protobuf 6.33.5+ ([#17743](#17743)) ([d267342](d267342)) </details> <details><summary>google-cloud-logging: 3.16.2</summary> ## [3.16.2](google-cloud-logging-v3.16.1...google-cloud-logging-v3.16.2) (2026-08-06) ### Bug Fixes * bump grpcio to 1.59.0; require Python 3.10+ ([#17351](#17351)) ([a53487a](a53487a)) * require Protobuf 6.33.5+ ([#17743](#17743)) ([d267342](d267342)) </details> <details><summary>google-cloud-memorystore: 0.5.4</summary> ## [0.5.4](google-cloud-memorystore-v0.5.3...google-cloud-memorystore-v0.5.4) (2026-08-06) ### Features * update googleapis and regenerate ([#17893](#17893)) ([e70ab6f](e70ab6f)) </details> <details><summary>google-cloud-ndb: 2.5.1</summary> ## [2.5.1](google-cloud-ndb-v2.5.0...google-cloud-ndb-v2.5.1) (2026-08-06) ### Bug Fixes * require Protobuf 6.33.5+ ([#17743](#17743)) ([d267342](d267342)) </details> <details><summary>google-cloud-network-connectivity: 2.17.0</summary> ## [2.17.0](google-cloud-network-connectivity-v2.16.0...google-cloud-network-connectivity-v2.17.0) (2026-08-06) ### Features * update googleapis and regenerate ([#17893](#17893)) ([e70ab6f](e70ab6f)) </details> <details><summary>google-cloud-network-services: 0.10.2</summary> ## [0.10.2](google-cloud-network-services-v0.10.1...google-cloud-network-services-v0.10.2) (2026-08-06) ### Features * update googleapis and regenerate ([#17893](#17893)) ([e70ab6f](e70ab6f)) * update googleapis and regenerate ([#17933](#17933)) ([f7a23a0](f7a23a0)) </details> <details><summary>google-cloud-pubsub: 2.39.1</summary> ## [2.39.1](google-cloud-pubsub-v2.39.0...google-cloud-pubsub-v2.39.1) (2026-08-06) ### Bug Fixes * bump grpcio to 1.59.0; require Python 3.10+ ([#17351](#17351)) ([a53487a](a53487a)) </details> <details><summary>google-cloud-quotas: 0.6.2</summary> ## [0.6.2](google-cloud-quotas-v0.6.1...google-cloud-quotas-v0.6.2) (2026-08-06) ### Features * update googleapis and regenerate ([#17893](#17893)) ([e70ab6f](e70ab6f)) </details> <details><summary>google-cloud-redis-cluster: 0.5.2</summary> ## [0.5.2](google-cloud-redis-cluster-v0.5.1...google-cloud-redis-cluster-v0.5.2) (2026-08-06) ### Features * update googleapis and regenerate ([#17893](#17893)) ([e70ab6f](e70ab6f)) </details> <details><summary>google-cloud-service-control: 1.21.0</summary> ## [1.21.0](google-cloud-service-control-v1.20.0...google-cloud-service-control-v1.21.0) (2026-08-06) ### Features * update googleapis and regenerate ([#17893](#17893)) ([e70ab6f](e70ab6f)) </details> <details><summary>google-cloud-spanner: 3.69.1</summary> ## [3.69.1](google-cloud-spanner-v3.69.0...google-cloud-spanner-v3.69.1) (2026-08-06) ### Bug Fixes * bump grpcio to 1.59.0; require Python 3.10+ ([#17351](#17351)) ([a53487a](a53487a)) * **metrics:** fix GFE and AFE metrics publishing ([#17561](#17561)) ([abf1178](abf1178)) * **spanner:** escape embedded backticks in dbapi escape_name ([#17810](#17810)) ([c8b0b28](c8b0b28)) * **spanner:** implement dict protocol and nested unwrapping for JsonObject ([#17915](#17915)) ([06c1f05](06c1f05)), refs [#15870](#15870) </details> <details><summary>google-cloud-storage: 3.13.1</summary> ## [3.13.1](google-cloud-storage-v3.13.0...google-cloud-storage-v3.13.1) (2026-08-06) ### Bug Fixes * bump grpcio to 1.59.0; require Python 3.10+ ([#17351](#17351)) ([a53487a](a53487a)) * require Protobuf 6.33.5+ ([#17743](#17743)) ([d267342](d267342)) </details> <details><summary>google-cloud-storage-control: 1.13.0</summary> ## [1.13.0](google-cloud-storage-control-v1.12.0...google-cloud-storage-control-v1.13.0) (2026-08-06) ### Features * update googleapis and regenerate ([#17893](#17893)) ([e70ab6f](e70ab6f)) </details> <details><summary>google-cloud-support: 0.5.2</summary> ## [0.5.2](google-cloud-support-v0.5.1...google-cloud-support-v0.5.2) (2026-08-06) ### Features * update googleapis and regenerate ([#17933](#17933)) ([f7a23a0](f7a23a0)) </details> <details><summary>google-cloud-tasks: 2.24.0</summary> ## [2.24.0](google-cloud-tasks-v2.23.0...google-cloud-tasks-v2.24.0) (2026-08-06) ### Features * update googleapis and regenerate ([#17893](#17893)) ([e70ab6f](e70ab6f)) </details> <details><summary>google-cloud-testutils: 1.9.2</summary> ## [1.9.2](google-cloud-testutils-v1.9.1...google-cloud-testutils-v1.9.2) (2026-08-06) ### Bug Fixes * require Protobuf 6.33.5+ ([#17743](#17743)) ([d267342](d267342)) </details> <details><summary>google-developer-knowledge: 0.1.1</summary> ## [0.1.1](google-developer-knowledge-v0.1.0...google-developer-knowledge-v0.1.1) (2026-08-06) ### Features * update googleapis and regenerate ([#17893](#17893)) ([e70ab6f](e70ab6f)) </details> <details><summary>google-maps-navconnect: 0.2.1</summary> ## [0.2.1](google-maps-navconnect-v0.2.0...google-maps-navconnect-v0.2.1) (2026-08-06) ### Features * update googleapis and regenerate ([#17893](#17893)) ([e70ab6f](e70ab6f)) </details> <details><summary>google-resumable-media: 2.10.1</summary> ## [2.10.1](google-resumable-media-v2.10.0...google-resumable-media-v2.10.1) (2026-08-06) ### Bug Fixes * require Protobuf 6.33.5+ ([#17743](#17743)) ([d267342](d267342)) </details> <details><summary>googleapis-common-protos: 1.75.1</summary> ## [1.75.1](googleapis-common-protos-v1.75.0...googleapis-common-protos-v1.75.1) (2026-08-06) ### Bug Fixes * bump grpcio to 1.59.0; require Python 3.10+ ([#17351](#17351)) ([a53487a](a53487a)) * require Protobuf 6.33.5+ ([#17743](#17743)) ([d267342](d267342)) </details> <details><summary>grafeas: 1.24.0</summary> ## [1.24.0](grafeas-v1.23.0...grafeas-v1.24.0) (2026-08-06) ### Features * update googleapis and regenerate ([#17933](#17933)) ([f7a23a0](f7a23a0)) </details> <details><summary>grpc-google-iam-v1: 0.14.5</summary> ## [0.14.5](grpc-google-iam-v1-v0.14.4...grpc-google-iam-v1-v0.14.5) (2026-08-06) ### Bug Fixes * bump grpcio to 1.59.0; require Python 3.10+ ([#17351](#17351)) ([a53487a](a53487a)) * require Protobuf 6.33.5+ ([#17743](#17743)) ([d267342](d267342)) </details> <details><summary>proto-plus: 1.28.3</summary> ## [1.28.3](proto-plus-v1.28.2...proto-plus-v1.28.3) (2026-08-06) ### Bug Fixes * bump grpcio to 1.59.0; require Python 3.10+ ([#17351](#17351)) ([a53487a](a53487a)) * **proto-plus:** add context to TypeErrors during message manipulation ([#17682](#17682)) ([08f21a6](08f21a6)) * require Protobuf 6.33.5+ ([#17743](#17743)) ([d267342](d267342)) </details> <details><summary>sqlalchemy-bigquery: 1.17.2</summary> ## [1.17.2](sqlalchemy-bigquery-v1.17.1...sqlalchemy-bigquery-v1.17.2) (2026-08-06) ### Bug Fixes * require Protobuf 6.33.5+ ([#17743](#17743)) ([d267342](d267342)) </details> --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). --------- Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
This Pull Request addresses flakiness in the socket leak system test and improves resource cleanup in the Database API (DB-API) connection.
Problem
test_dbapi_connection_does_not_leak_socketswas flaky in Continuous Integration (CI) environments because it relied on thepsutillibrary to count Operating System sockets. Socket pooling, delayed garbage collection, and background noise (other processes in multiprocess systems creating sockets in parallelized systems) make counting Operating System sockets non-deterministic.Solution
requests.Sessionandgrpc.Channelobjects using instance-level monkey patching. This ensures that every session or channel opened by the connection is explicitly accounted for and closed, independent of Operating System quirks.Connection.close()inconnection.pyto close all created cursors before closing the clients themselves. This ensures that cursors release their references to query data and transports first. Added defensive checks to prevent errors if clients areNone.Note
psutilor artificial sleeps (that were used to try and wait long enough for garbage collection to kick in), making it faster and fully deterministic.setsto track objects thus avoiding false positives due to double closures.Closes#17956