feat(storage): Enhance Otel Span Attributes with BucketId and Location details for every Bucket/Blob operation

# feat(storage): Enhance Otel Span Attributes with BucketId and Location details for every Bucket/Blob operation as part of ACO (App-centric Observability)
This PR implements **App-centric Observability (ACO)** tracing
compatibility for the GCS Python SDK (`google-cloud-storage`). All
OpenTelemetry trace spans produced by bucket and blob operations now
seamlessly incorporate mandatory destination resource annotations
(`gcp.resource.destination.id` and `gcp.resource.destination.location`).
---
## Core Architecture & Design
### 1. Centralized, DRY Telemetry Helper (`_helpers.py`)
- All OpenTelemetry span context generation, attribute injection, and
exception trapping are centralized in a module-level context manager
`create_trace_span_helper` in
[`_helpers.py`](file:///usr/local/google/home/chandrasiri/storage_related/org-google-cloud-python/packages/google-cloud-storage/google/cloud/storage/_helpers.py).
- **Zero modifications to the core tracing module**:
[`_opentelemetry_tracing.py`](file:///usr/local/google/home/chandrasiri/storage_related/org-google-cloud-python/packages/google-cloud-storage/google/cloud/storage/_opentelemetry_tracing.py)
remains completely pristine and identical to `main`.
- Seamlessly wrapped all critical read/write operations across
`blob.py`, `bucket.py`, and `client.py` (e.g., `download_as_bytes`,
`upload_from_string`, `get_bucket`, `lookup_bucket`, etc.).
### 2. Bounded LRU Metadata Cache (`_lru_cache.py`,
`_bucket_metadata_cache.py`)
- **LRU Capacity Bounding**: Implemented `LRUCache` utilizing an
`OrderedDict` to support O(1) operations and strict capacity bounding to
eliminate memory leaks in long-running applications.
- **Concurrent Singleflight Warming**: Implemented `BucketMetadataCache`
to store bucket locations and project numbers. On cache misses, it
spawns background threads (`_fetch_background`) using singleflight
tracking (`_inflight_fetches`) to prevent server stampedes / thundering
herds.
- **Fallback Annotations on 403**: On GCS `403 Forbidden` permissions
errors, the cache permanently registers fallback annotations
(`projects/_/buckets/{name}`) to completely avoid retry storms on
subsequent API calls.
### 3. Resilient 404 Existence Eviction (`_http.py`, `_helpers.py`,
`bucket.py`)
- **Smart Out-of-band 404 Verification**: When a `404 NotFound` error
occurs during media transfers or REST calls, a background thread is
spawned (with concurrency protection via `_inflight_checks`) to check if
the bucket was deleted out-of-band (`bucket.exists()`). If `exists()`
returns `False`, the bucket is cleanly evicted from the cache.
- **Instant Synchronous Eviction**: Direct `Bucket.delete()` calls
synchronously and instantly evict the bucket name from the cache,
ensuring real-time consistency.
---
## Extensive Testing Suite
### 1. 100% Sleep-Free System Tests (`test_aco_observability.py`)
Added a comprehensive system test suite
[`test_aco_observability.py`](file:///usr/local/google/home/chandrasiri/storage_related/org-google-cloud-python/packages/google-cloud-storage/tests/system/test_aco_observability.py)
executing against a live GCS backend:
- **Sequential Priming**: Verifies cache miss return times, background
priming, and subsequent span enrichment.
- **403 Fallback**: Verifies minimal fallback registration on Forbidden
responses.
- **Cache Stampede Protection**: Simulates 15 concurrent threads on a
cache miss and asserts only 1 GCS call is fired.
- **Smart 404 Eviction**: Deletes a bucket out-of-band and verifies
async cache clean-up on 404.
- **Synchronous Delete Eviction**: Asserts immediate cache eviction on
SDK deletion.
- **LRU Capacity Bounding**: Populates the cache beyond its limits and
verifies proper LRU eviction.
- **Deterministic Synchronization**: Uses **`threading.Event` (zero
static sleeps)** for thread coordination, guaranteeing thundering-fast
execution and completely eliminating timing flakiness.
### 2. Robust Unit Tests
- Added
[`test__lru_cache.py`](file:///usr/local/google/home/chandrasiri/storage_related/org-google-cloud-python/packages/google-cloud-storage/tests/unit/test__lru_cache.py)
(LRU correctness, bounding, eviction).
- Added
[`test__bucket_metadata_cache.py`](file:///usr/local/google/home/chandrasiri/storage_related/org-google-cloud-python/packages/google-cloud-storage/tests/unit/test__bucket_metadata_cache.py)
(concurrency, location resolution, 403 fallback, singleflight).
- Added `test_delete_hit_evicts_from_cache` inside
[`test_bucket.py`](file:///usr/local/google/home/chandrasiri/storage_related/org-google-cloud-python/packages/google-cloud-storage/tests/unit/test_bucket.py).
---
## Validation Results
All checks, unit tests, and live GCS system tests pass flawlessly:
- **Unit Tests**: 835 passed in 17.82s
- **System Tests**: 8 passed in 26.94s
- **Format & Linter**: 100% clean (`black` / `flake8`)

Author: chandra-siri

packages/google-cloud-storage/google/cloud/storage/_bucket_metadata_cache.py

@@ -0,0 +1,150 @@
+# Copyright 2026 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""In-memory LRU cache for bucket metadata supporting App-centric Observability (ACO)."""
+
+importlogging
+importthreading
+
+fromgoogle.api_coreimportexceptionsasapi_exceptions
+fromgoogle.cloud.exceptionsimportNotFound
+fromgoogle.cloud.storage._lru_cacheimportLRUCache
+
+logger=logging.getLogger(__name__)
+
+
+classBucketMetadataCache:
+"""Thread-safe LRU cache for storing GCS bucket metadata (project number and location).
+
+ Supports Singleflight asynchronous background fetching to prevent stampedes on cache misses.
+ """
+
+def__init__(self, client, max_size=10000):
+self._client=client
+self._cache=LRUCache(max_size)
+self._lock=threading.Lock()
+self._inflight_fetches=set()
+self._inflight_checks=set()
+
+defget(self, bucket_name):
+"""Thread-safely retrieve cached metadata without queueing fetch."""
+withself._lock:
+returnself._cache.get(bucket_name)
+
+defget_or_queue_fetch(self, bucket_name):
+"""Retrieve bucket metadata or queue a background fetch on cache miss.
+
+ Returns None immediately on cache miss so caller does not block.
+ """
+withself._lock:
+ifbucket_nameinself._cache:
+returnself._cache.get(bucket_name)
+elifbucket_nameinself._inflight_fetches:
+# This handles a thundering herd where 'n' threads
+# simultaneously experience a cache miss while 1 is already
+# fetching metadata. The remaining n - 1 threads should
+# bypass starting duplicate fetches.
+returnNone
+else:
+# fire a background thread and get bucket metadata.
+self._inflight_fetches.add(bucket_name)
+threading.Thread(
+target=self._fetch_background, args=(bucket_name,), daemon=True
+ ).start()
+returnNone
+
+defcheck_and_evict(self, bucket_name):
+"""Asynchronously verify if a bucket exists on 404 and evict if deleted."""
+withself._lock:
+ifbucket_namenotinself._cache:
+return
+ifbucket_nameinself._inflight_checks:
+return
+self._inflight_checks.add(bucket_name)
+threading.Thread(
+target=self._verify_existence_background,
+args=(bucket_name,),
+daemon=True,
+ ).start()
+
+def_verify_existence_background(self, bucket_name):
+try:
+bucket=self._client.bucket(bucket_name)
+ifnotbucket.exists():
+self.evict(bucket_name)
+exceptExceptionase:
+logger.debug(
+f"Background verification for bucket existence failed for {bucket_name}: {e}"
+ )
+finally:
+withself._lock:
+self._inflight_checks.discard(bucket_name)
+
+def_fetch_background(self, bucket_name):
+"""Asynchronously fetch bucket metadata and update the cache."""
+try:
+bucket=self._client.get_bucket(bucket_name, timeout=10.0)
+self.update_from_bucket(bucket)
+except (NotFound, api_exceptions.NotFound):
+self.evict(bucket_name)
+exceptapi_exceptions.Forbidden:
+# On 403 (Forbidden), cache fallback values permanently to avoid retry storms
+self.update_cache(
+bucket_name, f"projects/_/buckets/{bucket_name}", "global"
+ )
+exceptExceptionase:
+logger.debug(
+f"Background fetch for bucket metadata failed for {bucket_name}: {e}"
+ )
+finally:
+withself._lock:
+self._inflight_fetches.discard(bucket_name)
+
+defupdate_from_bucket(self, bucket):
+"""Update cache from a Bucket instance."""
+ifnotbucketornotbucket.name:
+return
+
+project_number=getattr(bucket, "project_number", None)
+location=getattr(bucket, "location", None) or"global"
+location=location.lower()
+location_type=getattr(bucket, "location_type", None) or"region"
+location_type=location_type.lower()
+
+iflocation_typein ("multi-region", "dual-region"):
+location="global"
+
+ifproject_number:
+destination_id=f"projects/{project_number}/buckets/{bucket.name}"
+else:
+destination_id=f"projects/_/buckets/{bucket.name}"
+
+self.update_cache(bucket.name, destination_id, location)
+
+defupdate_cache(self, bucket_name, destination_id, location):
+"""Thread-safely update or insert a cache entry with bounded size."""
+withself._lock:
+self._cache.put(bucket_name, (destination_id, location))
+
+defevict(self, bucket_name):
+"""Remove a bucket from the cache (e.g., on 404)."""
+withself._lock:
+self._cache.delete(bucket_name)
+
+defclear(self):
+"""Clear all cached metadata."""
+withself._lock:
+self._cache.clear()
+self._inflight_fetches.clear()
+self._inflight_checks.clear()

packages/google-cloud-storage/google/cloud/storage/_helpers.py

@@ -19,20 +19,30 @@
 
 importbase64
 importdatetime
+importlogging
 importos
 importsecrets
 importsys
+fromcontextlibimportcontextmanager
 fromhashlibimportmd5
 fromurllib.parseimporturlsplit, urlunsplit
 fromuuidimportuuid4
 
+fromgoogle.api_coreimportexceptionsasapi_exceptions
+fromgoogle.cloud.exceptionsimportNotFound
+
 fromgoogle.authimportenvironment_vars
 
 fromgoogle.cloud.storage.constantsimport_DEFAULT_TIMEOUT
 fromgoogle.cloud.storage.retryimport (
 DEFAULT_RETRY,
 DEFAULT_RETRY_IF_METAGENERATION_SPECIFIED,
 )
+fromgoogle.cloud.storage._opentelemetry_tracingimport (
+create_trace_spanas_base_create_trace_span,
+)
+
+_logger=logging.getLogger(__name__)
 
 STORAGE_EMULATOR_ENV_VAR="STORAGE_EMULATOR_HOST"# Despite name, includes scheme.
 """Environment variable defining host for Storage emulator."""
@@ -137,6 +147,62 @@ def _validate_name(name):
 returnname
 
 
+@contextmanager
+defcreate_trace_span_helper(client, bucket_name, name, attributes=None, **kwargs):
+span_attrs=dict(attributes) ifattributeselse {}
+
+if (
+bucket_name
+andisinstance(bucket_name, str)
+andclient
+andhasattr(client, "_bucket_metadata_cache")
+andclient._bucket_metadata_cache
+ ):
+try:
+ifnamein (
+"Storage.Client.getBucket",
+"Storage.Client.lookupBucket",
+"Storage.Bucket.reload",
+"Storage.Bucket.exists",
+ ):
+cached=client._bucket_metadata_cache.get(bucket_name)
+else:
+cached=client._bucket_metadata_cache.get_or_queue_fetch(bucket_name)
+
+ifcachedandisinstance(cached, tuple) andlen(cached) ==2:
+dest_id, loc=cached
+span_attrs.update(
+ {
+"gcp.resource.destination.id": dest_id,
+"gcp.resource.destination.location": loc,
+ }
+ )
+exceptExceptionase:
+_logger.debug(f"Failed cache lookup in create_trace_span_helper: {e}")
+
+if"client"notinkwargsandclient:
+kwargs["client"] =client
+
+with_base_create_trace_span(name, attributes=span_attrs, **kwargs) asspan:
+try:
+yieldspan
+except (NotFound, api_exceptions.NotFound):
+if (
+bucket_name
+andisinstance(bucket_name, str)
+andclient
+andhasattr(client, "_bucket_metadata_cache")
+andclient._bucket_metadata_cache
+ ):
+try:
+client._bucket_metadata_cache.check_and_evict(bucket_name)
+exceptExceptionase:
+_logger.debug(
+f"Failed cache eviction on 404 in create_trace_span_helper: {e}"
+ )
+raise
+
+
 class_PropertyMixin(object):
 """Abstract mixin for cloud storage classes with associated properties.
 
@@ -185,6 +251,42 @@ def _require_client(self, client):
 client=self.client
 returnclient
 
+@contextmanager
+def_create_trace_span(self, name, attributes=None, **kwargs):
+fromgoogle.cloud.storage.blobimportBlob
+fromgoogle.cloud.storage.bucketimportBucket
+
+ifisinstance(self, Bucket):
+client=self.client
+bucket_name=self.name
+elifisinstance(self, Blob):
+bucket=getattr(self, "bucket", None)
+client= (
+getattr(bucket, "client", None)
+ifbucketandhasattr(bucket, "client")
+elseNone
+ )
+bucket_name=getattr(bucket, "name", None) ifbucketelseNone
+else:
+client=None
+bucket_name=None
+
+ifcallable(bucket_name):
+try:
+bucket_name=bucket_name()
+exceptExceptionase:
+_logger.debug(
+f"Failed callable bucket_name resolution in _create_trace_span: {e}"
+ )
+
+client_override=kwargs.pop("client", None)
+active_client=client_overrideorclient
+
+withcreate_trace_span_helper(
+active_client, bucket_name, name, attributes=attributes, **kwargs
+ ) asspan:
+yieldspan
+
 def_encryption_headers(self):
 """Return any encryption headers needed to fetch the object.
 

packages/google-cloud-storage/google/cloud/storage/_http.py

@@ -15,10 +15,20 @@
 """Create / interact with Google Cloud Storage connections."""
 
 importfunctools
+importlogging
+importre
 
+fromgoogle.api_coreimportexceptionsasapi_exceptions
 fromgoogle.cloudimport_http
+fromgoogle.cloud.exceptionsimportNotFound
 fromgoogle.cloud.storageimport__version__, _helpers
-fromgoogle.cloud.storage._opentelemetry_tracingimportcreate_trace_span
+fromgoogle.cloud.storage._opentelemetry_tracingimport (
+create_trace_span,
+enable_otel_traces,
+HAS_OPENTELEMETRY,
+)
+
+logger=logging.getLogger(__name__)
 
 
 classConnection(_http.JSONConnection):
@@ -71,11 +81,30 @@ def api_request(self, *args, **kwargs):
 span_attributes= {
 "gccl-invocation-id": invocation_id,
  }
+client=self._client
+if (
+HAS_OPENTELEMETRY
+andenable_otel_traces
+andhasattr(client, "_bucket_metadata_cache")
+andclient._bucket_metadata_cache
+ ):
+path=kwargs.get("path") or""
+match=re.search(r"/b/([^/?#]+)", path)
+ifmatch:
+try:
+cached=client._bucket_metadata_cache.get(match.group(1))
+ifcachedandisinstance(cached, tuple) andlen(cached) ==2:
+dest_id, loc=cached
+span_attributes["gcp.resource.destination.id"] =dest_id
+span_attributes["gcp.resource.destination.location"] =loc
+exceptExceptionase:
+logger.debug(f"Failed cache.get_or_queue_fetch in api_request: {e}")
+
 call=functools.partial(super(Connection, self).api_request, *args, **kwargs)
 withcreate_trace_span(
 name="Storage.Connection.api_request",
 attributes=span_attributes,
-client=self._client,
+client=client,
 api_request=kwargs,
 retry=retry,
  ):
@@ -87,4 +116,24 @@ def api_request(self, *args, **kwargs):
 pass
 ifretry:
 call=retry(call)
-returncall()
+try:
+returncall()
+except (NotFound, api_exceptions.NotFound):
+if (
+HAS_OPENTELEMETRY
+andenable_otel_traces
+andhasattr(client, "_bucket_metadata_cache")
+andclient._bucket_metadata_cache
+ ):
+path=kwargs.get("path") or""
+match=re.search(r"/b/([^/?#]+)", path)
+ifmatch:
+try:
+client._bucket_metadata_cache.check_and_evict(
+match.group(1)
+ )
+exceptExceptionase:
+logger.debug(
+f"Failed cache.check_and_evict on 404 in api_request: {e}"
+ )
+raise