feat(tracing): correlate business spans with obs via dedicated wrapper span

[NiteshDhanpal]

What & why

Model the business ↔ observability correlation edge on the emit side, so a persisted business span can pivot to its Tempo/Datadog trace and back. No schema migration — the ids ride the existing operation_metadata JSONB (→ ClickHouse metadata_raw).

Two worlds today: business spans (trace_id = task id, stored in application_trace_span + CH) and obs traces (OTel/ddtrace in Tempo/Datadog) share no id. This adds the link.

Changes

• obs_ids.py — correlation keys standardized to obs_trace_id/obs_span_id (underscored → Postgres JSON-path addressable); removed the non-working dual mode (it needed an in-process ddtrace↔OTel bridge that can't exist; degrades to dd_only); obs_correlation() hardened to never raise.
• obs_span.py (new) — dedicated per-business-span wrapper obs span:
  • Opens a real span named for the step and makes it active, so obs_span_id is stable/meaningful (not an arbitrary innermost httpx span); nested instrumentation parents under it.
  • Backends: OTel (lgtm) / ddtrace (dd_only, only when a request trace is active — avoids orphan roots).
  • Reverse tag: stamps agentex.business_span_id / agentex.business_trace_id on the obs span → bidirectional pivot.
  • Error status propagated to the obs span (failed step is not a false green).
  • child_of nesting (ddtrace): ddtrace's start_span does not auto-parent — passes child_of=current_trace_context() so a turn's spans roll up into ONE trace instead of N roots.
• trace.py — wraps start_span/end_span (sync + async). Observability can never fail an app call (guarded; no-op when unconfigured); the two backends never interfere.

Behavior (Turn 2 of the 3-turn example)

business step	before (obs_span_id)	with wrapper
get_state	rB	wB1
retrieve_docs	rB	wB2
chat_completion	rB	wB3
create_message	rB	wB4

Distinct, named obs_span_id per step; all under the one turn obs trace B.

✅ Live validation — infra-staging, dd_only, real rocket-mock turn (52 spans)

Deployed this branch (git-dep) to sgp-rocket-mock-agent on sgp-infra-staging, fired a real message/send, and read the spans straight out of egp application_trace_span.operation_metadata:

• Edge populates: 52/52 business spans carry obs_trace_id + obs_span_id in operation_metadata — no migration.

• Wrapper works:obs_span_id is distinct per step (52 distinct) — a dedicated named span per business step, not the coarse request span.

• child_of fix — before/after:

  	DISTINCT obs_trace_id	DISTINCT obs_span_id
  before (start_span no child_of)	52 (each a new root)	52
  after (child_of=current_ctx)	1 (per-turn roll-up)	52

  → all 52 spans now share the one request/distributed trace id (ca5e90d9…), each a distinct named wrapper span nested under it — the per-turn (TurnTrace) shape.

• Sample operation_metadata: {"turn":0,"agent":"rocket_mock_agent","__source__":"agentex","obs_trace_id":"ca5e90d9…","obs_span_id":"…","__agent_id__":"c7a9692f…"}

Tests

test_obs_ids.py + test_obs_span.py: mode degrade + keys, both backends, non-interference, never-fails, reverse tag, error status, child_of nesting, and the Turn-2 example. Full tracing suite green.

Not in this PR (follow-ups)

• lgtm/Tempo population needs an OTel TracerProvider in the agent (operator auto-instrumentation or app-level) — SDK side is complete and backend-agnostic.
• The sgp-obs-tracing-middleware library still carries dual/bridge.py — separate cleanup.

🤖 Generated with Claude Code

Greptile Summary

Adds bidirectional correlation between business spans and observability traces.

• Opens dedicated OTel or ddtrace wrapper spans for non-Temporal business spans and persists their correlation IDs.
• Uses Temporal activity spans for correlation where wrappers cannot safely cross activity boundaries.
• Adds ACP-to-Temporal producer spans and tests backend selection, correlation, nesting, error propagation, and cleanup behavior.

Confidence Score: 4/5

The PR is not yet safe to merge because synchronous processor failures can still leak wrapper spans and leave their observability context active.

The wrapper handle is registered before synchronous processor callbacks, but an exception from a callback escapes start_span without removing or closing that handle; because no Span is returned, callers cannot reach end_span to perform cleanup.

Files Needing Attention: src/agentex/lib/core/tracing/trace.py

Important Files Changed

Filename	Overview
src/agentex/lib/core/tracing/trace.py	Integrates wrapper-span lifecycle with synchronous and asynchronous business spans; the previously reported synchronous processor-failure cleanup defect remains.
src/agentex/lib/core/tracing/obs_span.py	Implements guarded OTel and ddtrace wrapper creation, reverse correlation tags, error propagation, and backend-specific closure.
src/agentex/lib/core/tracing/obs_ids.py	Standardizes correlation metadata keys, removes dual-mode selection, and makes ID resolution fail open.
src/agentex/lib/core/temporal/services/temporal_task_service.py	Wraps workflow and signal dispatches in OpenTelemetry producer spans to propagate active trace context.
tests/lib/core/tracing/test_obs_span.py	Covers wrapper creation, correlation, reverse tags, error status, backend isolation, Temporal behavior, and cross-instance closure.
tests/lib/core/tracing/test_obs_ids.py	Covers mode normalization, backend selection, fail-open behavior, and correlation ID formatting.

_{Reviews (4): Last reviewed commit: "feat(tracing): span the ACP->Temporal di..." | Re-trigger Greptile}

[harvhan]

The registry assumes start_span and end_span run in the same process, but the Temporal path dispatches START_SPAN and END_SPAN as separate activities (adk/_modules/tracing.py), and AsyncTrace runs inside those activities. With more than one worker replica, a span's END routinely lands on a different worker than its START: the handle on the START worker is never popped, so this dict grows without bound, and the wrapper span is never ended. In lgtm mode an unended span is never exported, so the persisted obs_span_id points at a span that does not exist in Tempo. Even on a single worker the two activities run in different asyncio tasks, so context.detach logs a "Failed to detach context" traceback per span.

Suggestion: skip the wrapper inside Temporal activities (temporalio.activity.in_activity()) and fall back to obs_correlation(). With #485's TracingInterceptor the ambient context inside the activity carries the turn's propagated trace, so temporal still gets trace level correlation, just without the named per step wrapper, which cannot work across activities anyway since the business work runs elsewhere.

[harvhan]

Same failure class as the sync path issue Greptile flagged: if model_copy(deep=True) raises here, the handle leaks in _OBS_HANDLES and the wrapper's context stays attached in the caller's task, so later spans in the task would parent under a stale wrapper. Worth covering both paths with the same try/except when you fix the sync one.