Uh oh!
There was an error while loading. Please reload this page.
telemetry: move client stream spans from world-vercel to core - #2901
Conversation
The lockfile resolved both 1.9.0 and 1.9.1, so the copy that registers the tracer provider (via @vercel/otel in the app) and the copy a package imports could differ. The API's global-registration version check rejects a consumer newer than the registered copy and silently hands back a noop tracer — which is why world-vercel's spans (workflow.stream.write/ chunk_rtt, read.connect, its http spans) never reached Datadog from deployed apps while core's spans flowed in the same process. Root-caused via the DEBUG=workflow:* run on #2900: import succeeds, no warn, spans dropped. Pin a single version via a workspace override so every bundle shares one API instance. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
🦋 Changeset detectedLatest commit: 8ced2a8 The changes in this PR will be included in the next version bump. This PR includes changesets to release 17 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
🧪 E2E Test Results❌ Some tests failed Summary
❌ Failed Tests▲ Vercel Production (1 failed)astro (1 failed):
Details by Category❌ ▲ Vercel Production
✅ 💻 Local Development
✅ 📦 Local Production
✅ 🐘 Local Postgres
✅ 🪟 Windows
✅ 📋 Other
✅ vercel-multi-region
❌ Some E2E test jobs failed:
Check the workflow run for details. |
📊 Workflow Benchmarkscommit Backend:
📜 Previous results (4)4d08c87Tue, 14 Jul 2026 00:51:19 GMT · run logs
63eba7bTue, 14 Jul 2026 00:18:56 GMT · run logs
cfd543aMon, 13 Jul 2026 19:21:32 GMT · run logs
4b037cbMon, 13 Jul 2026 18:26:14 GMT · run logs
Avg deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body execution · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (time outside step bodies, client start → last step body exit) · SL: stream latency (first chunk write → visible to the reader) Scenarios — stream: one step that streams chunks back to the client; no hooks, so the run stays in turbo mode · hook + stream: registers a hook before the same streaming step, which exits turbo mode · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges 🟢/🔴 mark percentiles within/above target. Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · STSO (1-20) 20/30/60 · STSO (101-120) 30/45/90 · STSO (1001-1020) 40/60/120 TTFS/WO compare client vs deployment clocks and SL compares the step runner’s clock vs the client’s (NTP-synced in CI). WO ends at the last step body exit, the closest observable proxy for the final step-completion request. |
… on turbopack workbench The dedupe alone did not restore world-vercel span emission (verified on this PR's own preview: stream traffic flowed, zero workflow.stream.write spans). Under DEBUG=workflow:*, both packages now log once how their module instance of @opentelemetry/api sees the world — global registration version, provider/delegate/tracer/probe constructor names, and whether a probe span is recording. Diffing the core line (spans work) against the world-vercel line (spans dropped) in one deployment's logs pinpoints the divergence. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…, close span) world-vercel's instrumentedFetch spans never export from deployed apps (root cause still open — see PR discussion), so the operationally needed client-side latency signals move one layer up to core, whose spans are proven to export: - workflow.stream.write.chunk_rtt on the workflow.stream.flush span: the World write RPC duration, network included (same attribute key as world-vercel's per-request span so queries are layer-agnostic). - workflow.stream.read.connect_ms on the workflow.stream.read span: the world.streams.get await (read dispatch -> stream handle). - new workflow.stream.close span: the close RPC round trip. Bonus: measured at the World interface, these cover world-local and world-postgres too, not just Vercel deployments. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Completes the read-side picture: workflow.stream.read.complete is back-dated to the read dispatch so its duration is the total read, with chunk/byte counts for throughput. Cancelled reads emit nothing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
TooTallNate
left a comment
There was a problem hiding this comment.
Reviewed the root-cause work, the span relocation, and the dedupe; ran both suites. Approving with one pre-merge ask.
The root-cause analysis is the best part of this PR. The override comment nails the mechanism (two resolved @opentelemetry/api copies → the global-registration version check hands the non-registering copy a noop tracer → world-vercel's spans silently dropped while core's flowed), and keeping the DEBUG-gated one-shot diagnostics that found it — with core and world-vercel emitting the same shape side by side (registration version, provider/delegate/tracer/probe constructors, recording status) — turns a one-off investigation into a permanent triage tool for exactly this failure class.
Relocation details verified:
connect_msis measured around the singleworld.streams.getacquisition (transparent v3 reconnects happen inside the world stream, so it's genuinely connect-once) and stamped onto the existing production-proven read span.read.completeemits only on drain — cancelled reads emit nothing, matching the docs.- The close span is correctly unconditional (the close RPC happens even for an empty stream), and the pre-existing "empty close" test correctly scopes its assertion to flush spans, so there's no inconsistency.
- Keeping
workflow.stream.write.chunk_rttas the attribute key on the flush span preserves query/dashboard continuity with world-vercel's per-RPC span attribute — and the resulting double-emission in environments where world-vercel's OTEL does work is coherent layering (per-batch above per-RPC), not duplication. - 1454 core + 235 world-vercel tests pass locally; the new InMemorySpanExporter tests cover the read (ttfc+connect), read-complete, and close paths.
Pre-merge ask: merge main and let the Multi-Region lane actually run. The current E2E Vercel Multi-Region Tests failure is "No test files found" — e2e-region.test.ts landed on main (#1981) after this branch's base, so the lane exits 1 without running anything. That's a staleness artifact, not a defect — but this PR changes workspace-wide dependency resolution (the @opentelemetry/api override + 470-line lockfile churn), which is precisely the kind of change every lane should validate, and the multi-region suite has never run against it. A main merge + fresh lockfile + green run closes that gap.
Two questions/notes, non-blocking:
- world-vercel's own remaining spans are still exposed to the noop-tracer failure in user apps. The workspace override fixes the monorepo/workbenches, and the relocation rescues the client stream signals — but world-vercel still emits its per-RPC stream spans and HTTP client spans through its own
@opentelemetry/apiimport, whose resolution in a user's app is whatever their package manager decides. core avoids this via its optional peer"@opentelemetry/api": "1"; world-vercel's manifest is untouched here. Is aligning world-vercel's dependency declaration the follow-up tracked in #2900? Worth stating in the PR description either way, since this PR reads like the fix but is really the workaround plus hygiene. - Docs nit: the
chunk_rtttable row leads with the old per-chunk ack semantics ("emissions of a chunk to the wire… ack for that chunk") — on the flush span it's the whole batch's write RPC duration, which only the second sentence conveys. Leading with the batch semantics would better match what's actually emitted now.
Ordinary serialized streams read through createReconnectingFramedStream (which calls world.streams.get directly), so connect_ms / ttfc / read.complete never fired for that path — only WorkflowServerReadableStream was instrumented. Wire the same helpers into the framed reader: first- connect duration, first-frame TTFC, and completion totals — plus workflow.stream.read.reconnects, which only this path can know. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
karthikscale3
commented
Jul 14, 2026
Addressed the review finding: This also retroactively explains why preview replays never produced read spans during verification — those reads went through the framed path, which was uninstrumented. |
No backport to The substance of this commit (new To override, re-run the Backport to stable workflow manually via |
Summary
world-vercel's client spans don't export from deployed apps (root cause under investigation — evidence in comments; core's spans are proven to export), so the client-observed stream latency signals move to
@workflow/core:workflow.stream.write.chunk_rtt— write RPC duration incl. network, on theworkflow.stream.flushspanworkflow.stream.read.connect_ms— read dispatch → stream handle, on theworkflow.stream.readspanworkflow.stream.close— new span; duration = close RPC round tripworkflow.stream.read.complete— new span; duration = total read, with.chunks/.bytesfor throughput (and.reconnectson the framed path)Measured at the World interface, so world-local/world-postgres are covered too. Also:
@opentelemetry/apideduped to one workspace instance (hygiene) and DEBUG-gated OTEL diagnostics retained from the investigation.Verified live on this PR's preview:
chunk_rtt73–85ms,close.rpc_ms72–96ms.connect_ms/read.completeare unit-tested on both read paths (WorkflowServerReadableStreamandcreateReconnectingFramedStream) — the latter was initially missed, which is why preview reads produced no spans; covered as of4d08c87.Testing
packages/core: 1454 tests pass;
pnpm build/typecheck/ biome clean.🤖 Generated with Claude Code