Uh oh!
There was an error while loading. Please reload this page.
debug: DEBUG=workflow:* on turbopack workbench to diagnose world-vercel span gap - #2900
Closed
karthikscale3 wants to merge 1 commit into
Closed
debug: DEBUG=workflow:* on turbopack workbench to diagnose world-vercel span gap#2900karthikscale3 wants to merge 1 commit into
karthikscale3 wants to merge 1 commit into
Conversation
Diagnostic for the world-vercel span-emission gap: current main deployments emit core spans (workflow.stream.flush, workflow.stream.read) but none of world-vercel's (workflow.stream.write/chunk_rtt, read.connect, generic http spans) — the signature of trace() running with a null tracer. The DEBUG-gated warn added in #2891 will print the @opentelemetry/api load-failure reason in this PR's preview function logs; its absence (with spans still missing) points at the dual-API-copy noop-tracer variant instead. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
🦋 Changeset detectedLatest commit: fefe902 The changes in this PR will be included in the next version bump. This PR includes changesets to release 0 packagesWhen changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Contributor
Contributor
🧪 E2E Test Results✅ All tests passed Summary
Details by Category✅ ▲ Vercel Production
✅ 💻 Local Development
✅ 📦 Local Production
✅ 🐘 Local Postgres
✅ 🪟 Windows
✅ 📋 Other
❌ Some E2E test jobs failed:
Check the workflow run for details. |
Contributor
📊 Workflow Benchmarkscommit Backend:
Avg deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body execution · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (time outside step bodies, client start → last step body exit) · SL: stream latency (first chunk write → visible to the reader) Scenarios — stream: one step that streams chunks back to the client; no hooks, so the run stays in turbo mode · hook + stream: registers a hook before the same streaming step, which exits turbo mode · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges 🟢/🔴 mark percentiles within/above target. Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · STSO (1-20) 20/30/60 · STSO (101-120) 30/45/90 · STSO (1001-1020) 40/60/120 TTFS/WO compare client vs deployment clocks and SL compares the step runner’s clock vs the client’s (NTP-synced in CI). WO ends at the last step body exit, the closest observable proxy for the final step-completion request. |
karthikscale3
commented
Jul 13, 2026
ContributorAuthor
Served its purpose — the DEBUG run proved the @opentelemetry/api import succeeds in deployed apps (no load-failure warn while httpLog confirmed the flag live), which redirected the investigation to the export layer. Findings and follow-ups live on #2901. |
karthikscale3 added a commit
that referenced
this pull request
Jul 14, 2026
* fix(deps): dedupe @opentelemetry/api to a single workspace instance The lockfile resolved both 1.9.0 and 1.9.1, so the copy that registers the tracer provider (via @vercel/otel in the app) and the copy a package imports could differ. The API's global-registration version check rejects a consumer newer than the registered copy and silently hands back a noop tracer — which is why world-vercel's spans (workflow.stream.write/ chunk_rtt, read.connect, its http spans) never reached Datadog from deployed apps while core's spans flowed in the same process. Root-caused via the DEBUG=workflow:* run on #2900: import succeeds, no warn, spans dropped. Pin a single version via a workspace override so every bundle shares one API instance. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * debug: one-shot OTEL runtime diagnostic in core + world-vercel; DEBUG on turbopack workbench The dedupe alone did not restore world-vercel span emission (verified on this PR's own preview: stream traffic flowed, zero workflow.stream.write spans). Under DEBUG=workflow:*, both packages now log once how their module instance of @opentelemetry/api sees the world — global registration version, provider/delegate/tracer/probe constructor names, and whether a probe span is recording. Diffing the core line (spans work) against the world-vercel line (spans dropped) in one deployment's logs pinpoints the divergence. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * debug: log span identity for named world-vercel spans; namespace otel probes per package Diag round 1 showed world-vercel's tracer records and instrumentedFetch handles the stream PUTs, yet the named spans are unfindable in the backend. Round 2: log traceId/spanId/isRecording for every named instrumentedFetch span under DEBUG so export can be checked for a specific span id, and split the probe span names (.core / .world_vercel) so per-package export is attributable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * telemetry: emit stream RPC latencies from core (chunk_rtt, connect_ms, close span) world-vercel's instrumentedFetch spans never export from deployed apps (root cause still open — see PR discussion), so the operationally needed client-side latency signals move one layer up to core, whose spans are proven to export: - workflow.stream.write.chunk_rtt on the workflow.stream.flush span: the World write RPC duration, network included (same attribute key as world-vercel's per-request span so queries are layer-agnostic). - workflow.stream.read.connect_ms on the workflow.stream.read span: the world.streams.get await (read dispatch -> stream handle). - new workflow.stream.close span: the close RPC round trip. Bonus: measured at the World interface, these cover world-local and world-postgres too, not just Vercel deployments. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * telemetry: emit read-completion span (total duration, chunks, bytes) Completes the read-side picture: workflow.stream.read.complete is back-dated to the read dispatch so its duration is the total read, with chunk/byte counts for throughput. Cancelled reads emit nothing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: drop DEBUG from turbopack workbench; tighten changeset Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * telemetry: cover createReconnectingFramedStream in read telemetry Ordinary serialized streams read through createReconnectingFramedStream (which calls world.streams.get directly), so connect_ms / ttfc / read.complete never fired for that path — only WorkflowServerReadableStream was instrumented. Wire the same helpers into the framed reader: first- connect duration, first-frame TTFC, and completion totals — plus workflow.stream.read.reconnects, which only this path can know. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose — diagnostic, not for merge as-is
Current main deployments emit
@workflow/corespans (workflow.stream.flush,workflow.stream.read) but zero@workflow/world-vercelspans (workflow.stream.write/chunk_rtt,workflow.stream.read.connect, and even its generic http spans) — verified per-trace on deploymentad04a5e: each stream PUT shows only the@vercel/otelfetch span + the server span, with no world-vercel client span between them. That's the signature of world-vercel'strace()running with a null tracer: itsimport('@opentelemetry/api')fails (or resolves an incompatible copy) in the deployed bundle, and the failure is silently latched.#2891 added a
DEBUG=workflow:*-gated warn that prints the load-failure reason. This PR turns that flag on for the nextjs-turbopack workbench deployments so this PR's own CI preview + e2e run produces the verdict.How to read the result
After CI's e2e runs against the preview deployment, check the preview's function logs (or Datadog logs for the preview host) for:
@opentelemetry/apicopy that fails the global-registration compatibility check. Fix = dedupe/externalize so it shares the app's copy.Either way, close or repurpose this PR once the reason is captured — the real fix lands separately.
🤖 Generated with Claude Code