Uh oh!
There was an error while loading. Please reload this page.
otel(world-vercel): inject trace context on v4 event requests - #2533
Merged
Conversation
The v4 event path (createEvent / getEvent / listEvents) routes through fetchV4 → global fetch with a custom undici dispatcher, bypassing both the makeRequest path (where the explicit W3C trace-context injection lives) and ambient undici auto-instrumentation. As a result, v4 event traffic from the flow route carried no traceparent, so workflow-server could not parent its spans to the invocation — its spans never joined the /flow execution trace, even though v2/v3 reads/writes (via makeRequest) did join. fetchV4 now calls injectTraceContextIntoHeaders before fetch, the single choke point for all v4 create/get/list requests, mirroring makeRequest. No-op when no OpenTelemetry SDK is registered. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
🦋 Changeset detectedLatest commit: 069beae The changes in this PR will be included in the next version bump. This PR includes changesets to release 17 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Contributor
Contributor
📊 Benchmark Results
workflow with no steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express | Next.js (Turbopack) | Nitro workflow with 1 step💻 Local Development
▲ Production (Vercel)
🔍 Observability: Next.js (Turbopack) | Express | Nitro workflow with 10 sequential steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express | Nitro | Next.js (Turbopack) workflow with 25 sequential steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Express | Next.js (Turbopack) workflow with 50 sequential steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express | Next.js (Turbopack) | Nitro Promise.all with 10 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Express | Next.js (Turbopack) Promise.all with 25 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Next.js (Turbopack) | Nitro | Express Promise.all with 50 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Next.js (Turbopack) | Express | Nitro Promise.race with 10 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Next.js (Turbopack) | Nitro | Express Promise.race with 25 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express | Next.js (Turbopack) | Nitro Promise.race with 50 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Next.js (Turbopack) | Express | Nitro workflow with 10 sequential data payload steps (10KB)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Next.js (Turbopack) | Express workflow with 25 sequential data payload steps (10KB)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Express | Next.js (Turbopack) workflow with 50 sequential data payload steps (10KB)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Express | Next.js (Turbopack) workflow with 10 concurrent data payload steps (10KB)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Next.js (Turbopack) | Express | Nitro workflow with 25 concurrent data payload steps (10KB)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Next.js (Turbopack) | Express | Nitro workflow with 50 concurrent data payload steps (10KB)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Next.js (Turbopack) | Express Stream Benchmarks(includes TTFB metrics)workflow with stream💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Express | Next.js (Turbopack) stream pipeline with 5 transform steps (1MB)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Next.js (Turbopack) | Express 10 parallel streams (1MB each)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express | Nitro | Next.js (Turbopack) fan-out fan-in 10 streams (1MB each)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Express | Next.js (Turbopack) SummaryFastest Framework by WorldWinner determined by most benchmark wins
Fastest World by FrameworkWinner determined by most benchmark wins
Column Definitions
Worlds:
|
Contributor
🧪 E2E Test Results❌ Some tests failed Summary
❌ Failed Tests▲ Vercel Production (2 failed)fastify (1 failed):
vite (1 failed):
Details by Category❌ ▲ Vercel Production
✅ 💻 Local Development
✅ 📦 Local Production
✅ 🐘 Local Postgres
✅ 🪟 Windows
✅ 📋 Other
❌ Some E2E test jobs failed:
Check the workflow run for details. |
…P paths Codify the guardrail that the v4 regression revealed: any outgoing world-vercel request must call injectTraceContextIntoHeaders (auto- instrumentation can't be relied on with the custom dispatcher / global fetch), with a test in trace-propagation.test.ts. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
pranaygp
approved these changes
Jun 19, 2026
Contributor
No backport to The fix builds entirely on To override, re-run the Backport to stable workflow manually via |
This was referenced Jun 19, 2026
pranaygp added a commit
that referenced
this pull request
Jun 22, 2026
…testing * origin/main: Version Packages (beta) (#2540) perf(core): memoize step return value hydration across inline replays (#2472) [core] Fix abort signal not reflected in subsequent step (replay-ordering flake) (#2412) Amend lazy discovery removal changeset (#2560) [docs] Document minimum SDK version for using hook.getConflict (#2423) Update default CODEOWNERS (#2556) Optimize and fix the default eager build mode (#2546) Revert "fix(world-vercel): cancel v4 event frame stream on early exit (#2547)" (#2554) [core] Turbo mode: fast-path the first invocation (#2526) Remove lazy discovery from workflow/next (#2545) fix(world-vercel): cancel v4 event frame stream on early exit (#2547) feat(docs): add eve and AI SDK to product switcher (#2543) [vitest] Fix local imports failing to load in test step bundles (#2351) [builders] Fix unicode-escape crash in workflow graph extraction (#2324) Version Packages (beta) (#2495) otel(world-vercel): inject trace context on v4 event requests (#2533) Bump undici to 7.28.0 (#2534) Default source maps to dev-on / prod-off (#2529) otel: nest linked-mode invocations under the delivery context (route + execution in one trace) (#2527) perf(core): parallel inline steps + optimistic lazy step start (#2516)
pranaygp added a commit
to marcopiraccini/workflow
that referenced
this pull request
Jun 22, 2026
* origin/main: (120 commits) Version Packages (beta) (vercel#2540) perf(core): memoize step return value hydration across inline replays (vercel#2472) [core] Fix abort signal not reflected in subsequent step (replay-ordering flake) (vercel#2412) Amend lazy discovery removal changeset (vercel#2560) [docs] Document minimum SDK version for using hook.getConflict (vercel#2423) Update default CODEOWNERS (vercel#2556) Optimize and fix the default eager build mode (vercel#2546) Revert "fix(world-vercel): cancel v4 event frame stream on early exit (vercel#2547)" (vercel#2554) [core] Turbo mode: fast-path the first invocation (vercel#2526) Remove lazy discovery from workflow/next (vercel#2545) fix(world-vercel): cancel v4 event frame stream on early exit (vercel#2547) feat(docs): add eve and AI SDK to product switcher (vercel#2543) [vitest] Fix local imports failing to load in test step bundles (vercel#2351) [builders] Fix unicode-escape crash in workflow graph extraction (vercel#2324) Version Packages (beta) (vercel#2495) otel(world-vercel): inject trace context on v4 event requests (vercel#2533) Bump undici to 7.28.0 (vercel#2534) Default source maps to dev-on / prod-off (vercel#2529) otel: nest linked-mode invocations under the delivery context (route + execution in one trace) (vercel#2527) perf(core): parallel inline steps + optimistic lazy step start (vercel#2516) ...
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
v4 event requests don't propagate W3C trace context to workflow-server, so workflow-server spans never join the
/flowexecution trace — while v2/v3 calls do. This restores propagation for v4.Root cause
The explicit trace-context injection added for cross-service correlation lives in
world-vercel'smakeRequest(the v2/v3 path):The v4 event path (
createWorkflowRunEventV4/getEventV4/getWorkflowRunEventsV4) was routed through a separatefetchV4→ globalfetchhelper (to keep v4 traffic visible in Vercel's outgoing-requests view).fetchV4builds headers fromgetHttpConfig(auth/UA only), setsX-Request-Time, and callsfetch(...)with a custom undici dispatcher — it never callsinjectTraceContextIntoHeaders, and the custom dispatcher means ambient undici auto-instrumentation can't be relied on for injection either.Net effect, observable in traces:
/api/run/...→ run-status reads viaGET /api/v2/runs/:runId) goes throughmakeRequest→traceparentinjected → workflow-server's server span joins the trace. ✓/flowroute (event writes viaPOST /api/v4/runs/.../events) goes throughfetchV4→ notraceparent→ workflow-server starts a fresh root and never joins the invocation trace. ✗This is not sampling — the v4 requests genuinely carry no trace context.
Fix
Inject the active trace context in
fetchV4, the single choke point for all v4 create/get/list requests, mirroringmakeRequest:Now workflow-server joins the
/flowexecution trace the same way it joins the start-site trace. No-op when no OpenTelemetry SDK is registered.Guardrail
Adds a note to
AGENTS.md(Architecture Notes → "Trace context propagation") requiring every new@workflow/world-vercelHTTP path to callinjectTraceContextIntoHeadersand cover it intrace-propagation.test.ts, citing this v4 regression — so a future v5 events API doesn't repeat it.Testing
trace-propagation.test.ts: drivesgetWorkflowRunEventsV4inside an active span (MockAgent + realfetchspy, W3C propagator registered) and asserts the outgoing v4 request carriestraceparentmatching the active span. Fails without the injection.@workflow/world-vercel: full suite 163 passed; build + typecheck clean.🤖 Generated with Claude Code