Uh oh!
There was an error while loading. Please reload this page.
feat(world-vercel): client-observed stream write/read e2e latency spans - #2857
Conversation
Stream write/read requests already share the instrumented HTTP envelope (a CLIENT span + W3C trace-context injection), but the spans were named for the bare HTTP verb (`http PUT`/`http GET`) and carried only generic HTTP attributes — so stream latency couldn't be sliced per run/stream. Name these spans for their operation (`workflow.stream.write` / `workflow.stream.read`) and tag them with `workflow.run.id`, `workflow.stream.name`, `workflow.stream.operation` (write | write_multi | close | read), and `workflow.stream.start_index` (read). Implemented via new optional `spanName`/`attributes` fields on `instrumentedFetch`, so other callers are unaffected. Additive OTEL only: no behavior change when no OpenTelemetry SDK is registered (the span is undefined and attributes are dropped). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
🦋 Changeset detectedLatest commit: 9e42061 The changes in this PR will be included in the next version bump. This PR includes changesets to release 17 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
🧪 E2E Test Results❌ Some tests failed Summary
❌ Failed Tests▲ Vercel Production (1 failed)vite (1 failed):
Details by Category❌ ▲ Vercel Production
✅ 💻 Local Development
✅ 📦 Local Production
✅ 🐘 Local Postgres
✅ 🪟 Windows
✅ 📋 Other
❌ Some E2E test jobs failed:
Check the workflow run for details. |
📊 Workflow Benchmarkscommit Backend:
📜 Previous results (4)a97582eThu, 09 Jul 2026 21:13:27 GMT · run logs
a435ba0Thu, 09 Jul 2026 20:40:36 GMT · run logs
98cd410Thu, 09 Jul 2026 20:10:49 GMT · run logs
3de7361Thu, 09 Jul 2026 16:39:51 GMT · run logs
Avg deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body execution · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (time outside step bodies, client start → last step body exit) · SL: stream latency (first chunk write → visible to the reader) Scenarios — stream: one step that streams chunks back to the client; no hooks, so the run stays in turbo mode · hook + stream: registers a hook before the same streaming step, which exits turbo mode · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges 🟢/🔴 mark percentiles within/above target. Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · STSO (1-20) 20/30/60 · STSO (101-120) 30/45/90 · STSO (1001-1020) 40/60/120 TTFS/WO compare client vs deployment clocks and SL compares the step runner’s clock vs the client’s (NTP-synced in CI). WO ends at the last step body exit, the closest observable proxy for the final step-completion request. |
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
| // chunks are skipped. No-op when no OpenTelemetry SDK is registered. | ||
| const readSpanKind = await getSpanKind('CLIENT'); | ||
| let firstChunkSeen = false; | ||
| const ttfcProbe = new TransformStream<Uint8Array, Uint8Array>({ |
There was a problem hiding this comment.
Instead of doing a transform stream, can we emit the span from the core serializer WorkflowWriteableStream on first push?
There was a problem hiding this comment.
Done in a435ba0 — moved the TTFC measurement into WorkflowServerReadableStream (core), emitting workflow.stream.read on the first non-empty chunk enqueued to the consumer, and dropped the TransformStream wrapper. world-vercel keeps only the workflow.stream.read.connect HTTP span; the recordElapsedSpan helper now lives in @workflow/core. This also makes the span backend-agnostic (any world), and it now includes deframing since it's measured at the reader.
…ansform Per review: move the client-observed time-to-first-chunk measurement out of a TransformStream wrapper in world-vercel and into WorkflowServerReadableStream in core, emitting workflow.stream.read on the first non-empty chunk reaching the consumer. Removes the passthrough, measures at the reader abstraction, and is backend-agnostic. world-vercel keeps the workflow.stream.read.connect HTTP span; the recordElapsedSpan helper now lives in @workflow/core. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Uh oh!
There was an error while loading. Please reload this page.
Signed-off-by: Peter Wielander <mittgfu@gmail.com>
VaguelySerious
left a comment
There was a problem hiding this comment.
Happy to unblock, can adjust metrics later. It seems like the main e2e latency (first write scheduled to first read received on client-side) is still missing
Uh oh!
There was an error while loading. Please reload this page.
| | `workflow.stream.name` | The stream name, on stream write/read spans. | | ||
| | `workflow.stream.operation` | The stream operation: `write`, `write_multi`, `close`, or `read`. | | ||
| | `workflow.stream.write.e2e_ms` | Client→backend round-trip for a stream write, in ms — the backend acks only after capturing the chunk, so this is the client-observed write latency including the network hop. | | ||
| | `workflow.stream.read.ttfc_ms` | Client-observed end-to-end time-to-first-chunk for a live read, in ms — read dispatch → first non-empty chunk in the reader, including the network hop. | |
There was a problem hiding this comment.
| |`workflow.stream.read.ttfc_ms`|Client-observed end-to-end time-to-first-chunk for a live read, in ms — read dispatch → first non-empty chunk in the reader, including the network hop. | | |
| |`workflow.stream.read.ttfc_ms`|Time between opening a read connection and observing and receiving the first chunk back. | |
Co-authored-by: Peter Wielander <mittgfu@gmail.com> Signed-off-by: Karthik Kalyan <105607645+karthikscale3@users.noreply.github.com>
Uh oh!
There was an error while loading. Please reload this page.
…set wording Per review, rename the write attribute to workflow.stream.write.chunk_rtt (it's a per-chunk client<->server round-trip, not a full e2e), update the docs row wording for both write and read attributes, and shorten the changeset. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
karthikscale3
commented
Jul 9, 2026
On the note that the main e2e (first write → first read on the client) is still missing — agreed, and here's where it landed so it's not lost:
So the e2e story is covered as a server-mediated propagation metric rather than a cross-client wall-clock. Happy to iterate on naming/shape in a follow-up as you suggested. |
No backport to The bulk of this commit builds on To override, re-run the Backport to stable workflow manually via |
What
Adds OpenTelemetry client spans that make stream write and read end-to-end latency measurable from the SDK — the client→server timing that server-side metrics can't see (the network hop + client-side legs). These back
getWritable()/run.readable.Write
workflow.stream.writespan (forwrite/writeMulti/close),SpanKind.CLIENT.workflow.stream.write.e2e_msattribute = client→backend round-trip. The PUT is request/response and the backend acks only after capturing the chunk, so this is the client-observed write latency including the network hop. (It equals the span's own duration; exposed as a named attribute for direct querying.)Read
workflow.stream.read.connectspan — dispatch → response headers (the network-connect portion).workflow.stream.readspan — its duration is the client-observed end-to-end time-to-first-chunk: read dispatch → first non-empty chunk in the reader, including the network hop — also stamped asworkflow.stream.read.ttfc_ms. Since that value is only known once the first chunk arrives, it's emitted as a span back-dated to dispatch time (newrecordElapsedSpanhelper). v3+ servers flush a leading zero-length chunk to commit headers, which is skipped. If no chunk ever arrives, noworkflow.stream.readspan is emitted (.connectstill is).Common attributes
workflow.run.id(dotted — the standardized run-id attribute),workflow.stream.name,workflow.stream.operation(write|write_multi|close|read),workflow.stream.start_index(read).How
instrumentedFetchgains three optional, defaulted fields —spanName,attributes, anddurationAttribute(stamps the measured round-trip under a named key). Reuses the existing envelope (W3C trace-context injection, timeout, error mapping, DEBUG logging); no new span layer, no extra requests.response.bodythat emits the back-datedworkflow.stream.readspan on the first non-empty chunk.Backward compatibility
Additive OTEL only. With no OpenTelemetry SDK registered,
trace()/recordElapsedSpanno-op and spans/attributes are dropped — behavior is byte-identical to today. The newinstrumentedFetchoptions are optional and defaulted, so all other callers are unaffected. (Client spans are emitted only when the reader/writer has OTEL registered; server metrics cover the rest.)Docs
Documented the stream spans + attributes in
docs/content/docs/v5/observability/tracing.mdx.Docs Preview
Test plan
packages/world-vercelunit suite: 229 passed (pnpm test).trace-propagation.test.ts: asserts the write span name + attributes +traceparentinjection, and covers the newrecordElapsedSpanback-dated span (duration reflects the elapsed interval + attribute set).pnpm --filter "@workflow/world-vercel..." build).