Uh oh!
There was an error while loading. Please reload this page.
- Notifications
You must be signed in to change notification settings - Fork 349
Report RSFS/replay latency telemetry on step terminal events#2929
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Uh oh!
There was an error while loading. Please reload this page.
Merged
Changes from all commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
d3e2687
[core/world/world-vercel] Report RSFS/replay latency telemetry on ste…
VaguelySerious 2d5af95
refactor(core,world,world-vercel): rename replay latency field to fir…
VaguelySerious 187510b
Narrow firstReplay to finalSchedulingReplay per review feedback
VaguelySerious 90ffbdc
Fix: RSFS telemetry is systematically dropped on the turbo optimistic…
vercel[bot] File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Uh oh!
There was an error while loading. Please reload this page.
Jump to
Jump to file
Failed to load files.
Loading
Uh oh!
There was an error while loading. Please reload this page.
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| --- | ||
| '@workflow/core': minor | ||
| '@workflow/world': minor | ||
| '@workflow/world-vercel': minor | ||
| --- | ||
| Report run-started-to-first-step (rsfs) and final-scheduling-replay (finalSchedulingReplay) latency telemetry on step completion events. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -393,6 +393,10 @@ export async function executeStep( | ||
| !isOptimisticInlineStartExplicitlyDisabled())); | ||
| let step: Step; | ||
| // `Date.now()` taken immediately before the `step_started` create is | ||
| // issued (either path below) — anchors RSFS's end point. See | ||
| // StepLatencyEventData.rsfs and the call sites below. | ||
| let stepStartPostSentAtMs: number | undefined; | ||
| // Settled outcome of the in-flight optimistic `step_started`. Handlers are | ||
| // attached synchronously (`.then(ok, err)`) so a fast rejection never | ||
| // surfaces as an unhandledRejection while the body runs. | ||
| @@ -423,8 +427,12 @@ export async function executeStep( | ||
| // round-trip overlaps the body rather than blocking it. Outside turbo the | ||
| // barrier is undefined and this is a plain create. | ||
| const startedPromise = (params.runReadyBarrier ?? Promise.resolve()).then( | ||
| () => | ||
vercel[bot] marked this conversation as resolved.
Uh oh!There was an error while loading. Please reload this page. | ||
| world.events.create(workflowRunId, { | ||
| () => { | ||
| // Taken right before the create fires, not before the barrier — | ||
| // RSFS measures the run_started-to-POST stretch, and the barrier | ||
| // wait IS part of that stretch under turbo. | ||
| stepStartPostSentAtMs = Date.now(); | ||
| return world.events.create(workflowRunId, { | ||
| eventType: 'step_started', | ||
| specVersion: SPEC_VERSION_CURRENT, | ||
| correlationId: stepId, | ||
| @@ -437,7 +445,8 @@ export async function executeStep( | ||
| ? { ownerMessageId: params.ownerMessageId } | ||
| : {}), | ||
| }, | ||
| }) | ||
| }); | ||
| } | ||
| ); | ||
| optimisticStartSettled = startedPromise.then( | ||
| () => ({ ok: true as const }), | ||
| @@ -474,6 +483,7 @@ export async function executeStep( | ||
| params.ownerMessageId !== undefined | ||
| ? { ownerMessageId: params.ownerMessageId } | ||
| : {}; | ||
| stepStartPostSentAtMs = Date.now(); | ||
| const startResult = await world.events.create(workflowRunId, { | ||
| eventType: 'step_started', | ||
| specVersion: SPEC_VERSION_CURRENT, | ||
| @@ -596,6 +606,30 @@ export async function executeStep( | ||
| // catch below) can attach it to step_failed too. | ||
| let latencyEventData: StepLatencyEventData | undefined; | ||
| // Backfill RSFS onto the already-computed telemetry once the optimistic | ||
| // turbo start has settled. On that path the step-start POST fires inside | ||
| // the run-ready barrier's `.then`, so `stepStartPostSentAtMs` — and | ||
| // therefore RSFS — is usually still unset when `latencyEventData` is | ||
| // first computed just before user code (the barrier is still in flight | ||
| // for any non-trivial `run_started` round-trip, which is exactly the | ||
| // slow-run_started case RSFS exists to measure). By the time | ||
| // `reconcileOptimisticStart()` has awaited the barrier the POST timestamp | ||
| // is known, so we patch RSFS in before the terminal event is written. | ||
| // Without this, slow-run_started samples are dropped, biasing RSFS | ||
| // percentiles low (missing-not-at-random). Recomputes only RSFS — | ||
| // TTFS/STSO stay anchored to `executionStartTime` as computed above. | ||
| const backfillOptimisticRsfs = (): void => { | ||
| if (!latencyEventData || latencyEventData.rsfs !== undefined) return; | ||
| const anchorMs = params.latencyTracking?.rsfsAnchorMs; | ||
| if (anchorMs === undefined || stepStartPostSentAtMs === undefined) return; | ||
| latencyEventData.rsfs = Math.max(0, stepStartPostSentAtMs - anchorMs); | ||
| if (span) { | ||
| span.setAttributes({ | ||
| ...Attribute.StepRsfsMs(latencyEventData.rsfs), | ||
| }); | ||
| } | ||
| }; | ||
| try { | ||
| const attempt = step.attempt; | ||
| @@ -656,17 +690,26 @@ export async function executeStep( | ||
| attempt, | ||
| lazyStepStart: params.lazyStepInput !== undefined, | ||
| optimisticStart, | ||
| stepStartPostSentAtMs, | ||
| }); | ||
| if (latencyEventData) { | ||
| // Mirror the latency telemetry onto the step span so traces show | ||
| // TTFS/STSO alongside the flame graph, not just Datadog metrics. | ||
| // TTFS/STSO/RSFS alongside the flame graph, not just Datadog metrics. | ||
| span?.setAttributes({ | ||
| ...(latencyEventData.ttfs !== undefined | ||
| ? Attribute.StepTtfsMs(latencyEventData.ttfs) | ||
| : {}), | ||
| ...(latencyEventData.stso !== undefined | ||
| ? Attribute.StepStsoMs(latencyEventData.stso) | ||
| : {}), | ||
| ...(latencyEventData.rsfs !== undefined | ||
| ? Attribute.StepRsfsMs(latencyEventData.rsfs) | ||
| : {}), | ||
| ...(latencyEventData.finalSchedulingReplay !== undefined | ||
| ? Attribute.StepFinalSchedulingReplayMs( | ||
| latencyEventData.finalSchedulingReplay | ||
| ) | ||
| : {}), | ||
| ...Attribute.StepLatencyOptimizations( | ||
| latencyEventData.optimizations ?? [] | ||
| ), | ||
| @@ -802,6 +845,9 @@ export async function executeStep( | ||
| if (optimisticStart) { | ||
| const reconcile = await reconcileOptimisticStart(); | ||
| if (reconcile) return reconcile; | ||
| // Barrier resolved — the step-start POST timestamp is now known, so | ||
| // RSFS can be attached to the step_completed event below. | ||
| backfillOptimisticRsfs(); | ||
| } | ||
| // Commit must-be-durable ops (e.g. a step-initiated abort's | ||
| @@ -830,6 +876,8 @@ export async function executeStep( | ||
| if (optimisticStart) { | ||
| const reconcile = await reconcileOptimisticStart(); | ||
| if (reconcile) return reconcile; | ||
| // Barrier resolved — attach RSFS to the step_failed event(s) below. | ||
| backfillOptimisticRsfs(); | ||
| } | ||
| // Order any must-be-durable ops (e.g. a step-initiated abort's | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Oops, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could we either aggregate replay time across every pre-first-step replay pass, or narrow this metric definition/name? This captures only the current suspension
runWorkflow()duration. Valid RSFS/TTFS paths such as workflow-bodysetAttributes()can replay more than once before the first step, so RSFS covers the whole detour whilereplayreports only the final pass; a redelivery can omit earlier work too. That makes it misleading to interpretreplayas the portion of RSFS spent in replay compute. Accumulating until the first step (or renaming this to something likefinal_scheduling_replay_ms) would make the semantics explicit. Also, the individual-pass duration is already represented by the existingworkflow.run …span; if this metric is intentionally duplicated to provide unsampled/full-population distributions, can we document that rationale?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Doing
Documenting better
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Addressed in 187510b: renamed to
finalSchedulingReplay(lifecycle.step.final_scheduling_replay_ms) and sharpened the comments at both measurement sites — they now state explicitly that this is the finalrunWorkflow()pass only (not accumulated across pre-first-step passes, so not "the replay portion of RSFS"), and that the duplication with theworkflow.runspan is deliberate: the server emits it as an unsampled full-population distribution, which the ~7%-sampled span population can't provide.Server-side counterpart: #626 merged mid-rename with the old name, so the matching server rename is in vercel/workflow-server#628 (deploys before this ships, or the field would be dropped by the allowlist).