Skip to content

Reduce Vercel E2E polling load - #3316

Merged
karthikscale3 merged 2 commits into
mainfrom
codex/reduce-e2e-polling-load
Aug 4, 2026
Merged

Reduce Vercel E2E polling load#3316
karthikscale3 merged 2 commits into
mainfrom
codex/reduce-e2e-polling-load

Conversation

@karthikscale3

@karthikscale3karthikscale3 commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Summary

  • make the Run.returnValue polling interval configurable with WORKFLOW_RETURN_VALUE_POLL_INTERVAL_MS
  • set all Vercel E2E jobs to poll every 5 seconds
  • document and test the new runtime setting

Why

The Vercel E2E matrix was running many test variants concurrently, and each active Run.returnValue waited for completion by polling the Workflow API every second. That combined fan-out produced the recent api-workflow traffic spike attributed to Vercel Labs CI.

This keeps the SDK default at 1 second for existing users while reducing CI polling frequency. The tradeoff is up to roughly 4 seconds of additional completion-detection latency in these E2E jobs.

Validation

  • pnpm vitest run packages/core/src/runtime/runs.test.ts (22 tests)
  • pnpm --filter @workflow/core build
  • pnpm --filter @workflow/core typecheck
  • parsed .github/workflows/tests.yml as YAML
  • git diff --check

Note: validation ran on Node 25, while the repository declares support for Node 18/20/22/24; pnpm emitted an engine warning, but all checks above passed.

@changeset-bot

changeset-botBot commented Aug 4, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 2691538

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 16 packages
NameType
@workflow/corePatch
@workflow/buildersPatch
@workflow/cliPatch
@workflow/nextPatch
@workflow/nitroPatch
@workflow/vitestPatch
@workflow/web-sharedPatch
@workflow/webPatch
workflowPatch
@workflow/world-testingPatch
@workflow/astroPatch
@workflow/nestPatch
@workflow/rollupPatch
@workflow/sveltekitPatch
@workflow/vitePatch
@workflow/nuxtPatch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel

vercelBot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

@github-actions

github-actionsBot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

📊 Workflow Benchmarks

commit 2691538 · Tue, 04 Aug 2026 02:25:17 GMT · run logs

Backend: vercel · app: nextjs-turbopack

MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep242 (-77%) 💚1355 🔴 (+21%) 🔻1409 🔴 (+24%) 🔻1780 🔴 (+22%) 🔻30
TTFSstream246 (+6.5%)1369 🔴 (+26%) 🔻1379 🔴 (+24%) 🔻1603 🔴 (+26%) 🔻30
TTFShook + stream396 (-19%) 💚1545 🔴 (+11%)1567 🔴 (+8.9%)1647 🔴 (-7.7%)30
STSO1020 steps (inline)98 (±0%)136 (-11%)159 (-13%)333 (+13%)1018
STSO1020 steps (queue-hop)2460 (-14%)2460 (-14%)2460 (-14%)2460 (-14%)1
WO1020 steps140127 (-5.8%)140127 (-5.8%)140127 (-5.8%)140127 (-5.8%)1
SLstream latency84 (-24%) 💚191 🔴 (-1.5%)213 🔴 (-26%) 💚606 🔴 (+57%) 🔻30
SOstream overhead (text)118 (-4.1%)227 (+5.6%)311 (+26%) 🔻979 (+238%) 🔻30
SOstream overhead (structured)133 (+15%)245 (+25%) 🔻326 (+39%) 🔻912 (+164%) 🔻30
📈 STSO distribution vs main (inline / queue-hop histograms)

1020 steps (inline)

Cumulative STSO time: main 144885ms → this run 136335ms (Δ -8550ms, -6%)

 50-100 ms ┃ main 3 this 4 +1
100-150 ms ████████████████████░░░┃ main 722 this 883 +161
150-200 ms ██┃███ main 234 this 97 -137
200-250 ms ┃ main 37 this 17 -20
250-300 ms ┃ main 12 this 3 -9
300-350 ms ┃ main 5 this 5 +0
350-400 ms ┃ main 4 this 2 -2
400-450 ms ┃ main 0 this 3 +3
500-550 ms ┃ main 0 this 1 +1
600-650 ms ┃ main 1 this 0 -1
650-700 ms ┃ main 0 this 1 +1
800-850 ms ┃ main 0 this 1 +1
1100-1150 ms ┃ main 0 this 1 +1

1020 steps (queue-hop)

Cumulative STSO time: main 2849ms → this run 2460ms (Δ -389ms, -14%)

2000-2500 ms ░░░░░░░░░░░░░░░░░░░░░░░┃ main 0 this 1 +1
2500-3000 ms ┃███████████████████████ main 1 this 0 -1
ℹ️ Metric definitions & methodology

The collapsed STSO distribution section above buckets every step gap of the sequential-steps run (not a sampled window), split by whether the step ending the gap ran inline — in the same warm process as the step before it, so the gap is pure framework overhead — or after a queue-hop — the first step of a fresh process, which pays queue dispatch, client reinit and event-log replay. Bars overlay the two runs: is main, marks where this run lands, bridges the gap when this run has more samples in a bucket.

Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.

Metrics — TTFS: time to first step body (in-deployment start() → first step body, deployment clocks) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · SL: stream latency (in-deployment write → read propagation, readAt - writtenAt) · SO: stream overhead (end-to-end write+consume time beyond the modelled generation window)

Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · stream latency: parallel reader/writer steps on a dedicated stream; SL is the in-deployment write->read propagation (readAt - writtenAt) · stream overhead (text): writer streams 300 variable-length text token deltas paced at 100/s for 3s (a haiku-size LLM's token throughput) while a parallel reader drains the whole stream; SO is the end-to-end write+consume time beyond the 3s generation window (overhead/backpressure) · stream overhead (structured): same workload as stream overhead (text), but each delta is an AI-SDK-style structured object ({ type: 'text-delta', id, text }) instead of a raw string, so the SO gap vs the text scenario is the added serialization cost

🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · SO 250/500/1000

All metrics are measured from deployment-side timestamps only. Runs are triggered by an in-deployment route that stamps the anchor (clientStart) right before start(), so the CI runner’s request and its path through api.vercel.com sit outside every measured window. TTFS = in-deployment start() → first step body (turbo uses the in-process fast path, non-turbo the dispatch path), and includes the VQS dispatch hop plus any /flow cold start. STSO/WO are measured between step bodies on the deployment. SL is measured inside the workflow (parallel reader/writer steps), so it no longer includes the api.vercel.com read path.

Cold starts are kept in the numbers on purpose — they are part of real bursty-workload latency. The workbench deployment cold-starts the /flow invocation for a large fraction of runs, inflating P75+; the Best column shows the fastest (warm-start) sample for comparison.

@github-actions

github-actionsBot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

🧪 E2E Test Results

Some tests failed

❌ Failed E2E Tests

💻 Local Development (11 failed)

astro-stable-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

express-stable-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

fastify-stable-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

hono-stable-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

nextjs-turbopack-canary-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

nextjs-turbopack-stable-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

nextjs-webpack-stable-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

nitro-stable-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

nuxt-stable-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

sveltekit-stable-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

vite-stable-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable
📦 Local Production (11 failed)

astro-stable-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

express-stable-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

fastify-stable-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

hono-stable-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

nextjs-turbopack-canary-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

nextjs-turbopack-stable-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

nextjs-webpack-canary-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

nitro-stable-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

nuxt-stable-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

sveltekit-stable-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

vite-stable-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable
🐘 Local Postgres (12 failed)

astro-stable-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

express-stable-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

fastify-stable-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

hono-stable-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

nextjs-turbopack-canary-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

nextjs-turbopack-stable-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

nextjs-webpack-canary-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

nextjs-webpack-stable-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

nitro-stable-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

nuxt-stable-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

sveltekit-stable-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

vite-stable-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

📋 Other (7 failed)

e2e-local-dev-nest-stable-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

e2e-local-dev-tanstack-start-node (1 failed):

  • fetchWorkflow | wrun_41KZ58TW7X0GQ427GJNXD1K5YC

e2e-local-dev-tanstack-start-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

e2e-local-postgres-nest-stable-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

e2e-local-postgres-tanstack-start-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

e2e-local-prod-nest-stable-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

e2e-local-prod-tanstack-start-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

E2E Test Summary

Summary
PassedFailedSkippedTotal
✅ ▲ Vercel Production278004963276
❌ 💻 Local Development2986114353432
❌ 📦 Local Production3123114543588
❌ 🐘 Local Postgres3278124543744
✅ 🪟 Windows15600156
❌ 📋 Other206174282496
✅ vercel-multi-region270027
Total1441141226716719
Details by Category

✅ ▲ Vercel Production

AppPassedFailedSkipped
✅ astro-node127029
✅ astro-quickjs127029
✅ example-node127029
✅ example-quickjs127029
✅ express-node127029
✅ express-quickjs127029
✅ fastify-node127029
✅ fastify-quickjs127029
✅ hono-node127029
✅ hono-quickjs127029
✅ nextjs-turbopack-node15204
✅ nextjs-webpack-node15204
✅ nextjs-webpack-quickjs15204
✅ nitro-node127029
✅ nitro-quickjs127029
✅ nuxt-node127029
✅ nuxt-quickjs127029
✅ sveltekit-node146010
✅ sveltekit-quickjs146010
✅ vite-node127029
✅ vite-quickjs127029

❌ 💻 Local Development

AppPassedFailedSkipped
✅ astro-stable-node130026
❌ astro-stable-quickjs129126
✅ express-stable-node130026
❌ express-stable-quickjs129126
✅ fastify-stable-node130026
❌ fastify-stable-quickjs129126
✅ hono-stable-node130026
❌ hono-stable-quickjs129126
✅ nextjs-turbopack-canary-node137019
❌ nextjs-turbopack-canary-quickjs136119
✅ nextjs-turbopack-stable-node15600
❌ nextjs-turbopack-stable-quickjs15510
✅ nextjs-webpack-canary-node137019
❌ nextjs-webpack-stable-quickjs15510
✅ nitro-stable-node130026
❌ nitro-stable-quickjs129126
✅ nuxt-stable-node130026
❌ nuxt-stable-quickjs129126
✅ sveltekit-stable-node14907
❌ sveltekit-stable-quickjs14817
✅ vite-stable-node130026
❌ vite-stable-quickjs129126

❌ 📦 Local Production

AppPassedFailedSkipped
✅ astro-stable-node130026
❌ astro-stable-quickjs129126
✅ express-stable-node130026
❌ express-stable-quickjs129126
✅ fastify-stable-node130026
❌ fastify-stable-quickjs129126
✅ hono-stable-node130026
❌ hono-stable-quickjs129126
✅ nextjs-turbopack-canary-node137019
❌ nextjs-turbopack-canary-quickjs136119
✅ nextjs-turbopack-stable-node15600
❌ nextjs-turbopack-stable-quickjs15510
✅ nextjs-webpack-canary-node137019
❌ nextjs-webpack-canary-quickjs136119
✅ nextjs-webpack-stable-node15600
✅ nitro-stable-node130026
❌ nitro-stable-quickjs129126
✅ nuxt-stable-node130026
❌ nuxt-stable-quickjs129126
✅ sveltekit-stable-node14907
❌ sveltekit-stable-quickjs14817
✅ vite-stable-node130026
❌ vite-stable-quickjs129126

❌ 🐘 Local Postgres

AppPassedFailedSkipped
✅ astro-stable-node130026
❌ astro-stable-quickjs129126
✅ express-stable-node130026
❌ express-stable-quickjs129126
✅ fastify-stable-node130026
❌ fastify-stable-quickjs129126
✅ hono-stable-node130026
❌ hono-stable-quickjs129126
✅ nextjs-turbopack-canary-node137019
❌ nextjs-turbopack-canary-quickjs136119
✅ nextjs-turbopack-stable-node15600
❌ nextjs-turbopack-stable-quickjs15510
✅ nextjs-webpack-canary-node137019
❌ nextjs-webpack-canary-quickjs136119
✅ nextjs-webpack-stable-node15600
❌ nextjs-webpack-stable-quickjs15510
✅ nitro-stable-node130026
❌ nitro-stable-quickjs129126
✅ nuxt-stable-node130026
❌ nuxt-stable-quickjs129126
✅ sveltekit-stable-node14907
❌ sveltekit-stable-quickjs14817
✅ vite-stable-node130026
❌ vite-stable-quickjs129126

✅ 🪟 Windows

AppPassedFailedSkipped
✅ nextjs-turbopack-node15600

❌ 📋 Other

AppPassedFailedSkipped
✅ e2e-local-dev-nest-stable-node130026
❌ e2e-local-dev-nest-stable-quickjs129126
❌ e2e-local-dev-tanstack-start-node129126
❌ e2e-local-dev-tanstack-start-quickjs129126
✅ e2e-local-postgres-nest-stable-node130026
❌ e2e-local-postgres-nest-stable-quickjs129126
✅ e2e-local-postgres-tanstack-start-node130026
❌ e2e-local-postgres-tanstack-start-quickjs129126
✅ e2e-local-prod-nest-stable-node130026
❌ e2e-local-prod-nest-stable-quickjs129126
✅ e2e-local-prod-tanstack-start-node130026
❌ e2e-local-prod-tanstack-start-quickjs129126
✅ e2e-vercel-prod-nest-node127029
✅ e2e-vercel-prod-nest-quickjs127029
✅ e2e-vercel-prod-tanstack-start-node127029
✅ e2e-vercel-prod-tanstack-start-quickjs127029

✅ vercel-multi-region

AppPassedFailedSkipped
✅ nextjs-turbopack2700

📋 View full workflow run

@karthikscale3
karthikscale3 marked this pull request as ready for review August 4, 2026 02:04
@karthikscale3
karthikscale3 requested review from a team, fantix and msullivan as code ownersAugust 4, 2026 02:04
Comment thread.github/workflows/tests.yml Outdated
statuses: read
strategy:
fail-fast: false
max-parallel: 6

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure about this - it'll make CI take a total of 50 minutes to run, quite annoying

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah thought it might lower the load further. will skip for now

@karthikscale3
karthikscale3 merged commit e084e08 into mainAug 4, 2026
116 of 162 checks passed
@karthikscale3
karthikscale3 deleted the codex/reduce-e2e-polling-load branch August 4, 2026 02:30
@github-actions

Copy link
Copy Markdown
Contributor

No backport to stable for e084e08 (AI decision).

This commit introduces a new user-configurable runtime setting (WORKFLOW_RETURN_VALUE_POLL_INTERVAL_MS) that adds a config surface and a new internal export to @workflow/core, which is feature/enhancement work rather than a fix for a defect on stable. The motivation is reducing this repo's own CI polling traffic, not a correctness, crash, or resource-leak problem affecting stable users, whose existing 1-second default behavior is unchanged. The CI (tests.yml) tweak and docs addition only exist to exercise and document the new option, so they do not stand on their own as build/release plumbing fixes.

To override, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:

e084e08ac0b4f1830659cb316df50d50962aada4

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@karthikscale3@VaguelySerious