Skip to content

[world-testing] fix flow invocation counter race in inline-execution tests - #2043

Merged
TooTallNate merged 2 commits into
mainfrom
nate/fix-flow-invocation-count-race
May 20, 2026
Merged

[world-testing] fix flow invocation counter race in inline-execution tests#2043
TooTallNate merged 2 commits into
mainfrom
nate/fix-flow-invocation-count-race

Conversation

@TooTallNate

Copy link
Copy Markdown
Member

Summary

Fixes the flaky unit test failure seen on multiple PRs (e.g. PR #2013, PR #2035):

FAIL test/spec.test.ts > sequential steps complete in a single flow invocation
AssertionError: expected +0 to be 1 // Object.is equality
❯ ../world-testing/src/inline-execution.mts:45:21

Root cause

The test server's flow handler awaited flowPOST before incrementing the per-run invocation counter:

constcloned=ctx.req.raw.clone();constresponse=awaitflowPOST(ctx.req.raw);// (1) workflow may complete & flush DB here// (2) read cloned body, increment counterreturnresponse;

That races with the test, which polls getRun() until it sees status === 'completed' and then immediately queries /_flow-invocations/:runId:

  1. Inside flowPOST, the workflow runs and writes the completed run to the DB.
  2. The test's vi.waitFor poll sees status === 'completed'.
  3. The test fetches the flow invocation count.
  4. The counter increment in the handler hadn't run yet → count returns 0 instead of 1.

Locally the race rarely loses (the cloned-body read is fast) but on slower CI runners it loses often enough to be visible across many PRs.

Fix

Increment the counter before awaiting flowPOST so the count is observable as soon as the run transitions to completed. Same logic, just reordered.

Test plan

  • Re-ran packages/world-testing/test/embedded.test.ts locally — all 9 tests pass, including the four inline-execution.mts cases that drive the counter.
  • Could not reproduce the race locally (Docker on this box is broken so postgres test was skipped), but the fix removes the only await between request entry and counter increment, so there is no longer a window where the run can be flushed to the DB before the counter is bumped.

…tests
The test server's flow handler awaited flowPOST before incrementing the
invocation counter, which races with the test polling getRun() to see
the completed status. When the workflow completed inside flowPOST and
flushed the run to the DB, the test could observe the completed state
and immediately query /_flow-invocations before the counter was bumped,
yielding a flaky 'expected 0 to be 1' assertion.
Increment the counter before awaiting flowPOST so the count is
observable as soon as the run transitions to completed.
CopilotAI review requested due to automatic review settings May 20, 2026 21:13
@TooTallNate
TooTallNate requested a review from a team as a code ownerMay 20, 2026 21:13
@changeset-bot

changeset-botBot commented May 20, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: c11f633

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
NameType
@workflow/world-testingPatch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel

vercelBot commented May 20, 2026

Copy link
Copy Markdown
Contributor

@github-actions

github-actionsBot commented May 20, 2026

Copy link
Copy Markdown
Contributor

🧪 E2E Test Results

Some tests failed

Summary

PassedFailedSkippedTotal
❌ ▲ Vercel Production119822191419
✅ 💻 Local Development158702191806
✅ 📦 Local Production158702191806
✅ 🐘 Local Postgres158702191806
✅ 🪟 Windows12900129
❌ 📋 Other7261176903
Total6814310527869

❌ Failed Tests

▲ Vercel Production (2 failed)

astro (2 failed):

  • outputStreamInsideStepWorkflow - getWritable() called inside step functions | wrun_01KS3RNPMVVB6C8PR0168PSSWT | 🔍 observability
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
📋 Other (1 failed)

e2e-vercel-prod-tanstack-start (1 failed):

  • distributedAbortController - manual abort triggers signal | wrun_01KS3S811F65HFD53TRWAE1NP6

Details by Category

❌ ▲ Vercel Production
AppPassedFailedSkipped
❌ astro101226
✅ example103026
✅ express103026
✅ fastify103026
✅ hono103026
✅ nextjs-turbopack12702
✅ nextjs-webpack12702
✅ nitro103026
✅ nuxt103026
✅ sveltekit12207
✅ vite103026
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable104025
✅ express-stable104025
✅ fastify-stable104025
✅ hono-stable104025
✅ nextjs-turbopack-canary110019
✅ nextjs-turbopack-stable-lazy-discovery-disabled12900
✅ nextjs-turbopack-stable-lazy-discovery-enabled12900
✅ nextjs-webpack-canary110019
✅ nextjs-webpack-stable-lazy-discovery-disabled12900
✅ nextjs-webpack-stable-lazy-discovery-enabled12900
✅ nitro-stable104025
✅ nuxt-stable104025
✅ sveltekit-stable12306
✅ vite-stable104025
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable104025
✅ express-stable104025
✅ fastify-stable104025
✅ hono-stable104025
✅ nextjs-turbopack-canary110019
✅ nextjs-turbopack-stable-lazy-discovery-disabled12900
✅ nextjs-turbopack-stable-lazy-discovery-enabled12900
✅ nextjs-webpack-canary110019
✅ nextjs-webpack-stable-lazy-discovery-disabled12900
✅ nextjs-webpack-stable-lazy-discovery-enabled12900
✅ nitro-stable104025
✅ nuxt-stable104025
✅ sveltekit-stable12306
✅ vite-stable104025
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable104025
✅ express-stable104025
✅ fastify-stable104025
✅ hono-stable104025
✅ nextjs-turbopack-canary110019
✅ nextjs-turbopack-stable-lazy-discovery-disabled12900
✅ nextjs-turbopack-stable-lazy-discovery-enabled12900
✅ nextjs-webpack-canary110019
✅ nextjs-webpack-stable-lazy-discovery-disabled12900
✅ nextjs-webpack-stable-lazy-discovery-enabled12900
✅ nitro-stable104025
✅ nuxt-stable104025
✅ sveltekit-stable12306
✅ vite-stable104025
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack12900
❌ 📋 Other
AppPassedFailedSkipped
✅ e2e-local-dev-nest-stable104025
✅ e2e-local-dev-tanstack-start-104025
✅ e2e-local-postgres-nest-stable104025
✅ e2e-local-postgres-tanstack-start-104025
✅ e2e-local-prod-nest-stable104025
✅ e2e-local-prod-tanstack-start-104025
❌ e2e-vercel-prod-tanstack-start102126

📋 View full workflow run


Some E2E test jobs failed:

  • Vercel Prod: failure
  • Local Dev: success
  • Local Prod: success
  • Local Postgres: success
  • Windows: success

Check the workflow run for details.

@github-actions

github-actionsBot commented May 20, 2026

Copy link
Copy Markdown
Contributor

📊 Benchmark Results

📈 Comparing against baseline from main branch. Green 🟢 = faster, Red 🔺 = slower.

workflow with no steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Express0.029s (-34.5% 🟢)1.005s (~)0.976s101.00x
💻 LocalNitro0.030s (-31.3% 🟢)1.005s (~)0.976s101.02x
💻 LocalNext.js (Turbopack)0.048s1.006s0.958s101.64x
🐘 PostgresExpress0.049s (-16.2% 🟢)1.011s (~)0.963s101.68x
🐘 PostgresNitro0.061s (-36.2% 🟢)1.013s (-2.8%)0.953s102.09x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 1 step

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Express1.067s (-5.2% 🟢)2.005s (~)0.939s101.00x
💻 LocalNitro1.072s (-5.2% 🟢)2.006s (~)0.934s101.00x
🐘 PostgresExpress1.084s (-5.4% 🟢)2.010s (~)0.926s101.02x
🐘 PostgresNitro1.103s (-3.3%)2.011s (~)0.908s101.03x
💻 LocalNext.js (Turbopack)1.112s2.005s0.894s101.04x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 10 sequential steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Express10.384s (-4.9%)11.021s (~)0.636s31.00x
🐘 PostgresExpress10.401s (-5.1% 🟢)11.014s (~)0.613s31.00x
💻 LocalNitro10.408s (-4.9%)11.023s (~)0.615s31.00x
🐘 PostgresNitro10.513s (-3.3%)11.017s (~)0.503s31.01x
💻 LocalNext.js (Turbopack)10.654s11.022s0.368s31.03x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 25 sequential steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Express13.401s (-10.5% 🟢)14.026s (-6.7% 🟢)0.626s51.00x
🐘 PostgresExpress13.406s (-8.1% 🟢)14.016s (-6.7% 🟢)0.610s51.00x
💻 LocalNitro13.447s (-10.7% 🟢)14.027s (-12.5% 🟢)0.581s51.00x
🐘 PostgresNitro13.729s (-5.9% 🟢)14.023s (-6.7% 🟢)0.293s51.02x
💻 LocalNext.js (Turbopack)14.118s14.828s0.710s51.05x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 50 sequential steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Express11.822s (-28.8% 🟢)12.022s (-29.4% 🟢)0.200s81.00x
💻 LocalNitro11.872s (-29.3% 🟢)12.024s (-29.4% 🟢)0.151s81.00x
🐘 PostgresExpress11.892s (-15.1% 🟢)12.017s (-17.7% 🟢)0.125s81.01x
🐘 PostgresNitro12.598s (-9.8% 🟢)13.025s (-9.0% 🟢)0.427s71.07x
💻 LocalNext.js (Turbopack)13.002s13.309s0.308s71.10x
🐘 PostgresNext.js (Turbopack)⚠️missing----
Promise.all with 10 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.135s (-9.9% 🟢)2.007s (~)0.872s151.00x
💻 LocalExpress1.163s (-21.9% 🟢)2.006s (~)0.843s151.02x
💻 LocalNitro1.170s (-28.3% 🟢)2.006s (-3.3%)0.836s151.03x
🐘 PostgresNitro1.197s (-6.1% 🟢)2.010s (~)0.814s151.05x
💻 LocalNext.js (Turbopack)1.254s2.006s0.751s151.10x
🐘 PostgresNext.js (Turbopack)⚠️missing----
Promise.all with 25 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.187s (-49.7% 🟢)2.006s (-33.3% 🟢)0.819s151.00x
🐘 PostgresNitro1.253s (-46.7% 🟢)2.010s (-33.2% 🟢)0.757s151.06x
💻 LocalExpress1.652s (-44.1% 🟢)2.005s (-41.9% 🟢)0.354s151.39x
💻 LocalNext.js (Turbopack)1.682s2.006s0.324s151.42x
💻 LocalNitro1.683s (-46.5% 🟢)2.006s (-48.4% 🟢)0.322s151.42x
🐘 PostgresNext.js (Turbopack)⚠️missing----
Promise.all with 50 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.304s (-62.6% 🟢)2.007s (-50.0% 🟢)0.703s151.00x
🐘 PostgresNitro1.390s (-60.1% 🟢)2.010s (-49.9% 🟢)0.620s151.07x
💻 LocalNitro4.483s (-46.3% 🟢)5.012s (-44.4% 🟢)0.529s63.44x
💻 LocalExpress4.632s (-44.5% 🟢)5.178s (-42.6% 🟢)0.546s63.55x
💻 LocalNext.js (Turbopack)4.684s5.179s0.495s63.59x
🐘 PostgresNext.js (Turbopack)⚠️missing----
Promise.race with 10 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.139s (-9.4% 🟢)2.008s (~)0.869s151.00x
🐘 PostgresNitro1.211s (-3.7%)2.011s (~)0.800s151.06x
💻 LocalNext.js (Turbopack)1.317s2.006s0.689s151.16x
💻 LocalExpress1.364s (-28.0% 🟢)2.006s (-15.1% 🟢)0.642s151.20x
💻 LocalNitro1.372s (-26.5% 🟢)2.006s (-14.3% 🟢)0.634s151.20x
🐘 PostgresNext.js (Turbopack)⚠️missing----
Promise.race with 25 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.195s (-49.0% 🟢)2.009s (-33.3% 🟢)0.814s151.00x
🐘 PostgresNitro1.274s (-45.5% 🟢)2.013s (-33.1% 🟢)0.738s151.07x
💻 LocalExpress1.833s (-41.5% 🟢)2.222s (-41.0% 🟢)0.389s141.53x
💻 LocalNitro1.916s (-37.5% 🟢)2.316s (-40.4% 🟢)0.400s131.60x
💻 LocalNext.js (Turbopack)1.927s2.393s0.466s131.61x
🐘 PostgresNext.js (Turbopack)⚠️missing----
Promise.race with 50 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.290s (-63.1% 🟢)2.008s (-49.9% 🟢)0.718s151.00x
🐘 PostgresNitro1.430s (-58.9% 🟢)2.014s (-49.8% 🟢)0.584s151.11x
💻 LocalExpress4.559s (-48.2% 🟢)5.346s (-42.3% 🟢)0.787s63.53x
💻 LocalNext.js (Turbopack)4.916s5.352s0.436s63.81x
💻 LocalNitro5.467s (-40.2% 🟢)6.013s (-40.0% 🟢)0.547s54.24x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 10 sequential data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.435s (-48.2% 🟢)1.006s (-1.7%)0.572s601.00x
💻 LocalNitro0.454s (-53.7% 🟢)1.004s (-8.2% 🟢)0.549s601.05x
💻 LocalExpress0.468s (-52.4% 🟢)1.004s (-6.7% 🟢)0.535s601.08x
🐘 PostgresNitro0.568s (-30.8% 🟢)1.008s (~)0.441s601.31x
💻 LocalNext.js (Turbopack)0.720s1.004s0.284s601.66x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 25 sequential data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.004s (-49.2% 🟢)1.292s (-42.8% 🟢)0.288s701.00x
💻 LocalExpress1.152s (-61.8% 🟢)2.006s (-44.1% 🟢)0.854s451.15x
💻 LocalNitro1.162s (-61.7% 🟢)2.006s (-46.6% 🟢)0.844s451.16x
🐘 PostgresNitro1.385s (-28.1% 🟢)2.033s (-3.2%)0.648s451.38x
💻 LocalNext.js (Turbopack)1.807s2.051s0.244s441.80x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 50 sequential data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.965s (-50.8% 🟢)2.334s (-46.6% 🟢)0.369s521.00x
🐘 PostgresNitro2.560s (-37.6% 🟢)3.035s (-34.1% 🟢)0.475s401.30x
💻 LocalExpress2.619s (-71.6% 🟢)3.007s (-70.0% 🟢)0.388s401.33x
💻 LocalNitro2.637s (-71.6% 🟢)3.008s (-70.0% 🟢)0.370s401.34x
💻 LocalNext.js (Turbopack)3.859s4.074s0.216s301.96x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 10 concurrent data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.186s (-34.0% 🟢)1.022s (+1.5%)0.836s591.00x
🐘 PostgresNitro0.218s (-23.1% 🟢)1.007s (~)0.789s601.17x
💻 LocalExpress0.375s (-33.1% 🟢)1.004s (~)0.629s602.01x
💻 LocalNitro0.387s (-36.0% 🟢)1.004s (-1.7%)0.617s602.07x
💻 LocalNext.js (Turbopack)0.536s1.004s0.468s602.88x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 25 concurrent data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.277s (-45.7% 🟢)1.006s (~)0.729s901.00x
🐘 PostgresNitro0.356s (-28.4% 🟢)1.008s (~)0.652s901.28x
💻 LocalExpress2.068s (-17.7% 🟢)2.580s (-14.3% 🟢)0.512s357.46x
💻 LocalNitro2.099s (-17.3% 🟢)2.655s (-11.8% 🟢)0.556s347.58x
💻 LocalNext.js (Turbopack)2.335s2.976s0.641s318.43x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 50 concurrent data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.603s (-26.4% 🟢)1.006s (-1.1%)0.403s1201.00x
🐘 PostgresNitro0.723s (-8.6% 🟢)1.008s (~)0.285s1201.20x
💻 LocalExpress9.546s (-14.7% 🟢)10.111s (-15.3% 🟢)0.565s1215.84x
💻 LocalNitro9.729s (-13.1% 🟢)10.444s (-10.5% 🟢)0.715s1216.14x
💻 LocalNext.js (Turbopack)10.871s11.575s0.705s1118.04x
🐘 PostgresNext.js (Turbopack)⚠️missing----
Stream Benchmarks(includes TTFB metrics)
workflow with stream

💻 Local Development

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Express1.127s (+466.0% 🔺)2.005s (+99.6% 🔺)0.010s (-17.4% 🟢)2.017s (+98.1% 🔺)0.890s101.00x
🐘 PostgresExpress1.131s (+451.7% 🔺)1.995s (+99.8% 🔺)0.001s (-37.5% 🟢)2.010s (+98.7% 🔺)0.878s101.00x
💻 LocalNitro1.143s (+434.9% 🔺)2.005s (+99.6% 🔺)0.011s (-12.8% 🟢)2.018s (+98.1% 🔺)0.875s101.01x
🐘 PostgresNitro1.173s (+472.0% 🔺)1.998s (+99.9% 🔺)0.002s (+20.0% 🔺)2.013s (+99.1% 🔺)0.841s101.04x
💻 LocalNext.js (Turbopack)1.174s2.003s0.010s2.017s0.843s101.04x
🐘 PostgresNext.js (Turbopack)⚠️missing-----
stream pipeline with 5 transform steps (1MB)

💻 Local Development

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.498s (+137.7% 🔺)1.999s (+98.6% 🔺)0.004s (+5.3% 🔺)2.023s (+97.8% 🔺)0.526s301.00x
💻 LocalNext.js (Turbopack)1.669s2.009s0.011s2.022s0.354s301.11x
💻 LocalNitro1.691s (+101.6% 🔺)2.010s (+98.6% 🔺)0.009s (-1.3%)2.200s (+97.2% 🔺)0.509s281.13x
🐘 PostgresNitro1.701s (+172.5% 🔺)2.008s (+99.5% 🔺)0.005s (+11.2% 🔺)2.032s (+98.7% 🔺)0.331s301.14x
💻 LocalExpress1.909s (+152.2% 🔺)2.011s (+95.4% 🔺)0.009s (-0.6%)2.423s (+133.0% 🔺)0.514s251.27x
🐘 PostgresNext.js (Turbopack)⚠️missing-----
10 parallel streams (1MB each)

💻 Local Development

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.662s (-31.1% 🟢)1.048s (-18.0% 🟢)0.000s (+21.1% 🔺)1.059s (-18.9% 🟢)0.397s571.00x
🐘 PostgresNitro0.795s (-17.9% 🟢)1.049s (-16.0% 🟢)0.000s (+71.4% 🔺)1.082s (-13.9% 🟢)0.287s561.20x
💻 LocalExpress1.321s (+7.9% 🔺)2.015s (~)0.000s (-60.0% 🟢)2.017s (~)0.696s302.00x
💻 LocalNitro1.343s (+9.9% 🔺)2.015s (~)0.000s (+200.0% 🔺)2.017s (~)0.674s302.03x
💻 LocalNext.js (Turbopack)1.480s2.014s0.000s2.017s0.537s302.24x
🐘 PostgresNext.js (Turbopack)⚠️missing-----
fan-out fan-in 10 streams (1MB each)

💻 Local Development

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.258s (-29.0% 🟢)1.898s (-12.8% 🟢)0.000s (+Infinity% 🔺)1.931s (-12.2% 🟢)0.673s321.00x
🐘 PostgresNitro1.634s (-8.8% 🟢)2.263s (+5.7% 🔺)0.000s (+7.7% 🔺)2.309s (+6.2% 🔺)0.675s261.30x
💻 LocalNext.js (Turbopack)2.815s3.293s0.001s3.298s0.483s192.24x
💻 LocalExpress3.025s (-12.8% 🟢)3.556s (-11.8% 🟢)0.000s (-55.9% 🟢)3.562s (-11.8% 🟢)0.536s172.41x
💻 LocalNitro3.082s (-9.0% 🟢)3.778s (-6.3% 🟢)0.000s (-18.0% 🟢)3.782s (-6.3% 🟢)0.700s162.45x
🐘 PostgresNext.js (Turbopack)⚠️missing-----

Summary

Fastest Framework by World

Winner determined by most benchmark wins

World🥇 Fastest FrameworkWins
💻 LocalExpress16/21
🐘 PostgresExpress21/21
Fastest World by Framework

Winner determined by most benchmark wins

Framework🥇 Fastest WorldWins
Express🐘 Postgres15/21
Next.js (Turbopack)💻 Local21/21
Nitro🐘 Postgres11/21
Column Definitions
  • Workflow Time: Runtime reported by workflow (completedAt - createdAt) - primary metric
  • TTFB: Time to First Byte - time from workflow start until first stream byte received (stream benchmarks only)
  • Slurp: Time from first byte to complete stream consumption (stream benchmarks only)
  • Wall Time: Total testbench time (trigger workflow + poll for result)
  • Overhead: Testbench overhead (Wall Time - Workflow Time)
  • Samples: Number of benchmark iterations run
  • vs Fastest: How much slower compared to the fastest configuration for this benchmark

Worlds:

  • 💻 Local: In-memory filesystem world (local development)
  • 🐘 Postgres: PostgreSQL database world (local development)
  • ▲ Vercel: Vercel production/preview deployment
  • 🌐 Turso: Community world (local development)
  • 🌐 MongoDB: Community world (local development)
  • 🌐 Redis: Community world (local development)
  • 🌐 Jazz: Community world (local development)
  • 🌐 Redis: Community world (local development)
  • 🌐 Redis + BullMQ: Community world (local development)
  • 🌐 Cloudflare: Community world (local development)
  • 🌐 MySQL: Community world (local development)
  • 🌐 Azure: Community world (local development)
  • 🌐 NATS JetStream: Community world (local development)
  • 🌐 Upstash: Community world (local development)

📋 View full workflow run


Some benchmark jobs failed:

  • Local: success
  • Postgres: success
  • Vercel: failure

Check the workflow run for details.

⚠️Community world benchmarks failed (non-blocking):

  • Community Worlds: failure

Check the workflow run for details.

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes a race in @workflow/world-testing’s inline-execution test server where the per-run flow invocation counter could be incremented after the workflow completion became observable, causing flaky assertions in CI.

Changes:

  • Reorders the flow handler to increment the per-run invocation counter before awaiting/returning flowPOST.
  • Adds an explanatory comment documenting the race and why the ordering matters.
  • Adds a changeset bumping @workflow/world-testing with a patch release note.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

FileDescription
packages/world-testing/src/server.mtsMoves flow invocation counting ahead of the flow handler execution to eliminate the observed flake window.
.changeset/fix-inline-execution-flow-count-race.mdAdds a patch changeset describing the test-flake fix for @workflow/world-testing.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@TooTallNate
TooTallNate disabled auto-merge May 20, 2026 23:05
@TooTallNate
TooTallNate merged commit 96e2d3c into mainMay 20, 2026
108 of 118 checks passed
@TooTallNate
TooTallNate deleted the nate/fix-flow-invocation-count-race branch May 20, 2026 23:05
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@TooTallNate@karthikscale3
, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
[world-testing] fix flow invocation counter race in inline-execution tests by TooTallNate · Pull Request #2043 · vercel/workflow · GitHub
Skip to content

[world-testing] fix flow invocation counter race in inline-execution tests - #2043

Merged
TooTallNate merged 2 commits into
mainfrom
nate/fix-flow-invocation-count-race
May 20, 2026
Merged

[world-testing] fix flow invocation counter race in inline-execution tests#2043
TooTallNate merged 2 commits into
mainfrom
nate/fix-flow-invocation-count-race

Conversation

@TooTallNate

Copy link
Copy Markdown
Member

Summary

Fixes the flaky unit test failure seen on multiple PRs (e.g. PR #2013, PR #2035):

FAIL test/spec.test.ts > sequential steps complete in a single flow invocation
AssertionError: expected +0 to be 1 // Object.is equality
❯ ../world-testing/src/inline-execution.mts:45:21

Root cause

The test server's flow handler awaited flowPOST before incrementing the per-run invocation counter:

constcloned=ctx.req.raw.clone();constresponse=awaitflowPOST(ctx.req.raw);// (1) workflow may complete & flush DB here// (2) read cloned body, increment counterreturnresponse;

That races with the test, which polls getRun() until it sees status === 'completed' and then immediately queries /_flow-invocations/:runId:

  1. Inside flowPOST, the workflow runs and writes the completed run to the DB.
  2. The test's vi.waitFor poll sees status === 'completed'.
  3. The test fetches the flow invocation count.
  4. The counter increment in the handler hadn't run yet → count returns 0 instead of 1.

Locally the race rarely loses (the cloned-body read is fast) but on slower CI runners it loses often enough to be visible across many PRs.

Fix

Increment the counter before awaiting flowPOST so the count is observable as soon as the run transitions to completed. Same logic, just reordered.

Test plan

  • Re-ran packages/world-testing/test/embedded.test.ts locally — all 9 tests pass, including the four inline-execution.mts cases that drive the counter.
  • Could not reproduce the race locally (Docker on this box is broken so postgres test was skipped), but the fix removes the only await between request entry and counter increment, so there is no longer a window where the run can be flushed to the DB before the counter is bumped.

…tests
The test server's flow handler awaited flowPOST before incrementing the
invocation counter, which races with the test polling getRun() to see
the completed status. When the workflow completed inside flowPOST and
flushed the run to the DB, the test could observe the completed state
and immediately query /_flow-invocations before the counter was bumped,
yielding a flaky 'expected 0 to be 1' assertion.
Increment the counter before awaiting flowPOST so the count is
observable as soon as the run transitions to completed.
CopilotAI review requested due to automatic review settings May 20, 2026 21:13
@TooTallNate
TooTallNate requested a review from a team as a code ownerMay 20, 2026 21:13
@changeset-bot

changeset-botBot commented May 20, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: c11f633

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
NameType
@workflow/world-testingPatch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel

vercelBot commented May 20, 2026

Copy link
Copy Markdown
Contributor

@github-actions

github-actionsBot commented May 20, 2026

Copy link
Copy Markdown
Contributor

🧪 E2E Test Results

Some tests failed

Summary

PassedFailedSkippedTotal
❌ ▲ Vercel Production119822191419
✅ 💻 Local Development158702191806
✅ 📦 Local Production158702191806
✅ 🐘 Local Postgres158702191806
✅ 🪟 Windows12900129
❌ 📋 Other7261176903
Total6814310527869

❌ Failed Tests

▲ Vercel Production (2 failed)

astro (2 failed):

  • outputStreamInsideStepWorkflow - getWritable() called inside step functions | wrun_01KS3RNPMVVB6C8PR0168PSSWT | 🔍 observability
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
📋 Other (1 failed)

e2e-vercel-prod-tanstack-start (1 failed):

  • distributedAbortController - manual abort triggers signal | wrun_01KS3S811F65HFD53TRWAE1NP6

Details by Category

❌ ▲ Vercel Production
AppPassedFailedSkipped
❌ astro101226
✅ example103026
✅ express103026
✅ fastify103026
✅ hono103026
✅ nextjs-turbopack12702
✅ nextjs-webpack12702
✅ nitro103026
✅ nuxt103026
✅ sveltekit12207
✅ vite103026
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable104025
✅ express-stable104025
✅ fastify-stable104025
✅ hono-stable104025
✅ nextjs-turbopack-canary110019
✅ nextjs-turbopack-stable-lazy-discovery-disabled12900
✅ nextjs-turbopack-stable-lazy-discovery-enabled12900
✅ nextjs-webpack-canary110019
✅ nextjs-webpack-stable-lazy-discovery-disabled12900
✅ nextjs-webpack-stable-lazy-discovery-enabled12900
✅ nitro-stable104025
✅ nuxt-stable104025
✅ sveltekit-stable12306
✅ vite-stable104025
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable104025
✅ express-stable104025
✅ fastify-stable104025
✅ hono-stable104025
✅ nextjs-turbopack-canary110019
✅ nextjs-turbopack-stable-lazy-discovery-disabled12900
✅ nextjs-turbopack-stable-lazy-discovery-enabled12900
✅ nextjs-webpack-canary110019
✅ nextjs-webpack-stable-lazy-discovery-disabled12900
✅ nextjs-webpack-stable-lazy-discovery-enabled12900
✅ nitro-stable104025
✅ nuxt-stable104025
✅ sveltekit-stable12306
✅ vite-stable104025
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable104025
✅ express-stable104025
✅ fastify-stable104025
✅ hono-stable104025
✅ nextjs-turbopack-canary110019
✅ nextjs-turbopack-stable-lazy-discovery-disabled12900
✅ nextjs-turbopack-stable-lazy-discovery-enabled12900
✅ nextjs-webpack-canary110019
✅ nextjs-webpack-stable-lazy-discovery-disabled12900
✅ nextjs-webpack-stable-lazy-discovery-enabled12900
✅ nitro-stable104025
✅ nuxt-stable104025
✅ sveltekit-stable12306
✅ vite-stable104025
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack12900
❌ 📋 Other
AppPassedFailedSkipped
✅ e2e-local-dev-nest-stable104025
✅ e2e-local-dev-tanstack-start-104025
✅ e2e-local-postgres-nest-stable104025
✅ e2e-local-postgres-tanstack-start-104025
✅ e2e-local-prod-nest-stable104025
✅ e2e-local-prod-tanstack-start-104025
❌ e2e-vercel-prod-tanstack-start102126

📋 View full workflow run


Some E2E test jobs failed:

  • Vercel Prod: failure
  • Local Dev: success
  • Local Prod: success
  • Local Postgres: success
  • Windows: success

Check the workflow run for details.

@github-actions

github-actionsBot commented May 20, 2026

Copy link
Copy Markdown
Contributor

📊 Benchmark Results

📈 Comparing against baseline from main branch. Green 🟢 = faster, Red 🔺 = slower.

workflow with no steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Express0.029s (-34.5% 🟢)1.005s (~)0.976s101.00x
💻 LocalNitro0.030s (-31.3% 🟢)1.005s (~)0.976s101.02x
💻 LocalNext.js (Turbopack)0.048s1.006s0.958s101.64x
🐘 PostgresExpress0.049s (-16.2% 🟢)1.011s (~)0.963s101.68x
🐘 PostgresNitro0.061s (-36.2% 🟢)1.013s (-2.8%)0.953s102.09x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 1 step

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Express1.067s (-5.2% 🟢)2.005s (~)0.939s101.00x
💻 LocalNitro1.072s (-5.2% 🟢)2.006s (~)0.934s101.00x
🐘 PostgresExpress1.084s (-5.4% 🟢)2.010s (~)0.926s101.02x
🐘 PostgresNitro1.103s (-3.3%)2.011s (~)0.908s101.03x
💻 LocalNext.js (Turbopack)1.112s2.005s0.894s101.04x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 10 sequential steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Express10.384s (-4.9%)11.021s (~)0.636s31.00x
🐘 PostgresExpress10.401s (-5.1% 🟢)11.014s (~)0.613s31.00x
💻 LocalNitro10.408s (-4.9%)11.023s (~)0.615s31.00x
🐘 PostgresNitro10.513s (-3.3%)11.017s (~)0.503s31.01x
💻 LocalNext.js (Turbopack)10.654s11.022s0.368s31.03x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 25 sequential steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Express13.401s (-10.5% 🟢)14.026s (-6.7% 🟢)0.626s51.00x
🐘 PostgresExpress13.406s (-8.1% 🟢)14.016s (-6.7% 🟢)0.610s51.00x
💻 LocalNitro13.447s (-10.7% 🟢)14.027s (-12.5% 🟢)0.581s51.00x
🐘 PostgresNitro13.729s (-5.9% 🟢)14.023s (-6.7% 🟢)0.293s51.02x
💻 LocalNext.js (Turbopack)14.118s14.828s0.710s51.05x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 50 sequential steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Express11.822s (-28.8% 🟢)12.022s (-29.4% 🟢)0.200s81.00x
💻 LocalNitro11.872s (-29.3% 🟢)12.024s (-29.4% 🟢)0.151s81.00x
🐘 PostgresExpress11.892s (-15.1% 🟢)12.017s (-17.7% 🟢)0.125s81.01x
🐘 PostgresNitro12.598s (-9.8% 🟢)13.025s (-9.0% 🟢)0.427s71.07x
💻 LocalNext.js (Turbopack)13.002s13.309s0.308s71.10x
🐘 PostgresNext.js (Turbopack)⚠️missing----
Promise.all with 10 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.135s (-9.9% 🟢)2.007s (~)0.872s151.00x
💻 LocalExpress1.163s (-21.9% 🟢)2.006s (~)0.843s151.02x
💻 LocalNitro1.170s (-28.3% 🟢)2.006s (-3.3%)0.836s151.03x
🐘 PostgresNitro1.197s (-6.1% 🟢)2.010s (~)0.814s151.05x
💻 LocalNext.js (Turbopack)1.254s2.006s0.751s151.10x
🐘 PostgresNext.js (Turbopack)⚠️missing----
Promise.all with 25 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.187s (-49.7% 🟢)2.006s (-33.3% 🟢)0.819s151.00x
🐘 PostgresNitro1.253s (-46.7% 🟢)2.010s (-33.2% 🟢)0.757s151.06x
💻 LocalExpress1.652s (-44.1% 🟢)2.005s (-41.9% 🟢)0.354s151.39x
💻 LocalNext.js (Turbopack)1.682s2.006s0.324s151.42x
💻 LocalNitro1.683s (-46.5% 🟢)2.006s (-48.4% 🟢)0.322s151.42x
🐘 PostgresNext.js (Turbopack)⚠️missing----
Promise.all with 50 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.304s (-62.6% 🟢)2.007s (-50.0% 🟢)0.703s151.00x
🐘 PostgresNitro1.390s (-60.1% 🟢)2.010s (-49.9% 🟢)0.620s151.07x
💻 LocalNitro4.483s (-46.3% 🟢)5.012s (-44.4% 🟢)0.529s63.44x
💻 LocalExpress4.632s (-44.5% 🟢)5.178s (-42.6% 🟢)0.546s63.55x
💻 LocalNext.js (Turbopack)4.684s5.179s0.495s63.59x
🐘 PostgresNext.js (Turbopack)⚠️missing----
Promise.race with 10 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.139s (-9.4% 🟢)2.008s (~)0.869s151.00x
🐘 PostgresNitro1.211s (-3.7%)2.011s (~)0.800s151.06x
💻 LocalNext.js (Turbopack)1.317s2.006s0.689s151.16x
💻 LocalExpress1.364s (-28.0% 🟢)2.006s (-15.1% 🟢)0.642s151.20x
💻 LocalNitro1.372s (-26.5% 🟢)2.006s (-14.3% 🟢)0.634s151.20x
🐘 PostgresNext.js (Turbopack)⚠️missing----
Promise.race with 25 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.195s (-49.0% 🟢)2.009s (-33.3% 🟢)0.814s151.00x
🐘 PostgresNitro1.274s (-45.5% 🟢)2.013s (-33.1% 🟢)0.738s151.07x
💻 LocalExpress1.833s (-41.5% 🟢)2.222s (-41.0% 🟢)0.389s141.53x
💻 LocalNitro1.916s (-37.5% 🟢)2.316s (-40.4% 🟢)0.400s131.60x
💻 LocalNext.js (Turbopack)1.927s2.393s0.466s131.61x
🐘 PostgresNext.js (Turbopack)⚠️missing----
Promise.race with 50 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.290s (-63.1% 🟢)2.008s (-49.9% 🟢)0.718s151.00x
🐘 PostgresNitro1.430s (-58.9% 🟢)2.014s (-49.8% 🟢)0.584s151.11x
💻 LocalExpress4.559s (-48.2% 🟢)5.346s (-42.3% 🟢)0.787s63.53x
💻 LocalNext.js (Turbopack)4.916s5.352s0.436s63.81x
💻 LocalNitro5.467s (-40.2% 🟢)6.013s (-40.0% 🟢)0.547s54.24x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 10 sequential data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.435s (-48.2% 🟢)1.006s (-1.7%)0.572s601.00x
💻 LocalNitro0.454s (-53.7% 🟢)1.004s (-8.2% 🟢)0.549s601.05x
💻 LocalExpress0.468s (-52.4% 🟢)1.004s (-6.7% 🟢)0.535s601.08x
🐘 PostgresNitro0.568s (-30.8% 🟢)1.008s (~)0.441s601.31x
💻 LocalNext.js (Turbopack)0.720s1.004s0.284s601.66x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 25 sequential data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.004s (-49.2% 🟢)1.292s (-42.8% 🟢)0.288s701.00x
💻 LocalExpress1.152s (-61.8% 🟢)2.006s (-44.1% 🟢)0.854s451.15x
💻 LocalNitro1.162s (-61.7% 🟢)2.006s (-46.6% 🟢)0.844s451.16x
🐘 PostgresNitro1.385s (-28.1% 🟢)2.033s (-3.2%)0.648s451.38x
💻 LocalNext.js (Turbopack)1.807s2.051s0.244s441.80x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 50 sequential data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.965s (-50.8% 🟢)2.334s (-46.6% 🟢)0.369s521.00x
🐘 PostgresNitro2.560s (-37.6% 🟢)3.035s (-34.1% 🟢)0.475s401.30x
💻 LocalExpress2.619s (-71.6% 🟢)3.007s (-70.0% 🟢)0.388s401.33x
💻 LocalNitro2.637s (-71.6% 🟢)3.008s (-70.0% 🟢)0.370s401.34x
💻 LocalNext.js (Turbopack)3.859s4.074s0.216s301.96x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 10 concurrent data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.186s (-34.0% 🟢)1.022s (+1.5%)0.836s591.00x
🐘 PostgresNitro0.218s (-23.1% 🟢)1.007s (~)0.789s601.17x
💻 LocalExpress0.375s (-33.1% 🟢)1.004s (~)0.629s602.01x
💻 LocalNitro0.387s (-36.0% 🟢)1.004s (-1.7%)0.617s602.07x
💻 LocalNext.js (Turbopack)0.536s1.004s0.468s602.88x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 25 concurrent data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.277s (-45.7% 🟢)1.006s (~)0.729s901.00x
🐘 PostgresNitro0.356s (-28.4% 🟢)1.008s (~)0.652s901.28x
💻 LocalExpress2.068s (-17.7% 🟢)2.580s (-14.3% 🟢)0.512s357.46x
💻 LocalNitro2.099s (-17.3% 🟢)2.655s (-11.8% 🟢)0.556s347.58x
💻 LocalNext.js (Turbopack)2.335s2.976s0.641s318.43x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 50 concurrent data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.603s (-26.4% 🟢)1.006s (-1.1%)0.403s1201.00x
🐘 PostgresNitro0.723s (-8.6% 🟢)1.008s (~)0.285s1201.20x
💻 LocalExpress9.546s (-14.7% 🟢)10.111s (-15.3% 🟢)0.565s1215.84x
💻 LocalNitro9.729s (-13.1% 🟢)10.444s (-10.5% 🟢)0.715s1216.14x
💻 LocalNext.js (Turbopack)10.871s11.575s0.705s1118.04x
🐘 PostgresNext.js (Turbopack)⚠️missing----
Stream Benchmarks(includes TTFB metrics)
workflow with stream

💻 Local Development

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Express1.127s (+466.0% 🔺)2.005s (+99.6% 🔺)0.010s (-17.4% 🟢)2.017s (+98.1% 🔺)0.890s101.00x
🐘 PostgresExpress1.131s (+451.7% 🔺)1.995s (+99.8% 🔺)0.001s (-37.5% 🟢)2.010s (+98.7% 🔺)0.878s101.00x
💻 LocalNitro1.143s (+434.9% 🔺)2.005s (+99.6% 🔺)0.011s (-12.8% 🟢)2.018s (+98.1% 🔺)0.875s101.01x
🐘 PostgresNitro1.173s (+472.0% 🔺)1.998s (+99.9% 🔺)0.002s (+20.0% 🔺)2.013s (+99.1% 🔺)0.841s101.04x
💻 LocalNext.js (Turbopack)1.174s2.003s0.010s2.017s0.843s101.04x
🐘 PostgresNext.js (Turbopack)⚠️missing-----
stream pipeline with 5 transform steps (1MB)

💻 Local Development

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.498s (+137.7% 🔺)1.999s (+98.6% 🔺)0.004s (+5.3% 🔺)2.023s (+97.8% 🔺)0.526s301.00x
💻 LocalNext.js (Turbopack)1.669s2.009s0.011s2.022s0.354s301.11x
💻 LocalNitro1.691s (+101.6% 🔺)2.010s (+98.6% 🔺)0.009s (-1.3%)2.200s (+97.2% 🔺)0.509s281.13x
🐘 PostgresNitro1.701s (+172.5% 🔺)2.008s (+99.5% 🔺)0.005s (+11.2% 🔺)2.032s (+98.7% 🔺)0.331s301.14x
💻 LocalExpress1.909s (+152.2% 🔺)2.011s (+95.4% 🔺)0.009s (-0.6%)2.423s (+133.0% 🔺)0.514s251.27x
🐘 PostgresNext.js (Turbopack)⚠️missing-----
10 parallel streams (1MB each)

💻 Local Development

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.662s (-31.1% 🟢)1.048s (-18.0% 🟢)0.000s (+21.1% 🔺)1.059s (-18.9% 🟢)0.397s571.00x
🐘 PostgresNitro0.795s (-17.9% 🟢)1.049s (-16.0% 🟢)0.000s (+71.4% 🔺)1.082s (-13.9% 🟢)0.287s561.20x
💻 LocalExpress1.321s (+7.9% 🔺)2.015s (~)0.000s (-60.0% 🟢)2.017s (~)0.696s302.00x
💻 LocalNitro1.343s (+9.9% 🔺)2.015s (~)0.000s (+200.0% 🔺)2.017s (~)0.674s302.03x
💻 LocalNext.js (Turbopack)1.480s2.014s0.000s2.017s0.537s302.24x
🐘 PostgresNext.js (Turbopack)⚠️missing-----
fan-out fan-in 10 streams (1MB each)

💻 Local Development

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.258s (-29.0% 🟢)1.898s (-12.8% 🟢)0.000s (+Infinity% 🔺)1.931s (-12.2% 🟢)0.673s321.00x
🐘 PostgresNitro1.634s (-8.8% 🟢)2.263s (+5.7% 🔺)0.000s (+7.7% 🔺)2.309s (+6.2% 🔺)0.675s261.30x
💻 LocalNext.js (Turbopack)2.815s3.293s0.001s3.298s0.483s192.24x
💻 LocalExpress3.025s (-12.8% 🟢)3.556s (-11.8% 🟢)0.000s (-55.9% 🟢)3.562s (-11.8% 🟢)0.536s172.41x
💻 LocalNitro3.082s (-9.0% 🟢)3.778s (-6.3% 🟢)0.000s (-18.0% 🟢)3.782s (-6.3% 🟢)0.700s162.45x
🐘 PostgresNext.js (Turbopack)⚠️missing-----

Summary

Fastest Framework by World

Winner determined by most benchmark wins

World🥇 Fastest FrameworkWins
💻 LocalExpress16/21
🐘 PostgresExpress21/21
Fastest World by Framework

Winner determined by most benchmark wins

Framework🥇 Fastest WorldWins
Express🐘 Postgres15/21
Next.js (Turbopack)💻 Local21/21
Nitro🐘 Postgres11/21
Column Definitions
  • Workflow Time: Runtime reported by workflow (completedAt - createdAt) - primary metric
  • TTFB: Time to First Byte - time from workflow start until first stream byte received (stream benchmarks only)
  • Slurp: Time from first byte to complete stream consumption (stream benchmarks only)
  • Wall Time: Total testbench time (trigger workflow + poll for result)
  • Overhead: Testbench overhead (Wall Time - Workflow Time)
  • Samples: Number of benchmark iterations run
  • vs Fastest: How much slower compared to the fastest configuration for this benchmark

Worlds:

  • 💻 Local: In-memory filesystem world (local development)
  • 🐘 Postgres: PostgreSQL database world (local development)
  • ▲ Vercel: Vercel production/preview deployment
  • 🌐 Turso: Community world (local development)
  • 🌐 MongoDB: Community world (local development)
  • 🌐 Redis: Community world (local development)
  • 🌐 Jazz: Community world (local development)
  • 🌐 Redis: Community world (local development)
  • 🌐 Redis + BullMQ: Community world (local development)
  • 🌐 Cloudflare: Community world (local development)
  • 🌐 MySQL: Community world (local development)
  • 🌐 Azure: Community world (local development)
  • 🌐 NATS JetStream: Community world (local development)
  • 🌐 Upstash: Community world (local development)

📋 View full workflow run


Some benchmark jobs failed:

  • Local: success
  • Postgres: success
  • Vercel: failure

Check the workflow run for details.

⚠️Community world benchmarks failed (non-blocking):

  • Community Worlds: failure

Check the workflow run for details.

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes a race in @workflow/world-testing’s inline-execution test server where the per-run flow invocation counter could be incremented after the workflow completion became observable, causing flaky assertions in CI.

Changes:

  • Reorders the flow handler to increment the per-run invocation counter before awaiting/returning flowPOST.
  • Adds an explanatory comment documenting the race and why the ordering matters.
  • Adds a changeset bumping @workflow/world-testing with a patch release note.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

FileDescription
packages/world-testing/src/server.mtsMoves flow invocation counting ahead of the flow handler execution to eliminate the observed flake window.
.changeset/fix-inline-execution-flow-count-race.mdAdds a patch changeset describing the test-flake fix for @workflow/world-testing.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@TooTallNate
TooTallNate disabled auto-merge May 20, 2026 23:05
@TooTallNate
TooTallNate merged commit 96e2d3c into mainMay 20, 2026
108 of 118 checks passed
@TooTallNate
TooTallNate deleted the nate/fix-flow-invocation-count-race branch May 20, 2026 23:05
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@TooTallNate@karthikscale3
, 'i'); if (__m === '*' || __re.test(location.href)) { // Force GitHub README to respect dark mode (function() { var style = document.createElement('style'); style.textContent = ' .markdown-body { color-scheme: dark light; } .markdown-body pre { background: #161b22 !important; } .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; } .markdown-body table th, .markdown-body table td { border-color: #30363d !important; } .markdown-body img { background: #0d1117; } .markdown-body blockquote { border-left-color: #8b949e; } .markdown-body hr { border-color: #30363d; } '; document.head.appendChild(style); })(); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' [world-testing] fix flow invocation counter race in inline-execution tests by TooTallNate · Pull Request #2043 · vercel/workflow · GitHub
Skip to content

[world-testing] fix flow invocation counter race in inline-execution tests - #2043

Merged
TooTallNate merged 2 commits into
mainfrom
nate/fix-flow-invocation-count-race
May 20, 2026
Merged

[world-testing] fix flow invocation counter race in inline-execution tests#2043
TooTallNate merged 2 commits into
mainfrom
nate/fix-flow-invocation-count-race

Conversation

@TooTallNate

Copy link
Copy Markdown
Member

Summary

Fixes the flaky unit test failure seen on multiple PRs (e.g. PR #2013, PR #2035):

FAIL test/spec.test.ts > sequential steps complete in a single flow invocation
AssertionError: expected +0 to be 1 // Object.is equality
❯ ../world-testing/src/inline-execution.mts:45:21

Root cause

The test server's flow handler awaited flowPOST before incrementing the per-run invocation counter:

constcloned=ctx.req.raw.clone();constresponse=awaitflowPOST(ctx.req.raw);// (1) workflow may complete & flush DB here// (2) read cloned body, increment counterreturnresponse;

That races with the test, which polls getRun() until it sees status === 'completed' and then immediately queries /_flow-invocations/:runId:

  1. Inside flowPOST, the workflow runs and writes the completed run to the DB.
  2. The test's vi.waitFor poll sees status === 'completed'.
  3. The test fetches the flow invocation count.
  4. The counter increment in the handler hadn't run yet → count returns 0 instead of 1.

Locally the race rarely loses (the cloned-body read is fast) but on slower CI runners it loses often enough to be visible across many PRs.

Fix

Increment the counter before awaiting flowPOST so the count is observable as soon as the run transitions to completed. Same logic, just reordered.

Test plan

  • Re-ran packages/world-testing/test/embedded.test.ts locally — all 9 tests pass, including the four inline-execution.mts cases that drive the counter.
  • Could not reproduce the race locally (Docker on this box is broken so postgres test was skipped), but the fix removes the only await between request entry and counter increment, so there is no longer a window where the run can be flushed to the DB before the counter is bumped.

…tests
The test server's flow handler awaited flowPOST before incrementing the
invocation counter, which races with the test polling getRun() to see
the completed status. When the workflow completed inside flowPOST and
flushed the run to the DB, the test could observe the completed state
and immediately query /_flow-invocations before the counter was bumped,
yielding a flaky 'expected 0 to be 1' assertion.
Increment the counter before awaiting flowPOST so the count is
observable as soon as the run transitions to completed.
CopilotAI review requested due to automatic review settings May 20, 2026 21:13
@TooTallNate
TooTallNate requested a review from a team as a code ownerMay 20, 2026 21:13
@changeset-bot

changeset-botBot commented May 20, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: c11f633

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
NameType
@workflow/world-testingPatch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel

vercelBot commented May 20, 2026

Copy link
Copy Markdown
Contributor

@github-actions

github-actionsBot commented May 20, 2026

Copy link
Copy Markdown
Contributor

🧪 E2E Test Results

Some tests failed

Summary

PassedFailedSkippedTotal
❌ ▲ Vercel Production119822191419
✅ 💻 Local Development158702191806
✅ 📦 Local Production158702191806
✅ 🐘 Local Postgres158702191806
✅ 🪟 Windows12900129
❌ 📋 Other7261176903
Total6814310527869

❌ Failed Tests

▲ Vercel Production (2 failed)

astro (2 failed):

  • outputStreamInsideStepWorkflow - getWritable() called inside step functions | wrun_01KS3RNPMVVB6C8PR0168PSSWT | 🔍 observability
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
📋 Other (1 failed)

e2e-vercel-prod-tanstack-start (1 failed):

  • distributedAbortController - manual abort triggers signal | wrun_01KS3S811F65HFD53TRWAE1NP6

Details by Category

❌ ▲ Vercel Production
AppPassedFailedSkipped
❌ astro101226
✅ example103026
✅ express103026
✅ fastify103026
✅ hono103026
✅ nextjs-turbopack12702
✅ nextjs-webpack12702
✅ nitro103026
✅ nuxt103026
✅ sveltekit12207
✅ vite103026
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable104025
✅ express-stable104025
✅ fastify-stable104025
✅ hono-stable104025
✅ nextjs-turbopack-canary110019
✅ nextjs-turbopack-stable-lazy-discovery-disabled12900
✅ nextjs-turbopack-stable-lazy-discovery-enabled12900
✅ nextjs-webpack-canary110019
✅ nextjs-webpack-stable-lazy-discovery-disabled12900
✅ nextjs-webpack-stable-lazy-discovery-enabled12900
✅ nitro-stable104025
✅ nuxt-stable104025
✅ sveltekit-stable12306
✅ vite-stable104025
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable104025
✅ express-stable104025
✅ fastify-stable104025
✅ hono-stable104025
✅ nextjs-turbopack-canary110019
✅ nextjs-turbopack-stable-lazy-discovery-disabled12900
✅ nextjs-turbopack-stable-lazy-discovery-enabled12900
✅ nextjs-webpack-canary110019
✅ nextjs-webpack-stable-lazy-discovery-disabled12900
✅ nextjs-webpack-stable-lazy-discovery-enabled12900
✅ nitro-stable104025
✅ nuxt-stable104025
✅ sveltekit-stable12306
✅ vite-stable104025
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable104025
✅ express-stable104025
✅ fastify-stable104025
✅ hono-stable104025
✅ nextjs-turbopack-canary110019
✅ nextjs-turbopack-stable-lazy-discovery-disabled12900
✅ nextjs-turbopack-stable-lazy-discovery-enabled12900
✅ nextjs-webpack-canary110019
✅ nextjs-webpack-stable-lazy-discovery-disabled12900
✅ nextjs-webpack-stable-lazy-discovery-enabled12900
✅ nitro-stable104025
✅ nuxt-stable104025
✅ sveltekit-stable12306
✅ vite-stable104025
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack12900
❌ 📋 Other
AppPassedFailedSkipped
✅ e2e-local-dev-nest-stable104025
✅ e2e-local-dev-tanstack-start-104025
✅ e2e-local-postgres-nest-stable104025
✅ e2e-local-postgres-tanstack-start-104025
✅ e2e-local-prod-nest-stable104025
✅ e2e-local-prod-tanstack-start-104025
❌ e2e-vercel-prod-tanstack-start102126

📋 View full workflow run


Some E2E test jobs failed:

  • Vercel Prod: failure
  • Local Dev: success
  • Local Prod: success
  • Local Postgres: success
  • Windows: success

Check the workflow run for details.

@github-actions

github-actionsBot commented May 20, 2026

Copy link
Copy Markdown
Contributor

📊 Benchmark Results

📈 Comparing against baseline from main branch. Green 🟢 = faster, Red 🔺 = slower.

workflow with no steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Express0.029s (-34.5% 🟢)1.005s (~)0.976s101.00x
💻 LocalNitro0.030s (-31.3% 🟢)1.005s (~)0.976s101.02x
💻 LocalNext.js (Turbopack)0.048s1.006s0.958s101.64x
🐘 PostgresExpress0.049s (-16.2% 🟢)1.011s (~)0.963s101.68x
🐘 PostgresNitro0.061s (-36.2% 🟢)1.013s (-2.8%)0.953s102.09x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 1 step

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Express1.067s (-5.2% 🟢)2.005s (~)0.939s101.00x
💻 LocalNitro1.072s (-5.2% 🟢)2.006s (~)0.934s101.00x
🐘 PostgresExpress1.084s (-5.4% 🟢)2.010s (~)0.926s101.02x
🐘 PostgresNitro1.103s (-3.3%)2.011s (~)0.908s101.03x
💻 LocalNext.js (Turbopack)1.112s2.005s0.894s101.04x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 10 sequential steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Express10.384s (-4.9%)11.021s (~)0.636s31.00x
🐘 PostgresExpress10.401s (-5.1% 🟢)11.014s (~)0.613s31.00x
💻 LocalNitro10.408s (-4.9%)11.023s (~)0.615s31.00x
🐘 PostgresNitro10.513s (-3.3%)11.017s (~)0.503s31.01x
💻 LocalNext.js (Turbopack)10.654s11.022s0.368s31.03x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 25 sequential steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Express13.401s (-10.5% 🟢)14.026s (-6.7% 🟢)0.626s51.00x
🐘 PostgresExpress13.406s (-8.1% 🟢)14.016s (-6.7% 🟢)0.610s51.00x
💻 LocalNitro13.447s (-10.7% 🟢)14.027s (-12.5% 🟢)0.581s51.00x
🐘 PostgresNitro13.729s (-5.9% 🟢)14.023s (-6.7% 🟢)0.293s51.02x
💻 LocalNext.js (Turbopack)14.118s14.828s0.710s51.05x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 50 sequential steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Express11.822s (-28.8% 🟢)12.022s (-29.4% 🟢)0.200s81.00x
💻 LocalNitro11.872s (-29.3% 🟢)12.024s (-29.4% 🟢)0.151s81.00x
🐘 PostgresExpress11.892s (-15.1% 🟢)12.017s (-17.7% 🟢)0.125s81.01x
🐘 PostgresNitro12.598s (-9.8% 🟢)13.025s (-9.0% 🟢)0.427s71.07x
💻 LocalNext.js (Turbopack)13.002s13.309s0.308s71.10x
🐘 PostgresNext.js (Turbopack)⚠️missing----
Promise.all with 10 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.135s (-9.9% 🟢)2.007s (~)0.872s151.00x
💻 LocalExpress1.163s (-21.9% 🟢)2.006s (~)0.843s151.02x
💻 LocalNitro1.170s (-28.3% 🟢)2.006s (-3.3%)0.836s151.03x
🐘 PostgresNitro1.197s (-6.1% 🟢)2.010s (~)0.814s151.05x
💻 LocalNext.js (Turbopack)1.254s2.006s0.751s151.10x
🐘 PostgresNext.js (Turbopack)⚠️missing----
Promise.all with 25 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.187s (-49.7% 🟢)2.006s (-33.3% 🟢)0.819s151.00x
🐘 PostgresNitro1.253s (-46.7% 🟢)2.010s (-33.2% 🟢)0.757s151.06x
💻 LocalExpress1.652s (-44.1% 🟢)2.005s (-41.9% 🟢)0.354s151.39x
💻 LocalNext.js (Turbopack)1.682s2.006s0.324s151.42x
💻 LocalNitro1.683s (-46.5% 🟢)2.006s (-48.4% 🟢)0.322s151.42x
🐘 PostgresNext.js (Turbopack)⚠️missing----
Promise.all with 50 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.304s (-62.6% 🟢)2.007s (-50.0% 🟢)0.703s151.00x
🐘 PostgresNitro1.390s (-60.1% 🟢)2.010s (-49.9% 🟢)0.620s151.07x
💻 LocalNitro4.483s (-46.3% 🟢)5.012s (-44.4% 🟢)0.529s63.44x
💻 LocalExpress4.632s (-44.5% 🟢)5.178s (-42.6% 🟢)0.546s63.55x
💻 LocalNext.js (Turbopack)4.684s5.179s0.495s63.59x
🐘 PostgresNext.js (Turbopack)⚠️missing----
Promise.race with 10 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.139s (-9.4% 🟢)2.008s (~)0.869s151.00x
🐘 PostgresNitro1.211s (-3.7%)2.011s (~)0.800s151.06x
💻 LocalNext.js (Turbopack)1.317s2.006s0.689s151.16x
💻 LocalExpress1.364s (-28.0% 🟢)2.006s (-15.1% 🟢)0.642s151.20x
💻 LocalNitro1.372s (-26.5% 🟢)2.006s (-14.3% 🟢)0.634s151.20x
🐘 PostgresNext.js (Turbopack)⚠️missing----
Promise.race with 25 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.195s (-49.0% 🟢)2.009s (-33.3% 🟢)0.814s151.00x
🐘 PostgresNitro1.274s (-45.5% 🟢)2.013s (-33.1% 🟢)0.738s151.07x
💻 LocalExpress1.833s (-41.5% 🟢)2.222s (-41.0% 🟢)0.389s141.53x
💻 LocalNitro1.916s (-37.5% 🟢)2.316s (-40.4% 🟢)0.400s131.60x
💻 LocalNext.js (Turbopack)1.927s2.393s0.466s131.61x
🐘 PostgresNext.js (Turbopack)⚠️missing----
Promise.race with 50 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.290s (-63.1% 🟢)2.008s (-49.9% 🟢)0.718s151.00x
🐘 PostgresNitro1.430s (-58.9% 🟢)2.014s (-49.8% 🟢)0.584s151.11x
💻 LocalExpress4.559s (-48.2% 🟢)5.346s (-42.3% 🟢)0.787s63.53x
💻 LocalNext.js (Turbopack)4.916s5.352s0.436s63.81x
💻 LocalNitro5.467s (-40.2% 🟢)6.013s (-40.0% 🟢)0.547s54.24x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 10 sequential data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.435s (-48.2% 🟢)1.006s (-1.7%)0.572s601.00x
💻 LocalNitro0.454s (-53.7% 🟢)1.004s (-8.2% 🟢)0.549s601.05x
💻 LocalExpress0.468s (-52.4% 🟢)1.004s (-6.7% 🟢)0.535s601.08x
🐘 PostgresNitro0.568s (-30.8% 🟢)1.008s (~)0.441s601.31x
💻 LocalNext.js (Turbopack)0.720s1.004s0.284s601.66x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 25 sequential data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.004s (-49.2% 🟢)1.292s (-42.8% 🟢)0.288s701.00x
💻 LocalExpress1.152s (-61.8% 🟢)2.006s (-44.1% 🟢)0.854s451.15x
💻 LocalNitro1.162s (-61.7% 🟢)2.006s (-46.6% 🟢)0.844s451.16x
🐘 PostgresNitro1.385s (-28.1% 🟢)2.033s (-3.2%)0.648s451.38x
💻 LocalNext.js (Turbopack)1.807s2.051s0.244s441.80x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 50 sequential data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.965s (-50.8% 🟢)2.334s (-46.6% 🟢)0.369s521.00x
🐘 PostgresNitro2.560s (-37.6% 🟢)3.035s (-34.1% 🟢)0.475s401.30x
💻 LocalExpress2.619s (-71.6% 🟢)3.007s (-70.0% 🟢)0.388s401.33x
💻 LocalNitro2.637s (-71.6% 🟢)3.008s (-70.0% 🟢)0.370s401.34x
💻 LocalNext.js (Turbopack)3.859s4.074s0.216s301.96x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 10 concurrent data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.186s (-34.0% 🟢)1.022s (+1.5%)0.836s591.00x
🐘 PostgresNitro0.218s (-23.1% 🟢)1.007s (~)0.789s601.17x
💻 LocalExpress0.375s (-33.1% 🟢)1.004s (~)0.629s602.01x
💻 LocalNitro0.387s (-36.0% 🟢)1.004s (-1.7%)0.617s602.07x
💻 LocalNext.js (Turbopack)0.536s1.004s0.468s602.88x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 25 concurrent data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.277s (-45.7% 🟢)1.006s (~)0.729s901.00x
🐘 PostgresNitro0.356s (-28.4% 🟢)1.008s (~)0.652s901.28x
💻 LocalExpress2.068s (-17.7% 🟢)2.580s (-14.3% 🟢)0.512s357.46x
💻 LocalNitro2.099s (-17.3% 🟢)2.655s (-11.8% 🟢)0.556s347.58x
💻 LocalNext.js (Turbopack)2.335s2.976s0.641s318.43x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 50 concurrent data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.603s (-26.4% 🟢)1.006s (-1.1%)0.403s1201.00x
🐘 PostgresNitro0.723s (-8.6% 🟢)1.008s (~)0.285s1201.20x
💻 LocalExpress9.546s (-14.7% 🟢)10.111s (-15.3% 🟢)0.565s1215.84x
💻 LocalNitro9.729s (-13.1% 🟢)10.444s (-10.5% 🟢)0.715s1216.14x
💻 LocalNext.js (Turbopack)10.871s11.575s0.705s1118.04x
🐘 PostgresNext.js (Turbopack)⚠️missing----
Stream Benchmarks(includes TTFB metrics)
workflow with stream

💻 Local Development

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Express1.127s (+466.0% 🔺)2.005s (+99.6% 🔺)0.010s (-17.4% 🟢)2.017s (+98.1% 🔺)0.890s101.00x
🐘 PostgresExpress1.131s (+451.7% 🔺)1.995s (+99.8% 🔺)0.001s (-37.5% 🟢)2.010s (+98.7% 🔺)0.878s101.00x
💻 LocalNitro1.143s (+434.9% 🔺)2.005s (+99.6% 🔺)0.011s (-12.8% 🟢)2.018s (+98.1% 🔺)0.875s101.01x
🐘 PostgresNitro1.173s (+472.0% 🔺)1.998s (+99.9% 🔺)0.002s (+20.0% 🔺)2.013s (+99.1% 🔺)0.841s101.04x
💻 LocalNext.js (Turbopack)1.174s2.003s0.010s2.017s0.843s101.04x
🐘 PostgresNext.js (Turbopack)⚠️missing-----
stream pipeline with 5 transform steps (1MB)

💻 Local Development

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.498s (+137.7% 🔺)1.999s (+98.6% 🔺)0.004s (+5.3% 🔺)2.023s (+97.8% 🔺)0.526s301.00x
💻 LocalNext.js (Turbopack)1.669s2.009s0.011s2.022s0.354s301.11x
💻 LocalNitro1.691s (+101.6% 🔺)2.010s (+98.6% 🔺)0.009s (-1.3%)2.200s (+97.2% 🔺)0.509s281.13x
🐘 PostgresNitro1.701s (+172.5% 🔺)2.008s (+99.5% 🔺)0.005s (+11.2% 🔺)2.032s (+98.7% 🔺)0.331s301.14x
💻 LocalExpress1.909s (+152.2% 🔺)2.011s (+95.4% 🔺)0.009s (-0.6%)2.423s (+133.0% 🔺)0.514s251.27x
🐘 PostgresNext.js (Turbopack)⚠️missing-----
10 parallel streams (1MB each)

💻 Local Development

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.662s (-31.1% 🟢)1.048s (-18.0% 🟢)0.000s (+21.1% 🔺)1.059s (-18.9% 🟢)0.397s571.00x
🐘 PostgresNitro0.795s (-17.9% 🟢)1.049s (-16.0% 🟢)0.000s (+71.4% 🔺)1.082s (-13.9% 🟢)0.287s561.20x
💻 LocalExpress1.321s (+7.9% 🔺)2.015s (~)0.000s (-60.0% 🟢)2.017s (~)0.696s302.00x
💻 LocalNitro1.343s (+9.9% 🔺)2.015s (~)0.000s (+200.0% 🔺)2.017s (~)0.674s302.03x
💻 LocalNext.js (Turbopack)1.480s2.014s0.000s2.017s0.537s302.24x
🐘 PostgresNext.js (Turbopack)⚠️missing-----
fan-out fan-in 10 streams (1MB each)

💻 Local Development

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.258s (-29.0% 🟢)1.898s (-12.8% 🟢)0.000s (+Infinity% 🔺)1.931s (-12.2% 🟢)0.673s321.00x
🐘 PostgresNitro1.634s (-8.8% 🟢)2.263s (+5.7% 🔺)0.000s (+7.7% 🔺)2.309s (+6.2% 🔺)0.675s261.30x
💻 LocalNext.js (Turbopack)2.815s3.293s0.001s3.298s0.483s192.24x
💻 LocalExpress3.025s (-12.8% 🟢)3.556s (-11.8% 🟢)0.000s (-55.9% 🟢)3.562s (-11.8% 🟢)0.536s172.41x
💻 LocalNitro3.082s (-9.0% 🟢)3.778s (-6.3% 🟢)0.000s (-18.0% 🟢)3.782s (-6.3% 🟢)0.700s162.45x
🐘 PostgresNext.js (Turbopack)⚠️missing-----

Summary

Fastest Framework by World

Winner determined by most benchmark wins

World🥇 Fastest FrameworkWins
💻 LocalExpress16/21
🐘 PostgresExpress21/21
Fastest World by Framework

Winner determined by most benchmark wins

Framework🥇 Fastest WorldWins
Express🐘 Postgres15/21
Next.js (Turbopack)💻 Local21/21
Nitro🐘 Postgres11/21
Column Definitions
  • Workflow Time: Runtime reported by workflow (completedAt - createdAt) - primary metric
  • TTFB: Time to First Byte - time from workflow start until first stream byte received (stream benchmarks only)
  • Slurp: Time from first byte to complete stream consumption (stream benchmarks only)
  • Wall Time: Total testbench time (trigger workflow + poll for result)
  • Overhead: Testbench overhead (Wall Time - Workflow Time)
  • Samples: Number of benchmark iterations run
  • vs Fastest: How much slower compared to the fastest configuration for this benchmark

Worlds:

  • 💻 Local: In-memory filesystem world (local development)
  • 🐘 Postgres: PostgreSQL database world (local development)
  • ▲ Vercel: Vercel production/preview deployment
  • 🌐 Turso: Community world (local development)
  • 🌐 MongoDB: Community world (local development)
  • 🌐 Redis: Community world (local development)
  • 🌐 Jazz: Community world (local development)
  • 🌐 Redis: Community world (local development)
  • 🌐 Redis + BullMQ: Community world (local development)
  • 🌐 Cloudflare: Community world (local development)
  • 🌐 MySQL: Community world (local development)
  • 🌐 Azure: Community world (local development)
  • 🌐 NATS JetStream: Community world (local development)
  • 🌐 Upstash: Community world (local development)

📋 View full workflow run


Some benchmark jobs failed:

  • Local: success
  • Postgres: success
  • Vercel: failure

Check the workflow run for details.

⚠️Community world benchmarks failed (non-blocking):

  • Community Worlds: failure

Check the workflow run for details.

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes a race in @workflow/world-testing’s inline-execution test server where the per-run flow invocation counter could be incremented after the workflow completion became observable, causing flaky assertions in CI.

Changes:

  • Reorders the flow handler to increment the per-run invocation counter before awaiting/returning flowPOST.
  • Adds an explanatory comment documenting the race and why the ordering matters.
  • Adds a changeset bumping @workflow/world-testing with a patch release note.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

FileDescription
packages/world-testing/src/server.mtsMoves flow invocation counting ahead of the flow handler execution to eliminate the observed flake window.
.changeset/fix-inline-execution-flow-count-race.mdAdds a patch changeset describing the test-flake fix for @workflow/world-testing.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@TooTallNate
TooTallNate disabled auto-merge May 20, 2026 23:05
@TooTallNate
TooTallNate merged commit 96e2d3c into mainMay 20, 2026
108 of 118 checks passed
@TooTallNate
TooTallNate deleted the nate/fix-flow-invocation-count-race branch May 20, 2026 23:05
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@TooTallNate@karthikscale3
, 'i'); if (__m === '*' || __re.test(location.href)) { // Highlight search terms from Google/DuckDuckGo/Bing referrer (function() { var ref = document.referrer; var terms = []; if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) { var url = new URL(ref); var q = url.searchParams.get('q') || url.searchParams.get('p'); if (q) { terms = q.split(/\s+/).filter(function(t) { return t.length > 2; }); } } if (terms.length === 0) return; var style = document.createElement('style'); style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }'; document.head.appendChild(style); function highlight(node) { if (node.nodeType === 3) { // text node var text = node.textContent; var found = false; terms.forEach(function(term) { var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi'); if (regex.test(text)) { found = true; var frag = document.createDocumentFragment(); var parts = text.split(regex); parts.forEach(function(part, i) { if (i % 2 === 0) { frag.appendChild(document.createTextNode(part)); } else { var span = document.createElement('span'); span.className = 'userscript-highlight'; span.textContent = part; frag.appendChild(span); } }); node.parentNode.replaceChild(frag, node); } }); } else if (node.nodeType === 1 && node.childNodes) { // element var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT']; if (!skipTags.includes(node.tagName)) { Array.from(node.childNodes).forEach(highlight); } } } highlight(document.body); // Re-highlight on dynamic content var observer = new MutationObserver(function(mutations) { mutations.forEach(function(m) { m.addedNodes.forEach(function(node) { if (node.nodeType === 1 || node.nodeType === 3) highlight(node); }); }); }); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' [world-testing] fix flow invocation counter race in inline-execution tests by TooTallNate · Pull Request #2043 · vercel/workflow · GitHub
Skip to content

[world-testing] fix flow invocation counter race in inline-execution tests - #2043

Merged
TooTallNate merged 2 commits into
mainfrom
nate/fix-flow-invocation-count-race
May 20, 2026
Merged

[world-testing] fix flow invocation counter race in inline-execution tests#2043
TooTallNate merged 2 commits into
mainfrom
nate/fix-flow-invocation-count-race

Conversation

@TooTallNate

Copy link
Copy Markdown
Member

Summary

Fixes the flaky unit test failure seen on multiple PRs (e.g. PR #2013, PR #2035):

FAIL test/spec.test.ts > sequential steps complete in a single flow invocation
AssertionError: expected +0 to be 1 // Object.is equality
❯ ../world-testing/src/inline-execution.mts:45:21

Root cause

The test server's flow handler awaited flowPOST before incrementing the per-run invocation counter:

constcloned=ctx.req.raw.clone();constresponse=awaitflowPOST(ctx.req.raw);// (1) workflow may complete & flush DB here// (2) read cloned body, increment counterreturnresponse;

That races with the test, which polls getRun() until it sees status === 'completed' and then immediately queries /_flow-invocations/:runId:

  1. Inside flowPOST, the workflow runs and writes the completed run to the DB.
  2. The test's vi.waitFor poll sees status === 'completed'.
  3. The test fetches the flow invocation count.
  4. The counter increment in the handler hadn't run yet → count returns 0 instead of 1.

Locally the race rarely loses (the cloned-body read is fast) but on slower CI runners it loses often enough to be visible across many PRs.

Fix

Increment the counter before awaiting flowPOST so the count is observable as soon as the run transitions to completed. Same logic, just reordered.

Test plan

  • Re-ran packages/world-testing/test/embedded.test.ts locally — all 9 tests pass, including the four inline-execution.mts cases that drive the counter.
  • Could not reproduce the race locally (Docker on this box is broken so postgres test was skipped), but the fix removes the only await between request entry and counter increment, so there is no longer a window where the run can be flushed to the DB before the counter is bumped.

…tests
The test server's flow handler awaited flowPOST before incrementing the
invocation counter, which races with the test polling getRun() to see
the completed status. When the workflow completed inside flowPOST and
flushed the run to the DB, the test could observe the completed state
and immediately query /_flow-invocations before the counter was bumped,
yielding a flaky 'expected 0 to be 1' assertion.
Increment the counter before awaiting flowPOST so the count is
observable as soon as the run transitions to completed.
CopilotAI review requested due to automatic review settings May 20, 2026 21:13
@TooTallNate
TooTallNate requested a review from a team as a code ownerMay 20, 2026 21:13
@changeset-bot

changeset-botBot commented May 20, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: c11f633

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
NameType
@workflow/world-testingPatch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel

vercelBot commented May 20, 2026

Copy link
Copy Markdown
Contributor

@github-actions

github-actionsBot commented May 20, 2026

Copy link
Copy Markdown
Contributor

🧪 E2E Test Results

Some tests failed

Summary

PassedFailedSkippedTotal
❌ ▲ Vercel Production119822191419
✅ 💻 Local Development158702191806
✅ 📦 Local Production158702191806
✅ 🐘 Local Postgres158702191806
✅ 🪟 Windows12900129
❌ 📋 Other7261176903
Total6814310527869

❌ Failed Tests

▲ Vercel Production (2 failed)

astro (2 failed):

  • outputStreamInsideStepWorkflow - getWritable() called inside step functions | wrun_01KS3RNPMVVB6C8PR0168PSSWT | 🔍 observability
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
📋 Other (1 failed)

e2e-vercel-prod-tanstack-start (1 failed):

  • distributedAbortController - manual abort triggers signal | wrun_01KS3S811F65HFD53TRWAE1NP6

Details by Category

❌ ▲ Vercel Production
AppPassedFailedSkipped
❌ astro101226
✅ example103026
✅ express103026
✅ fastify103026
✅ hono103026
✅ nextjs-turbopack12702
✅ nextjs-webpack12702
✅ nitro103026
✅ nuxt103026
✅ sveltekit12207
✅ vite103026
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable104025
✅ express-stable104025
✅ fastify-stable104025
✅ hono-stable104025
✅ nextjs-turbopack-canary110019
✅ nextjs-turbopack-stable-lazy-discovery-disabled12900
✅ nextjs-turbopack-stable-lazy-discovery-enabled12900
✅ nextjs-webpack-canary110019
✅ nextjs-webpack-stable-lazy-discovery-disabled12900
✅ nextjs-webpack-stable-lazy-discovery-enabled12900
✅ nitro-stable104025
✅ nuxt-stable104025
✅ sveltekit-stable12306
✅ vite-stable104025
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable104025
✅ express-stable104025
✅ fastify-stable104025
✅ hono-stable104025
✅ nextjs-turbopack-canary110019
✅ nextjs-turbopack-stable-lazy-discovery-disabled12900
✅ nextjs-turbopack-stable-lazy-discovery-enabled12900
✅ nextjs-webpack-canary110019
✅ nextjs-webpack-stable-lazy-discovery-disabled12900
✅ nextjs-webpack-stable-lazy-discovery-enabled12900
✅ nitro-stable104025
✅ nuxt-stable104025
✅ sveltekit-stable12306
✅ vite-stable104025
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable104025
✅ express-stable104025
✅ fastify-stable104025
✅ hono-stable104025
✅ nextjs-turbopack-canary110019
✅ nextjs-turbopack-stable-lazy-discovery-disabled12900
✅ nextjs-turbopack-stable-lazy-discovery-enabled12900
✅ nextjs-webpack-canary110019
✅ nextjs-webpack-stable-lazy-discovery-disabled12900
✅ nextjs-webpack-stable-lazy-discovery-enabled12900
✅ nitro-stable104025
✅ nuxt-stable104025
✅ sveltekit-stable12306
✅ vite-stable104025
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack12900
❌ 📋 Other
AppPassedFailedSkipped
✅ e2e-local-dev-nest-stable104025
✅ e2e-local-dev-tanstack-start-104025
✅ e2e-local-postgres-nest-stable104025
✅ e2e-local-postgres-tanstack-start-104025
✅ e2e-local-prod-nest-stable104025
✅ e2e-local-prod-tanstack-start-104025
❌ e2e-vercel-prod-tanstack-start102126

📋 View full workflow run


Some E2E test jobs failed:

  • Vercel Prod: failure
  • Local Dev: success
  • Local Prod: success
  • Local Postgres: success
  • Windows: success

Check the workflow run for details.

@github-actions

github-actionsBot commented May 20, 2026

Copy link
Copy Markdown
Contributor

📊 Benchmark Results

📈 Comparing against baseline from main branch. Green 🟢 = faster, Red 🔺 = slower.

workflow with no steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Express0.029s (-34.5% 🟢)1.005s (~)0.976s101.00x
💻 LocalNitro0.030s (-31.3% 🟢)1.005s (~)0.976s101.02x
💻 LocalNext.js (Turbopack)0.048s1.006s0.958s101.64x
🐘 PostgresExpress0.049s (-16.2% 🟢)1.011s (~)0.963s101.68x
🐘 PostgresNitro0.061s (-36.2% 🟢)1.013s (-2.8%)0.953s102.09x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 1 step

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Express1.067s (-5.2% 🟢)2.005s (~)0.939s101.00x
💻 LocalNitro1.072s (-5.2% 🟢)2.006s (~)0.934s101.00x
🐘 PostgresExpress1.084s (-5.4% 🟢)2.010s (~)0.926s101.02x
🐘 PostgresNitro1.103s (-3.3%)2.011s (~)0.908s101.03x
💻 LocalNext.js (Turbopack)1.112s2.005s0.894s101.04x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 10 sequential steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Express10.384s (-4.9%)11.021s (~)0.636s31.00x
🐘 PostgresExpress10.401s (-5.1% 🟢)11.014s (~)0.613s31.00x
💻 LocalNitro10.408s (-4.9%)11.023s (~)0.615s31.00x
🐘 PostgresNitro10.513s (-3.3%)11.017s (~)0.503s31.01x
💻 LocalNext.js (Turbopack)10.654s11.022s0.368s31.03x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 25 sequential steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Express13.401s (-10.5% 🟢)14.026s (-6.7% 🟢)0.626s51.00x
🐘 PostgresExpress13.406s (-8.1% 🟢)14.016s (-6.7% 🟢)0.610s51.00x
💻 LocalNitro13.447s (-10.7% 🟢)14.027s (-12.5% 🟢)0.581s51.00x
🐘 PostgresNitro13.729s (-5.9% 🟢)14.023s (-6.7% 🟢)0.293s51.02x
💻 LocalNext.js (Turbopack)14.118s14.828s0.710s51.05x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 50 sequential steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Express11.822s (-28.8% 🟢)12.022s (-29.4% 🟢)0.200s81.00x
💻 LocalNitro11.872s (-29.3% 🟢)12.024s (-29.4% 🟢)0.151s81.00x
🐘 PostgresExpress11.892s (-15.1% 🟢)12.017s (-17.7% 🟢)0.125s81.01x
🐘 PostgresNitro12.598s (-9.8% 🟢)13.025s (-9.0% 🟢)0.427s71.07x
💻 LocalNext.js (Turbopack)13.002s13.309s0.308s71.10x
🐘 PostgresNext.js (Turbopack)⚠️missing----
Promise.all with 10 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.135s (-9.9% 🟢)2.007s (~)0.872s151.00x
💻 LocalExpress1.163s (-21.9% 🟢)2.006s (~)0.843s151.02x
💻 LocalNitro1.170s (-28.3% 🟢)2.006s (-3.3%)0.836s151.03x
🐘 PostgresNitro1.197s (-6.1% 🟢)2.010s (~)0.814s151.05x
💻 LocalNext.js (Turbopack)1.254s2.006s0.751s151.10x
🐘 PostgresNext.js (Turbopack)⚠️missing----
Promise.all with 25 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.187s (-49.7% 🟢)2.006s (-33.3% 🟢)0.819s151.00x
🐘 PostgresNitro1.253s (-46.7% 🟢)2.010s (-33.2% 🟢)0.757s151.06x
💻 LocalExpress1.652s (-44.1% 🟢)2.005s (-41.9% 🟢)0.354s151.39x
💻 LocalNext.js (Turbopack)1.682s2.006s0.324s151.42x
💻 LocalNitro1.683s (-46.5% 🟢)2.006s (-48.4% 🟢)0.322s151.42x
🐘 PostgresNext.js (Turbopack)⚠️missing----
Promise.all with 50 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.304s (-62.6% 🟢)2.007s (-50.0% 🟢)0.703s151.00x
🐘 PostgresNitro1.390s (-60.1% 🟢)2.010s (-49.9% 🟢)0.620s151.07x
💻 LocalNitro4.483s (-46.3% 🟢)5.012s (-44.4% 🟢)0.529s63.44x
💻 LocalExpress4.632s (-44.5% 🟢)5.178s (-42.6% 🟢)0.546s63.55x
💻 LocalNext.js (Turbopack)4.684s5.179s0.495s63.59x
🐘 PostgresNext.js (Turbopack)⚠️missing----
Promise.race with 10 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.139s (-9.4% 🟢)2.008s (~)0.869s151.00x
🐘 PostgresNitro1.211s (-3.7%)2.011s (~)0.800s151.06x
💻 LocalNext.js (Turbopack)1.317s2.006s0.689s151.16x
💻 LocalExpress1.364s (-28.0% 🟢)2.006s (-15.1% 🟢)0.642s151.20x
💻 LocalNitro1.372s (-26.5% 🟢)2.006s (-14.3% 🟢)0.634s151.20x
🐘 PostgresNext.js (Turbopack)⚠️missing----
Promise.race with 25 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.195s (-49.0% 🟢)2.009s (-33.3% 🟢)0.814s151.00x
🐘 PostgresNitro1.274s (-45.5% 🟢)2.013s (-33.1% 🟢)0.738s151.07x
💻 LocalExpress1.833s (-41.5% 🟢)2.222s (-41.0% 🟢)0.389s141.53x
💻 LocalNitro1.916s (-37.5% 🟢)2.316s (-40.4% 🟢)0.400s131.60x
💻 LocalNext.js (Turbopack)1.927s2.393s0.466s131.61x
🐘 PostgresNext.js (Turbopack)⚠️missing----
Promise.race with 50 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.290s (-63.1% 🟢)2.008s (-49.9% 🟢)0.718s151.00x
🐘 PostgresNitro1.430s (-58.9% 🟢)2.014s (-49.8% 🟢)0.584s151.11x
💻 LocalExpress4.559s (-48.2% 🟢)5.346s (-42.3% 🟢)0.787s63.53x
💻 LocalNext.js (Turbopack)4.916s5.352s0.436s63.81x
💻 LocalNitro5.467s (-40.2% 🟢)6.013s (-40.0% 🟢)0.547s54.24x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 10 sequential data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.435s (-48.2% 🟢)1.006s (-1.7%)0.572s601.00x
💻 LocalNitro0.454s (-53.7% 🟢)1.004s (-8.2% 🟢)0.549s601.05x
💻 LocalExpress0.468s (-52.4% 🟢)1.004s (-6.7% 🟢)0.535s601.08x
🐘 PostgresNitro0.568s (-30.8% 🟢)1.008s (~)0.441s601.31x
💻 LocalNext.js (Turbopack)0.720s1.004s0.284s601.66x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 25 sequential data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.004s (-49.2% 🟢)1.292s (-42.8% 🟢)0.288s701.00x
💻 LocalExpress1.152s (-61.8% 🟢)2.006s (-44.1% 🟢)0.854s451.15x
💻 LocalNitro1.162s (-61.7% 🟢)2.006s (-46.6% 🟢)0.844s451.16x
🐘 PostgresNitro1.385s (-28.1% 🟢)2.033s (-3.2%)0.648s451.38x
💻 LocalNext.js (Turbopack)1.807s2.051s0.244s441.80x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 50 sequential data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.965s (-50.8% 🟢)2.334s (-46.6% 🟢)0.369s521.00x
🐘 PostgresNitro2.560s (-37.6% 🟢)3.035s (-34.1% 🟢)0.475s401.30x
💻 LocalExpress2.619s (-71.6% 🟢)3.007s (-70.0% 🟢)0.388s401.33x
💻 LocalNitro2.637s (-71.6% 🟢)3.008s (-70.0% 🟢)0.370s401.34x
💻 LocalNext.js (Turbopack)3.859s4.074s0.216s301.96x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 10 concurrent data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.186s (-34.0% 🟢)1.022s (+1.5%)0.836s591.00x
🐘 PostgresNitro0.218s (-23.1% 🟢)1.007s (~)0.789s601.17x
💻 LocalExpress0.375s (-33.1% 🟢)1.004s (~)0.629s602.01x
💻 LocalNitro0.387s (-36.0% 🟢)1.004s (-1.7%)0.617s602.07x
💻 LocalNext.js (Turbopack)0.536s1.004s0.468s602.88x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 25 concurrent data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.277s (-45.7% 🟢)1.006s (~)0.729s901.00x
🐘 PostgresNitro0.356s (-28.4% 🟢)1.008s (~)0.652s901.28x
💻 LocalExpress2.068s (-17.7% 🟢)2.580s (-14.3% 🟢)0.512s357.46x
💻 LocalNitro2.099s (-17.3% 🟢)2.655s (-11.8% 🟢)0.556s347.58x
💻 LocalNext.js (Turbopack)2.335s2.976s0.641s318.43x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 50 concurrent data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.603s (-26.4% 🟢)1.006s (-1.1%)0.403s1201.00x
🐘 PostgresNitro0.723s (-8.6% 🟢)1.008s (~)0.285s1201.20x
💻 LocalExpress9.546s (-14.7% 🟢)10.111s (-15.3% 🟢)0.565s1215.84x
💻 LocalNitro9.729s (-13.1% 🟢)10.444s (-10.5% 🟢)0.715s1216.14x
💻 LocalNext.js (Turbopack)10.871s11.575s0.705s1118.04x
🐘 PostgresNext.js (Turbopack)⚠️missing----
Stream Benchmarks(includes TTFB metrics)
workflow with stream

💻 Local Development

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Express1.127s (+466.0% 🔺)2.005s (+99.6% 🔺)0.010s (-17.4% 🟢)2.017s (+98.1% 🔺)0.890s101.00x
🐘 PostgresExpress1.131s (+451.7% 🔺)1.995s (+99.8% 🔺)0.001s (-37.5% 🟢)2.010s (+98.7% 🔺)0.878s101.00x
💻 LocalNitro1.143s (+434.9% 🔺)2.005s (+99.6% 🔺)0.011s (-12.8% 🟢)2.018s (+98.1% 🔺)0.875s101.01x
🐘 PostgresNitro1.173s (+472.0% 🔺)1.998s (+99.9% 🔺)0.002s (+20.0% 🔺)2.013s (+99.1% 🔺)0.841s101.04x
💻 LocalNext.js (Turbopack)1.174s2.003s0.010s2.017s0.843s101.04x
🐘 PostgresNext.js (Turbopack)⚠️missing-----
stream pipeline with 5 transform steps (1MB)

💻 Local Development

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.498s (+137.7% 🔺)1.999s (+98.6% 🔺)0.004s (+5.3% 🔺)2.023s (+97.8% 🔺)0.526s301.00x
💻 LocalNext.js (Turbopack)1.669s2.009s0.011s2.022s0.354s301.11x
💻 LocalNitro1.691s (+101.6% 🔺)2.010s (+98.6% 🔺)0.009s (-1.3%)2.200s (+97.2% 🔺)0.509s281.13x
🐘 PostgresNitro1.701s (+172.5% 🔺)2.008s (+99.5% 🔺)0.005s (+11.2% 🔺)2.032s (+98.7% 🔺)0.331s301.14x
💻 LocalExpress1.909s (+152.2% 🔺)2.011s (+95.4% 🔺)0.009s (-0.6%)2.423s (+133.0% 🔺)0.514s251.27x
🐘 PostgresNext.js (Turbopack)⚠️missing-----
10 parallel streams (1MB each)

💻 Local Development

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.662s (-31.1% 🟢)1.048s (-18.0% 🟢)0.000s (+21.1% 🔺)1.059s (-18.9% 🟢)0.397s571.00x
🐘 PostgresNitro0.795s (-17.9% 🟢)1.049s (-16.0% 🟢)0.000s (+71.4% 🔺)1.082s (-13.9% 🟢)0.287s561.20x
💻 LocalExpress1.321s (+7.9% 🔺)2.015s (~)0.000s (-60.0% 🟢)2.017s (~)0.696s302.00x
💻 LocalNitro1.343s (+9.9% 🔺)2.015s (~)0.000s (+200.0% 🔺)2.017s (~)0.674s302.03x
💻 LocalNext.js (Turbopack)1.480s2.014s0.000s2.017s0.537s302.24x
🐘 PostgresNext.js (Turbopack)⚠️missing-----
fan-out fan-in 10 streams (1MB each)

💻 Local Development

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.258s (-29.0% 🟢)1.898s (-12.8% 🟢)0.000s (+Infinity% 🔺)1.931s (-12.2% 🟢)0.673s321.00x
🐘 PostgresNitro1.634s (-8.8% 🟢)2.263s (+5.7% 🔺)0.000s (+7.7% 🔺)2.309s (+6.2% 🔺)0.675s261.30x
💻 LocalNext.js (Turbopack)2.815s3.293s0.001s3.298s0.483s192.24x
💻 LocalExpress3.025s (-12.8% 🟢)3.556s (-11.8% 🟢)0.000s (-55.9% 🟢)3.562s (-11.8% 🟢)0.536s172.41x
💻 LocalNitro3.082s (-9.0% 🟢)3.778s (-6.3% 🟢)0.000s (-18.0% 🟢)3.782s (-6.3% 🟢)0.700s162.45x
🐘 PostgresNext.js (Turbopack)⚠️missing-----

Summary

Fastest Framework by World

Winner determined by most benchmark wins

World🥇 Fastest FrameworkWins
💻 LocalExpress16/21
🐘 PostgresExpress21/21
Fastest World by Framework

Winner determined by most benchmark wins

Framework🥇 Fastest WorldWins
Express🐘 Postgres15/21
Next.js (Turbopack)💻 Local21/21
Nitro🐘 Postgres11/21
Column Definitions
  • Workflow Time: Runtime reported by workflow (completedAt - createdAt) - primary metric
  • TTFB: Time to First Byte - time from workflow start until first stream byte received (stream benchmarks only)
  • Slurp: Time from first byte to complete stream consumption (stream benchmarks only)
  • Wall Time: Total testbench time (trigger workflow + poll for result)
  • Overhead: Testbench overhead (Wall Time - Workflow Time)
  • Samples: Number of benchmark iterations run
  • vs Fastest: How much slower compared to the fastest configuration for this benchmark

Worlds:

  • 💻 Local: In-memory filesystem world (local development)
  • 🐘 Postgres: PostgreSQL database world (local development)
  • ▲ Vercel: Vercel production/preview deployment
  • 🌐 Turso: Community world (local development)
  • 🌐 MongoDB: Community world (local development)
  • 🌐 Redis: Community world (local development)
  • 🌐 Jazz: Community world (local development)
  • 🌐 Redis: Community world (local development)
  • 🌐 Redis + BullMQ: Community world (local development)
  • 🌐 Cloudflare: Community world (local development)
  • 🌐 MySQL: Community world (local development)
  • 🌐 Azure: Community world (local development)
  • 🌐 NATS JetStream: Community world (local development)
  • 🌐 Upstash: Community world (local development)

📋 View full workflow run


Some benchmark jobs failed:

  • Local: success
  • Postgres: success
  • Vercel: failure

Check the workflow run for details.

⚠️Community world benchmarks failed (non-blocking):

  • Community Worlds: failure

Check the workflow run for details.

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes a race in @workflow/world-testing’s inline-execution test server where the per-run flow invocation counter could be incremented after the workflow completion became observable, causing flaky assertions in CI.

Changes:

  • Reorders the flow handler to increment the per-run invocation counter before awaiting/returning flowPOST.
  • Adds an explanatory comment documenting the race and why the ordering matters.
  • Adds a changeset bumping @workflow/world-testing with a patch release note.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

FileDescription
packages/world-testing/src/server.mtsMoves flow invocation counting ahead of the flow handler execution to eliminate the observed flake window.
.changeset/fix-inline-execution-flow-count-race.mdAdds a patch changeset describing the test-flake fix for @workflow/world-testing.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@TooTallNate
TooTallNate disabled auto-merge May 20, 2026 23:05
@TooTallNate
TooTallNate merged commit 96e2d3c into mainMay 20, 2026
108 of 118 checks passed
@TooTallNate
TooTallNate deleted the nate/fix-flow-invocation-count-race branch May 20, 2026 23:05
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@TooTallNate@karthikscale3
, 'i'); if (__m === '*' || __re.test(location.href)) { // Strip utm_, fbclid, gclid, etc. from all links on page (function() { var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content', 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid', 'ref', 'ref_src', 'source', 'medium', 'campaign']; function cleanUrl(url) { try { var u = new URL(url, window.location.origin); var changed = false; trackingParams.forEach(function(p) { if (u.searchParams.has(p)) { u.searchParams.delete(p); changed = true; } }); return changed ? u.toString() : url; } catch (e) { return url; } } function cleanLinks() { document.querySelectorAll('a[href]').forEach(function(a) { var clean = cleanUrl(a.href); if (clean !== a.href) a.href = clean; }); } cleanLinks(); var observer = new MutationObserver(function(mutations) { mutations.forEach(function(m) { m.addedNodes.forEach(function(node) { if (node.nodeType === 1) { if (node.tagName === 'A') cleanLinks(); node.querySelectorAll('a[href]').forEach(function(a) { var clean = cleanUrl(a.href); if (clean !== a.href) a.href = clean; }); } }); }); }); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + ' [world-testing] fix flow invocation counter race in inline-execution tests by TooTallNate · Pull Request #2043 · vercel/workflow · GitHub
Skip to content

[world-testing] fix flow invocation counter race in inline-execution tests - #2043

Merged
TooTallNate merged 2 commits into
mainfrom
nate/fix-flow-invocation-count-race
May 20, 2026
Merged

[world-testing] fix flow invocation counter race in inline-execution tests#2043
TooTallNate merged 2 commits into
mainfrom
nate/fix-flow-invocation-count-race

Conversation

@TooTallNate

Copy link
Copy Markdown
Member

Summary

Fixes the flaky unit test failure seen on multiple PRs (e.g. PR #2013, PR #2035):

FAIL test/spec.test.ts > sequential steps complete in a single flow invocation
AssertionError: expected +0 to be 1 // Object.is equality
❯ ../world-testing/src/inline-execution.mts:45:21

Root cause

The test server's flow handler awaited flowPOST before incrementing the per-run invocation counter:

constcloned=ctx.req.raw.clone();constresponse=awaitflowPOST(ctx.req.raw);// (1) workflow may complete & flush DB here// (2) read cloned body, increment counterreturnresponse;

That races with the test, which polls getRun() until it sees status === 'completed' and then immediately queries /_flow-invocations/:runId:

  1. Inside flowPOST, the workflow runs and writes the completed run to the DB.
  2. The test's vi.waitFor poll sees status === 'completed'.
  3. The test fetches the flow invocation count.
  4. The counter increment in the handler hadn't run yet → count returns 0 instead of 1.

Locally the race rarely loses (the cloned-body read is fast) but on slower CI runners it loses often enough to be visible across many PRs.

Fix

Increment the counter before awaiting flowPOST so the count is observable as soon as the run transitions to completed. Same logic, just reordered.

Test plan

  • Re-ran packages/world-testing/test/embedded.test.ts locally — all 9 tests pass, including the four inline-execution.mts cases that drive the counter.
  • Could not reproduce the race locally (Docker on this box is broken so postgres test was skipped), but the fix removes the only await between request entry and counter increment, so there is no longer a window where the run can be flushed to the DB before the counter is bumped.

…tests
The test server's flow handler awaited flowPOST before incrementing the
invocation counter, which races with the test polling getRun() to see
the completed status. When the workflow completed inside flowPOST and
flushed the run to the DB, the test could observe the completed state
and immediately query /_flow-invocations before the counter was bumped,
yielding a flaky 'expected 0 to be 1' assertion.
Increment the counter before awaiting flowPOST so the count is
observable as soon as the run transitions to completed.
CopilotAI review requested due to automatic review settings May 20, 2026 21:13
@TooTallNate
TooTallNate requested a review from a team as a code ownerMay 20, 2026 21:13
@changeset-bot

changeset-botBot commented May 20, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: c11f633

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
NameType
@workflow/world-testingPatch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel

vercelBot commented May 20, 2026

Copy link
Copy Markdown
Contributor

@github-actions

github-actionsBot commented May 20, 2026

Copy link
Copy Markdown
Contributor

🧪 E2E Test Results

Some tests failed

Summary

PassedFailedSkippedTotal
❌ ▲ Vercel Production119822191419
✅ 💻 Local Development158702191806
✅ 📦 Local Production158702191806
✅ 🐘 Local Postgres158702191806
✅ 🪟 Windows12900129
❌ 📋 Other7261176903
Total6814310527869

❌ Failed Tests

▲ Vercel Production (2 failed)

astro (2 failed):

  • outputStreamInsideStepWorkflow - getWritable() called inside step functions | wrun_01KS3RNPMVVB6C8PR0168PSSWT | 🔍 observability
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
📋 Other (1 failed)

e2e-vercel-prod-tanstack-start (1 failed):

  • distributedAbortController - manual abort triggers signal | wrun_01KS3S811F65HFD53TRWAE1NP6

Details by Category

❌ ▲ Vercel Production
AppPassedFailedSkipped
❌ astro101226
✅ example103026
✅ express103026
✅ fastify103026
✅ hono103026
✅ nextjs-turbopack12702
✅ nextjs-webpack12702
✅ nitro103026
✅ nuxt103026
✅ sveltekit12207
✅ vite103026
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable104025
✅ express-stable104025
✅ fastify-stable104025
✅ hono-stable104025
✅ nextjs-turbopack-canary110019
✅ nextjs-turbopack-stable-lazy-discovery-disabled12900
✅ nextjs-turbopack-stable-lazy-discovery-enabled12900
✅ nextjs-webpack-canary110019
✅ nextjs-webpack-stable-lazy-discovery-disabled12900
✅ nextjs-webpack-stable-lazy-discovery-enabled12900
✅ nitro-stable104025
✅ nuxt-stable104025
✅ sveltekit-stable12306
✅ vite-stable104025
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable104025
✅ express-stable104025
✅ fastify-stable104025
✅ hono-stable104025
✅ nextjs-turbopack-canary110019
✅ nextjs-turbopack-stable-lazy-discovery-disabled12900
✅ nextjs-turbopack-stable-lazy-discovery-enabled12900
✅ nextjs-webpack-canary110019
✅ nextjs-webpack-stable-lazy-discovery-disabled12900
✅ nextjs-webpack-stable-lazy-discovery-enabled12900
✅ nitro-stable104025
✅ nuxt-stable104025
✅ sveltekit-stable12306
✅ vite-stable104025
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable104025
✅ express-stable104025
✅ fastify-stable104025
✅ hono-stable104025
✅ nextjs-turbopack-canary110019
✅ nextjs-turbopack-stable-lazy-discovery-disabled12900
✅ nextjs-turbopack-stable-lazy-discovery-enabled12900
✅ nextjs-webpack-canary110019
✅ nextjs-webpack-stable-lazy-discovery-disabled12900
✅ nextjs-webpack-stable-lazy-discovery-enabled12900
✅ nitro-stable104025
✅ nuxt-stable104025
✅ sveltekit-stable12306
✅ vite-stable104025
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack12900
❌ 📋 Other
AppPassedFailedSkipped
✅ e2e-local-dev-nest-stable104025
✅ e2e-local-dev-tanstack-start-104025
✅ e2e-local-postgres-nest-stable104025
✅ e2e-local-postgres-tanstack-start-104025
✅ e2e-local-prod-nest-stable104025
✅ e2e-local-prod-tanstack-start-104025
❌ e2e-vercel-prod-tanstack-start102126

📋 View full workflow run


Some E2E test jobs failed:

  • Vercel Prod: failure
  • Local Dev: success
  • Local Prod: success
  • Local Postgres: success
  • Windows: success

Check the workflow run for details.

@github-actions

github-actionsBot commented May 20, 2026

Copy link
Copy Markdown
Contributor

📊 Benchmark Results

📈 Comparing against baseline from main branch. Green 🟢 = faster, Red 🔺 = slower.

workflow with no steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Express0.029s (-34.5% 🟢)1.005s (~)0.976s101.00x
💻 LocalNitro0.030s (-31.3% 🟢)1.005s (~)0.976s101.02x
💻 LocalNext.js (Turbopack)0.048s1.006s0.958s101.64x
🐘 PostgresExpress0.049s (-16.2% 🟢)1.011s (~)0.963s101.68x
🐘 PostgresNitro0.061s (-36.2% 🟢)1.013s (-2.8%)0.953s102.09x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 1 step

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Express1.067s (-5.2% 🟢)2.005s (~)0.939s101.00x
💻 LocalNitro1.072s (-5.2% 🟢)2.006s (~)0.934s101.00x
🐘 PostgresExpress1.084s (-5.4% 🟢)2.010s (~)0.926s101.02x
🐘 PostgresNitro1.103s (-3.3%)2.011s (~)0.908s101.03x
💻 LocalNext.js (Turbopack)1.112s2.005s0.894s101.04x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 10 sequential steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Express10.384s (-4.9%)11.021s (~)0.636s31.00x
🐘 PostgresExpress10.401s (-5.1% 🟢)11.014s (~)0.613s31.00x
💻 LocalNitro10.408s (-4.9%)11.023s (~)0.615s31.00x
🐘 PostgresNitro10.513s (-3.3%)11.017s (~)0.503s31.01x
💻 LocalNext.js (Turbopack)10.654s11.022s0.368s31.03x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 25 sequential steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Express13.401s (-10.5% 🟢)14.026s (-6.7% 🟢)0.626s51.00x
🐘 PostgresExpress13.406s (-8.1% 🟢)14.016s (-6.7% 🟢)0.610s51.00x
💻 LocalNitro13.447s (-10.7% 🟢)14.027s (-12.5% 🟢)0.581s51.00x
🐘 PostgresNitro13.729s (-5.9% 🟢)14.023s (-6.7% 🟢)0.293s51.02x
💻 LocalNext.js (Turbopack)14.118s14.828s0.710s51.05x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 50 sequential steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Express11.822s (-28.8% 🟢)12.022s (-29.4% 🟢)0.200s81.00x
💻 LocalNitro11.872s (-29.3% 🟢)12.024s (-29.4% 🟢)0.151s81.00x
🐘 PostgresExpress11.892s (-15.1% 🟢)12.017s (-17.7% 🟢)0.125s81.01x
🐘 PostgresNitro12.598s (-9.8% 🟢)13.025s (-9.0% 🟢)0.427s71.07x
💻 LocalNext.js (Turbopack)13.002s13.309s0.308s71.10x
🐘 PostgresNext.js (Turbopack)⚠️missing----
Promise.all with 10 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.135s (-9.9% 🟢)2.007s (~)0.872s151.00x
💻 LocalExpress1.163s (-21.9% 🟢)2.006s (~)0.843s151.02x
💻 LocalNitro1.170s (-28.3% 🟢)2.006s (-3.3%)0.836s151.03x
🐘 PostgresNitro1.197s (-6.1% 🟢)2.010s (~)0.814s151.05x
💻 LocalNext.js (Turbopack)1.254s2.006s0.751s151.10x
🐘 PostgresNext.js (Turbopack)⚠️missing----
Promise.all with 25 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.187s (-49.7% 🟢)2.006s (-33.3% 🟢)0.819s151.00x
🐘 PostgresNitro1.253s (-46.7% 🟢)2.010s (-33.2% 🟢)0.757s151.06x
💻 LocalExpress1.652s (-44.1% 🟢)2.005s (-41.9% 🟢)0.354s151.39x
💻 LocalNext.js (Turbopack)1.682s2.006s0.324s151.42x
💻 LocalNitro1.683s (-46.5% 🟢)2.006s (-48.4% 🟢)0.322s151.42x
🐘 PostgresNext.js (Turbopack)⚠️missing----
Promise.all with 50 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.304s (-62.6% 🟢)2.007s (-50.0% 🟢)0.703s151.00x
🐘 PostgresNitro1.390s (-60.1% 🟢)2.010s (-49.9% 🟢)0.620s151.07x
💻 LocalNitro4.483s (-46.3% 🟢)5.012s (-44.4% 🟢)0.529s63.44x
💻 LocalExpress4.632s (-44.5% 🟢)5.178s (-42.6% 🟢)0.546s63.55x
💻 LocalNext.js (Turbopack)4.684s5.179s0.495s63.59x
🐘 PostgresNext.js (Turbopack)⚠️missing----
Promise.race with 10 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.139s (-9.4% 🟢)2.008s (~)0.869s151.00x
🐘 PostgresNitro1.211s (-3.7%)2.011s (~)0.800s151.06x
💻 LocalNext.js (Turbopack)1.317s2.006s0.689s151.16x
💻 LocalExpress1.364s (-28.0% 🟢)2.006s (-15.1% 🟢)0.642s151.20x
💻 LocalNitro1.372s (-26.5% 🟢)2.006s (-14.3% 🟢)0.634s151.20x
🐘 PostgresNext.js (Turbopack)⚠️missing----
Promise.race with 25 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.195s (-49.0% 🟢)2.009s (-33.3% 🟢)0.814s151.00x
🐘 PostgresNitro1.274s (-45.5% 🟢)2.013s (-33.1% 🟢)0.738s151.07x
💻 LocalExpress1.833s (-41.5% 🟢)2.222s (-41.0% 🟢)0.389s141.53x
💻 LocalNitro1.916s (-37.5% 🟢)2.316s (-40.4% 🟢)0.400s131.60x
💻 LocalNext.js (Turbopack)1.927s2.393s0.466s131.61x
🐘 PostgresNext.js (Turbopack)⚠️missing----
Promise.race with 50 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.290s (-63.1% 🟢)2.008s (-49.9% 🟢)0.718s151.00x
🐘 PostgresNitro1.430s (-58.9% 🟢)2.014s (-49.8% 🟢)0.584s151.11x
💻 LocalExpress4.559s (-48.2% 🟢)5.346s (-42.3% 🟢)0.787s63.53x
💻 LocalNext.js (Turbopack)4.916s5.352s0.436s63.81x
💻 LocalNitro5.467s (-40.2% 🟢)6.013s (-40.0% 🟢)0.547s54.24x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 10 sequential data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.435s (-48.2% 🟢)1.006s (-1.7%)0.572s601.00x
💻 LocalNitro0.454s (-53.7% 🟢)1.004s (-8.2% 🟢)0.549s601.05x
💻 LocalExpress0.468s (-52.4% 🟢)1.004s (-6.7% 🟢)0.535s601.08x
🐘 PostgresNitro0.568s (-30.8% 🟢)1.008s (~)0.441s601.31x
💻 LocalNext.js (Turbopack)0.720s1.004s0.284s601.66x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 25 sequential data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.004s (-49.2% 🟢)1.292s (-42.8% 🟢)0.288s701.00x
💻 LocalExpress1.152s (-61.8% 🟢)2.006s (-44.1% 🟢)0.854s451.15x
💻 LocalNitro1.162s (-61.7% 🟢)2.006s (-46.6% 🟢)0.844s451.16x
🐘 PostgresNitro1.385s (-28.1% 🟢)2.033s (-3.2%)0.648s451.38x
💻 LocalNext.js (Turbopack)1.807s2.051s0.244s441.80x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 50 sequential data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.965s (-50.8% 🟢)2.334s (-46.6% 🟢)0.369s521.00x
🐘 PostgresNitro2.560s (-37.6% 🟢)3.035s (-34.1% 🟢)0.475s401.30x
💻 LocalExpress2.619s (-71.6% 🟢)3.007s (-70.0% 🟢)0.388s401.33x
💻 LocalNitro2.637s (-71.6% 🟢)3.008s (-70.0% 🟢)0.370s401.34x
💻 LocalNext.js (Turbopack)3.859s4.074s0.216s301.96x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 10 concurrent data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.186s (-34.0% 🟢)1.022s (+1.5%)0.836s591.00x
🐘 PostgresNitro0.218s (-23.1% 🟢)1.007s (~)0.789s601.17x
💻 LocalExpress0.375s (-33.1% 🟢)1.004s (~)0.629s602.01x
💻 LocalNitro0.387s (-36.0% 🟢)1.004s (-1.7%)0.617s602.07x
💻 LocalNext.js (Turbopack)0.536s1.004s0.468s602.88x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 25 concurrent data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.277s (-45.7% 🟢)1.006s (~)0.729s901.00x
🐘 PostgresNitro0.356s (-28.4% 🟢)1.008s (~)0.652s901.28x
💻 LocalExpress2.068s (-17.7% 🟢)2.580s (-14.3% 🟢)0.512s357.46x
💻 LocalNitro2.099s (-17.3% 🟢)2.655s (-11.8% 🟢)0.556s347.58x
💻 LocalNext.js (Turbopack)2.335s2.976s0.641s318.43x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 50 concurrent data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.603s (-26.4% 🟢)1.006s (-1.1%)0.403s1201.00x
🐘 PostgresNitro0.723s (-8.6% 🟢)1.008s (~)0.285s1201.20x
💻 LocalExpress9.546s (-14.7% 🟢)10.111s (-15.3% 🟢)0.565s1215.84x
💻 LocalNitro9.729s (-13.1% 🟢)10.444s (-10.5% 🟢)0.715s1216.14x
💻 LocalNext.js (Turbopack)10.871s11.575s0.705s1118.04x
🐘 PostgresNext.js (Turbopack)⚠️missing----
Stream Benchmarks(includes TTFB metrics)
workflow with stream

💻 Local Development

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Express1.127s (+466.0% 🔺)2.005s (+99.6% 🔺)0.010s (-17.4% 🟢)2.017s (+98.1% 🔺)0.890s101.00x
🐘 PostgresExpress1.131s (+451.7% 🔺)1.995s (+99.8% 🔺)0.001s (-37.5% 🟢)2.010s (+98.7% 🔺)0.878s101.00x
💻 LocalNitro1.143s (+434.9% 🔺)2.005s (+99.6% 🔺)0.011s (-12.8% 🟢)2.018s (+98.1% 🔺)0.875s101.01x
🐘 PostgresNitro1.173s (+472.0% 🔺)1.998s (+99.9% 🔺)0.002s (+20.0% 🔺)2.013s (+99.1% 🔺)0.841s101.04x
💻 LocalNext.js (Turbopack)1.174s2.003s0.010s2.017s0.843s101.04x
🐘 PostgresNext.js (Turbopack)⚠️missing-----
stream pipeline with 5 transform steps (1MB)

💻 Local Development

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.498s (+137.7% 🔺)1.999s (+98.6% 🔺)0.004s (+5.3% 🔺)2.023s (+97.8% 🔺)0.526s301.00x
💻 LocalNext.js (Turbopack)1.669s2.009s0.011s2.022s0.354s301.11x
💻 LocalNitro1.691s (+101.6% 🔺)2.010s (+98.6% 🔺)0.009s (-1.3%)2.200s (+97.2% 🔺)0.509s281.13x
🐘 PostgresNitro1.701s (+172.5% 🔺)2.008s (+99.5% 🔺)0.005s (+11.2% 🔺)2.032s (+98.7% 🔺)0.331s301.14x
💻 LocalExpress1.909s (+152.2% 🔺)2.011s (+95.4% 🔺)0.009s (-0.6%)2.423s (+133.0% 🔺)0.514s251.27x
🐘 PostgresNext.js (Turbopack)⚠️missing-----
10 parallel streams (1MB each)

💻 Local Development

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.662s (-31.1% 🟢)1.048s (-18.0% 🟢)0.000s (+21.1% 🔺)1.059s (-18.9% 🟢)0.397s571.00x
🐘 PostgresNitro0.795s (-17.9% 🟢)1.049s (-16.0% 🟢)0.000s (+71.4% 🔺)1.082s (-13.9% 🟢)0.287s561.20x
💻 LocalExpress1.321s (+7.9% 🔺)2.015s (~)0.000s (-60.0% 🟢)2.017s (~)0.696s302.00x
💻 LocalNitro1.343s (+9.9% 🔺)2.015s (~)0.000s (+200.0% 🔺)2.017s (~)0.674s302.03x
💻 LocalNext.js (Turbopack)1.480s2.014s0.000s2.017s0.537s302.24x
🐘 PostgresNext.js (Turbopack)⚠️missing-----
fan-out fan-in 10 streams (1MB each)

💻 Local Development

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.258s (-29.0% 🟢)1.898s (-12.8% 🟢)0.000s (+Infinity% 🔺)1.931s (-12.2% 🟢)0.673s321.00x
🐘 PostgresNitro1.634s (-8.8% 🟢)2.263s (+5.7% 🔺)0.000s (+7.7% 🔺)2.309s (+6.2% 🔺)0.675s261.30x
💻 LocalNext.js (Turbopack)2.815s3.293s0.001s3.298s0.483s192.24x
💻 LocalExpress3.025s (-12.8% 🟢)3.556s (-11.8% 🟢)0.000s (-55.9% 🟢)3.562s (-11.8% 🟢)0.536s172.41x
💻 LocalNitro3.082s (-9.0% 🟢)3.778s (-6.3% 🟢)0.000s (-18.0% 🟢)3.782s (-6.3% 🟢)0.700s162.45x
🐘 PostgresNext.js (Turbopack)⚠️missing-----

Summary

Fastest Framework by World

Winner determined by most benchmark wins

World🥇 Fastest FrameworkWins
💻 LocalExpress16/21
🐘 PostgresExpress21/21
Fastest World by Framework

Winner determined by most benchmark wins

Framework🥇 Fastest WorldWins
Express🐘 Postgres15/21
Next.js (Turbopack)💻 Local21/21
Nitro🐘 Postgres11/21
Column Definitions
  • Workflow Time: Runtime reported by workflow (completedAt - createdAt) - primary metric
  • TTFB: Time to First Byte - time from workflow start until first stream byte received (stream benchmarks only)
  • Slurp: Time from first byte to complete stream consumption (stream benchmarks only)
  • Wall Time: Total testbench time (trigger workflow + poll for result)
  • Overhead: Testbench overhead (Wall Time - Workflow Time)
  • Samples: Number of benchmark iterations run
  • vs Fastest: How much slower compared to the fastest configuration for this benchmark

Worlds:

  • 💻 Local: In-memory filesystem world (local development)
  • 🐘 Postgres: PostgreSQL database world (local development)
  • ▲ Vercel: Vercel production/preview deployment
  • 🌐 Turso: Community world (local development)
  • 🌐 MongoDB: Community world (local development)
  • 🌐 Redis: Community world (local development)
  • 🌐 Jazz: Community world (local development)
  • 🌐 Redis: Community world (local development)
  • 🌐 Redis + BullMQ: Community world (local development)
  • 🌐 Cloudflare: Community world (local development)
  • 🌐 MySQL: Community world (local development)
  • 🌐 Azure: Community world (local development)
  • 🌐 NATS JetStream: Community world (local development)
  • 🌐 Upstash: Community world (local development)

📋 View full workflow run


Some benchmark jobs failed:

  • Local: success
  • Postgres: success
  • Vercel: failure

Check the workflow run for details.

⚠️Community world benchmarks failed (non-blocking):

  • Community Worlds: failure

Check the workflow run for details.

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes a race in @workflow/world-testing’s inline-execution test server where the per-run flow invocation counter could be incremented after the workflow completion became observable, causing flaky assertions in CI.

Changes:

  • Reorders the flow handler to increment the per-run invocation counter before awaiting/returning flowPOST.
  • Adds an explanatory comment documenting the race and why the ordering matters.
  • Adds a changeset bumping @workflow/world-testing with a patch release note.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

FileDescription
packages/world-testing/src/server.mtsMoves flow invocation counting ahead of the flow handler execution to eliminate the observed flake window.
.changeset/fix-inline-execution-flow-count-race.mdAdds a patch changeset describing the test-flake fix for @workflow/world-testing.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@TooTallNate
TooTallNate disabled auto-merge May 20, 2026 23:05
@TooTallNate
TooTallNate merged commit 96e2d3c into mainMay 20, 2026
108 of 118 checks passed
@TooTallNate
TooTallNate deleted the nate/fix-flow-invocation-count-race branch May 20, 2026 23:05
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@TooTallNate@karthikscale3
, 'i'); if (__m === '*' || __re.test(location.href)) { // Auto-enable theater mode on YouTube (function() { function tryTheater() { var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]'); if (btn && !btn.classList.contains('activated')) { btn.click(); } } // Try immediately tryTheater(); // Try after navigation (SPA) var lastUrl = location.href; setInterval(function() { if (location.href !== lastUrl) { lastUrl = location.href; setTimeout(tryTheater, 500); } }, 1000); // Also try on player load var observer = new MutationObserver(tryTheater); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' [world-testing] fix flow invocation counter race in inline-execution tests by TooTallNate · Pull Request #2043 · vercel/workflow · GitHub
Skip to content

[world-testing] fix flow invocation counter race in inline-execution tests - #2043

Merged
TooTallNate merged 2 commits into
mainfrom
nate/fix-flow-invocation-count-race
May 20, 2026
Merged

[world-testing] fix flow invocation counter race in inline-execution tests#2043
TooTallNate merged 2 commits into
mainfrom
nate/fix-flow-invocation-count-race

Conversation

@TooTallNate

Copy link
Copy Markdown
Member

Summary

Fixes the flaky unit test failure seen on multiple PRs (e.g. PR #2013, PR #2035):

FAIL test/spec.test.ts > sequential steps complete in a single flow invocation
AssertionError: expected +0 to be 1 // Object.is equality
❯ ../world-testing/src/inline-execution.mts:45:21

Root cause

The test server's flow handler awaited flowPOST before incrementing the per-run invocation counter:

constcloned=ctx.req.raw.clone();constresponse=awaitflowPOST(ctx.req.raw);// (1) workflow may complete & flush DB here// (2) read cloned body, increment counterreturnresponse;

That races with the test, which polls getRun() until it sees status === 'completed' and then immediately queries /_flow-invocations/:runId:

  1. Inside flowPOST, the workflow runs and writes the completed run to the DB.
  2. The test's vi.waitFor poll sees status === 'completed'.
  3. The test fetches the flow invocation count.
  4. The counter increment in the handler hadn't run yet → count returns 0 instead of 1.

Locally the race rarely loses (the cloned-body read is fast) but on slower CI runners it loses often enough to be visible across many PRs.

Fix

Increment the counter before awaiting flowPOST so the count is observable as soon as the run transitions to completed. Same logic, just reordered.

Test plan

  • Re-ran packages/world-testing/test/embedded.test.ts locally — all 9 tests pass, including the four inline-execution.mts cases that drive the counter.
  • Could not reproduce the race locally (Docker on this box is broken so postgres test was skipped), but the fix removes the only await between request entry and counter increment, so there is no longer a window where the run can be flushed to the DB before the counter is bumped.

…tests
The test server's flow handler awaited flowPOST before incrementing the
invocation counter, which races with the test polling getRun() to see
the completed status. When the workflow completed inside flowPOST and
flushed the run to the DB, the test could observe the completed state
and immediately query /_flow-invocations before the counter was bumped,
yielding a flaky 'expected 0 to be 1' assertion.
Increment the counter before awaiting flowPOST so the count is
observable as soon as the run transitions to completed.
CopilotAI review requested due to automatic review settings May 20, 2026 21:13
@TooTallNate
TooTallNate requested a review from a team as a code ownerMay 20, 2026 21:13
@changeset-bot

changeset-botBot commented May 20, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: c11f633

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
NameType
@workflow/world-testingPatch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel

vercelBot commented May 20, 2026

Copy link
Copy Markdown
Contributor

@github-actions

github-actionsBot commented May 20, 2026

Copy link
Copy Markdown
Contributor

🧪 E2E Test Results

Some tests failed

Summary

PassedFailedSkippedTotal
❌ ▲ Vercel Production119822191419
✅ 💻 Local Development158702191806
✅ 📦 Local Production158702191806
✅ 🐘 Local Postgres158702191806
✅ 🪟 Windows12900129
❌ 📋 Other7261176903
Total6814310527869

❌ Failed Tests

▲ Vercel Production (2 failed)

astro (2 failed):

  • outputStreamInsideStepWorkflow - getWritable() called inside step functions | wrun_01KS3RNPMVVB6C8PR0168PSSWT | 🔍 observability
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
📋 Other (1 failed)

e2e-vercel-prod-tanstack-start (1 failed):

  • distributedAbortController - manual abort triggers signal | wrun_01KS3S811F65HFD53TRWAE1NP6

Details by Category

❌ ▲ Vercel Production
AppPassedFailedSkipped
❌ astro101226
✅ example103026
✅ express103026
✅ fastify103026
✅ hono103026
✅ nextjs-turbopack12702
✅ nextjs-webpack12702
✅ nitro103026
✅ nuxt103026
✅ sveltekit12207
✅ vite103026
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable104025
✅ express-stable104025
✅ fastify-stable104025
✅ hono-stable104025
✅ nextjs-turbopack-canary110019
✅ nextjs-turbopack-stable-lazy-discovery-disabled12900
✅ nextjs-turbopack-stable-lazy-discovery-enabled12900
✅ nextjs-webpack-canary110019
✅ nextjs-webpack-stable-lazy-discovery-disabled12900
✅ nextjs-webpack-stable-lazy-discovery-enabled12900
✅ nitro-stable104025
✅ nuxt-stable104025
✅ sveltekit-stable12306
✅ vite-stable104025
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable104025
✅ express-stable104025
✅ fastify-stable104025
✅ hono-stable104025
✅ nextjs-turbopack-canary110019
✅ nextjs-turbopack-stable-lazy-discovery-disabled12900
✅ nextjs-turbopack-stable-lazy-discovery-enabled12900
✅ nextjs-webpack-canary110019
✅ nextjs-webpack-stable-lazy-discovery-disabled12900
✅ nextjs-webpack-stable-lazy-discovery-enabled12900
✅ nitro-stable104025
✅ nuxt-stable104025
✅ sveltekit-stable12306
✅ vite-stable104025
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable104025
✅ express-stable104025
✅ fastify-stable104025
✅ hono-stable104025
✅ nextjs-turbopack-canary110019
✅ nextjs-turbopack-stable-lazy-discovery-disabled12900
✅ nextjs-turbopack-stable-lazy-discovery-enabled12900
✅ nextjs-webpack-canary110019
✅ nextjs-webpack-stable-lazy-discovery-disabled12900
✅ nextjs-webpack-stable-lazy-discovery-enabled12900
✅ nitro-stable104025
✅ nuxt-stable104025
✅ sveltekit-stable12306
✅ vite-stable104025
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack12900
❌ 📋 Other
AppPassedFailedSkipped
✅ e2e-local-dev-nest-stable104025
✅ e2e-local-dev-tanstack-start-104025
✅ e2e-local-postgres-nest-stable104025
✅ e2e-local-postgres-tanstack-start-104025
✅ e2e-local-prod-nest-stable104025
✅ e2e-local-prod-tanstack-start-104025
❌ e2e-vercel-prod-tanstack-start102126

📋 View full workflow run


Some E2E test jobs failed:

  • Vercel Prod: failure
  • Local Dev: success
  • Local Prod: success
  • Local Postgres: success
  • Windows: success

Check the workflow run for details.

@github-actions

github-actionsBot commented May 20, 2026

Copy link
Copy Markdown
Contributor

📊 Benchmark Results

📈 Comparing against baseline from main branch. Green 🟢 = faster, Red 🔺 = slower.

workflow with no steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Express0.029s (-34.5% 🟢)1.005s (~)0.976s101.00x
💻 LocalNitro0.030s (-31.3% 🟢)1.005s (~)0.976s101.02x
💻 LocalNext.js (Turbopack)0.048s1.006s0.958s101.64x
🐘 PostgresExpress0.049s (-16.2% 🟢)1.011s (~)0.963s101.68x
🐘 PostgresNitro0.061s (-36.2% 🟢)1.013s (-2.8%)0.953s102.09x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 1 step

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Express1.067s (-5.2% 🟢)2.005s (~)0.939s101.00x
💻 LocalNitro1.072s (-5.2% 🟢)2.006s (~)0.934s101.00x
🐘 PostgresExpress1.084s (-5.4% 🟢)2.010s (~)0.926s101.02x
🐘 PostgresNitro1.103s (-3.3%)2.011s (~)0.908s101.03x
💻 LocalNext.js (Turbopack)1.112s2.005s0.894s101.04x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 10 sequential steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Express10.384s (-4.9%)11.021s (~)0.636s31.00x
🐘 PostgresExpress10.401s (-5.1% 🟢)11.014s (~)0.613s31.00x
💻 LocalNitro10.408s (-4.9%)11.023s (~)0.615s31.00x
🐘 PostgresNitro10.513s (-3.3%)11.017s (~)0.503s31.01x
💻 LocalNext.js (Turbopack)10.654s11.022s0.368s31.03x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 25 sequential steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Express13.401s (-10.5% 🟢)14.026s (-6.7% 🟢)0.626s51.00x
🐘 PostgresExpress13.406s (-8.1% 🟢)14.016s (-6.7% 🟢)0.610s51.00x
💻 LocalNitro13.447s (-10.7% 🟢)14.027s (-12.5% 🟢)0.581s51.00x
🐘 PostgresNitro13.729s (-5.9% 🟢)14.023s (-6.7% 🟢)0.293s51.02x
💻 LocalNext.js (Turbopack)14.118s14.828s0.710s51.05x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 50 sequential steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Express11.822s (-28.8% 🟢)12.022s (-29.4% 🟢)0.200s81.00x
💻 LocalNitro11.872s (-29.3% 🟢)12.024s (-29.4% 🟢)0.151s81.00x
🐘 PostgresExpress11.892s (-15.1% 🟢)12.017s (-17.7% 🟢)0.125s81.01x
🐘 PostgresNitro12.598s (-9.8% 🟢)13.025s (-9.0% 🟢)0.427s71.07x
💻 LocalNext.js (Turbopack)13.002s13.309s0.308s71.10x
🐘 PostgresNext.js (Turbopack)⚠️missing----
Promise.all with 10 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.135s (-9.9% 🟢)2.007s (~)0.872s151.00x
💻 LocalExpress1.163s (-21.9% 🟢)2.006s (~)0.843s151.02x
💻 LocalNitro1.170s (-28.3% 🟢)2.006s (-3.3%)0.836s151.03x
🐘 PostgresNitro1.197s (-6.1% 🟢)2.010s (~)0.814s151.05x
💻 LocalNext.js (Turbopack)1.254s2.006s0.751s151.10x
🐘 PostgresNext.js (Turbopack)⚠️missing----
Promise.all with 25 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.187s (-49.7% 🟢)2.006s (-33.3% 🟢)0.819s151.00x
🐘 PostgresNitro1.253s (-46.7% 🟢)2.010s (-33.2% 🟢)0.757s151.06x
💻 LocalExpress1.652s (-44.1% 🟢)2.005s (-41.9% 🟢)0.354s151.39x
💻 LocalNext.js (Turbopack)1.682s2.006s0.324s151.42x
💻 LocalNitro1.683s (-46.5% 🟢)2.006s (-48.4% 🟢)0.322s151.42x
🐘 PostgresNext.js (Turbopack)⚠️missing----
Promise.all with 50 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.304s (-62.6% 🟢)2.007s (-50.0% 🟢)0.703s151.00x
🐘 PostgresNitro1.390s (-60.1% 🟢)2.010s (-49.9% 🟢)0.620s151.07x
💻 LocalNitro4.483s (-46.3% 🟢)5.012s (-44.4% 🟢)0.529s63.44x
💻 LocalExpress4.632s (-44.5% 🟢)5.178s (-42.6% 🟢)0.546s63.55x
💻 LocalNext.js (Turbopack)4.684s5.179s0.495s63.59x
🐘 PostgresNext.js (Turbopack)⚠️missing----
Promise.race with 10 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.139s (-9.4% 🟢)2.008s (~)0.869s151.00x
🐘 PostgresNitro1.211s (-3.7%)2.011s (~)0.800s151.06x
💻 LocalNext.js (Turbopack)1.317s2.006s0.689s151.16x
💻 LocalExpress1.364s (-28.0% 🟢)2.006s (-15.1% 🟢)0.642s151.20x
💻 LocalNitro1.372s (-26.5% 🟢)2.006s (-14.3% 🟢)0.634s151.20x
🐘 PostgresNext.js (Turbopack)⚠️missing----
Promise.race with 25 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.195s (-49.0% 🟢)2.009s (-33.3% 🟢)0.814s151.00x
🐘 PostgresNitro1.274s (-45.5% 🟢)2.013s (-33.1% 🟢)0.738s151.07x
💻 LocalExpress1.833s (-41.5% 🟢)2.222s (-41.0% 🟢)0.389s141.53x
💻 LocalNitro1.916s (-37.5% 🟢)2.316s (-40.4% 🟢)0.400s131.60x
💻 LocalNext.js (Turbopack)1.927s2.393s0.466s131.61x
🐘 PostgresNext.js (Turbopack)⚠️missing----
Promise.race with 50 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.290s (-63.1% 🟢)2.008s (-49.9% 🟢)0.718s151.00x
🐘 PostgresNitro1.430s (-58.9% 🟢)2.014s (-49.8% 🟢)0.584s151.11x
💻 LocalExpress4.559s (-48.2% 🟢)5.346s (-42.3% 🟢)0.787s63.53x
💻 LocalNext.js (Turbopack)4.916s5.352s0.436s63.81x
💻 LocalNitro5.467s (-40.2% 🟢)6.013s (-40.0% 🟢)0.547s54.24x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 10 sequential data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.435s (-48.2% 🟢)1.006s (-1.7%)0.572s601.00x
💻 LocalNitro0.454s (-53.7% 🟢)1.004s (-8.2% 🟢)0.549s601.05x
💻 LocalExpress0.468s (-52.4% 🟢)1.004s (-6.7% 🟢)0.535s601.08x
🐘 PostgresNitro0.568s (-30.8% 🟢)1.008s (~)0.441s601.31x
💻 LocalNext.js (Turbopack)0.720s1.004s0.284s601.66x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 25 sequential data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.004s (-49.2% 🟢)1.292s (-42.8% 🟢)0.288s701.00x
💻 LocalExpress1.152s (-61.8% 🟢)2.006s (-44.1% 🟢)0.854s451.15x
💻 LocalNitro1.162s (-61.7% 🟢)2.006s (-46.6% 🟢)0.844s451.16x
🐘 PostgresNitro1.385s (-28.1% 🟢)2.033s (-3.2%)0.648s451.38x
💻 LocalNext.js (Turbopack)1.807s2.051s0.244s441.80x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 50 sequential data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.965s (-50.8% 🟢)2.334s (-46.6% 🟢)0.369s521.00x
🐘 PostgresNitro2.560s (-37.6% 🟢)3.035s (-34.1% 🟢)0.475s401.30x
💻 LocalExpress2.619s (-71.6% 🟢)3.007s (-70.0% 🟢)0.388s401.33x
💻 LocalNitro2.637s (-71.6% 🟢)3.008s (-70.0% 🟢)0.370s401.34x
💻 LocalNext.js (Turbopack)3.859s4.074s0.216s301.96x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 10 concurrent data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.186s (-34.0% 🟢)1.022s (+1.5%)0.836s591.00x
🐘 PostgresNitro0.218s (-23.1% 🟢)1.007s (~)0.789s601.17x
💻 LocalExpress0.375s (-33.1% 🟢)1.004s (~)0.629s602.01x
💻 LocalNitro0.387s (-36.0% 🟢)1.004s (-1.7%)0.617s602.07x
💻 LocalNext.js (Turbopack)0.536s1.004s0.468s602.88x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 25 concurrent data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.277s (-45.7% 🟢)1.006s (~)0.729s901.00x
🐘 PostgresNitro0.356s (-28.4% 🟢)1.008s (~)0.652s901.28x
💻 LocalExpress2.068s (-17.7% 🟢)2.580s (-14.3% 🟢)0.512s357.46x
💻 LocalNitro2.099s (-17.3% 🟢)2.655s (-11.8% 🟢)0.556s347.58x
💻 LocalNext.js (Turbopack)2.335s2.976s0.641s318.43x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 50 concurrent data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.603s (-26.4% 🟢)1.006s (-1.1%)0.403s1201.00x
🐘 PostgresNitro0.723s (-8.6% 🟢)1.008s (~)0.285s1201.20x
💻 LocalExpress9.546s (-14.7% 🟢)10.111s (-15.3% 🟢)0.565s1215.84x
💻 LocalNitro9.729s (-13.1% 🟢)10.444s (-10.5% 🟢)0.715s1216.14x
💻 LocalNext.js (Turbopack)10.871s11.575s0.705s1118.04x
🐘 PostgresNext.js (Turbopack)⚠️missing----
Stream Benchmarks(includes TTFB metrics)
workflow with stream

💻 Local Development

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Express1.127s (+466.0% 🔺)2.005s (+99.6% 🔺)0.010s (-17.4% 🟢)2.017s (+98.1% 🔺)0.890s101.00x
🐘 PostgresExpress1.131s (+451.7% 🔺)1.995s (+99.8% 🔺)0.001s (-37.5% 🟢)2.010s (+98.7% 🔺)0.878s101.00x
💻 LocalNitro1.143s (+434.9% 🔺)2.005s (+99.6% 🔺)0.011s (-12.8% 🟢)2.018s (+98.1% 🔺)0.875s101.01x
🐘 PostgresNitro1.173s (+472.0% 🔺)1.998s (+99.9% 🔺)0.002s (+20.0% 🔺)2.013s (+99.1% 🔺)0.841s101.04x
💻 LocalNext.js (Turbopack)1.174s2.003s0.010s2.017s0.843s101.04x
🐘 PostgresNext.js (Turbopack)⚠️missing-----
stream pipeline with 5 transform steps (1MB)

💻 Local Development

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.498s (+137.7% 🔺)1.999s (+98.6% 🔺)0.004s (+5.3% 🔺)2.023s (+97.8% 🔺)0.526s301.00x
💻 LocalNext.js (Turbopack)1.669s2.009s0.011s2.022s0.354s301.11x
💻 LocalNitro1.691s (+101.6% 🔺)2.010s (+98.6% 🔺)0.009s (-1.3%)2.200s (+97.2% 🔺)0.509s281.13x
🐘 PostgresNitro1.701s (+172.5% 🔺)2.008s (+99.5% 🔺)0.005s (+11.2% 🔺)2.032s (+98.7% 🔺)0.331s301.14x
💻 LocalExpress1.909s (+152.2% 🔺)2.011s (+95.4% 🔺)0.009s (-0.6%)2.423s (+133.0% 🔺)0.514s251.27x
🐘 PostgresNext.js (Turbopack)⚠️missing-----
10 parallel streams (1MB each)

💻 Local Development

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.662s (-31.1% 🟢)1.048s (-18.0% 🟢)0.000s (+21.1% 🔺)1.059s (-18.9% 🟢)0.397s571.00x
🐘 PostgresNitro0.795s (-17.9% 🟢)1.049s (-16.0% 🟢)0.000s (+71.4% 🔺)1.082s (-13.9% 🟢)0.287s561.20x
💻 LocalExpress1.321s (+7.9% 🔺)2.015s (~)0.000s (-60.0% 🟢)2.017s (~)0.696s302.00x
💻 LocalNitro1.343s (+9.9% 🔺)2.015s (~)0.000s (+200.0% 🔺)2.017s (~)0.674s302.03x
💻 LocalNext.js (Turbopack)1.480s2.014s0.000s2.017s0.537s302.24x
🐘 PostgresNext.js (Turbopack)⚠️missing-----
fan-out fan-in 10 streams (1MB each)

💻 Local Development

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.258s (-29.0% 🟢)1.898s (-12.8% 🟢)0.000s (+Infinity% 🔺)1.931s (-12.2% 🟢)0.673s321.00x
🐘 PostgresNitro1.634s (-8.8% 🟢)2.263s (+5.7% 🔺)0.000s (+7.7% 🔺)2.309s (+6.2% 🔺)0.675s261.30x
💻 LocalNext.js (Turbopack)2.815s3.293s0.001s3.298s0.483s192.24x
💻 LocalExpress3.025s (-12.8% 🟢)3.556s (-11.8% 🟢)0.000s (-55.9% 🟢)3.562s (-11.8% 🟢)0.536s172.41x
💻 LocalNitro3.082s (-9.0% 🟢)3.778s (-6.3% 🟢)0.000s (-18.0% 🟢)3.782s (-6.3% 🟢)0.700s162.45x
🐘 PostgresNext.js (Turbopack)⚠️missing-----

Summary

Fastest Framework by World

Winner determined by most benchmark wins

World🥇 Fastest FrameworkWins
💻 LocalExpress16/21
🐘 PostgresExpress21/21
Fastest World by Framework

Winner determined by most benchmark wins

Framework🥇 Fastest WorldWins
Express🐘 Postgres15/21
Next.js (Turbopack)💻 Local21/21
Nitro🐘 Postgres11/21
Column Definitions
  • Workflow Time: Runtime reported by workflow (completedAt - createdAt) - primary metric
  • TTFB: Time to First Byte - time from workflow start until first stream byte received (stream benchmarks only)
  • Slurp: Time from first byte to complete stream consumption (stream benchmarks only)
  • Wall Time: Total testbench time (trigger workflow + poll for result)
  • Overhead: Testbench overhead (Wall Time - Workflow Time)
  • Samples: Number of benchmark iterations run
  • vs Fastest: How much slower compared to the fastest configuration for this benchmark

Worlds:

  • 💻 Local: In-memory filesystem world (local development)
  • 🐘 Postgres: PostgreSQL database world (local development)
  • ▲ Vercel: Vercel production/preview deployment
  • 🌐 Turso: Community world (local development)
  • 🌐 MongoDB: Community world (local development)
  • 🌐 Redis: Community world (local development)
  • 🌐 Jazz: Community world (local development)
  • 🌐 Redis: Community world (local development)
  • 🌐 Redis + BullMQ: Community world (local development)
  • 🌐 Cloudflare: Community world (local development)
  • 🌐 MySQL: Community world (local development)
  • 🌐 Azure: Community world (local development)
  • 🌐 NATS JetStream: Community world (local development)
  • 🌐 Upstash: Community world (local development)

📋 View full workflow run


Some benchmark jobs failed:

  • Local: success
  • Postgres: success
  • Vercel: failure

Check the workflow run for details.

⚠️Community world benchmarks failed (non-blocking):

  • Community Worlds: failure

Check the workflow run for details.

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes a race in @workflow/world-testing’s inline-execution test server where the per-run flow invocation counter could be incremented after the workflow completion became observable, causing flaky assertions in CI.

Changes:

  • Reorders the flow handler to increment the per-run invocation counter before awaiting/returning flowPOST.
  • Adds an explanatory comment documenting the race and why the ordering matters.
  • Adds a changeset bumping @workflow/world-testing with a patch release note.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

FileDescription
packages/world-testing/src/server.mtsMoves flow invocation counting ahead of the flow handler execution to eliminate the observed flake window.
.changeset/fix-inline-execution-flow-count-race.mdAdds a patch changeset describing the test-flake fix for @workflow/world-testing.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@TooTallNate
TooTallNate disabled auto-merge May 20, 2026 23:05
@TooTallNate
TooTallNate merged commit 96e2d3c into mainMay 20, 2026
108 of 118 checks passed
@TooTallNate
TooTallNate deleted the nate/fix-flow-invocation-count-race branch May 20, 2026 23:05
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@TooTallNate@karthikscale3
, 'i'); if (__m === '*' || __re.test(location.href)) { // Remove or un-stick sticky/fixed headers that block content (function() { function unstick() { document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) { if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') { el.style.position = 'static'; el.style.top = 'auto'; el.style.zIndex = 'auto'; } }); } unstick(); var observer = new MutationObserver(unstick); observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] }); })(); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' [world-testing] fix flow invocation counter race in inline-execution tests by TooTallNate · Pull Request #2043 · vercel/workflow · GitHub
Skip to content

[world-testing] fix flow invocation counter race in inline-execution tests - #2043

Merged
TooTallNate merged 2 commits into
mainfrom
nate/fix-flow-invocation-count-race
May 20, 2026
Merged

[world-testing] fix flow invocation counter race in inline-execution tests#2043
TooTallNate merged 2 commits into
mainfrom
nate/fix-flow-invocation-count-race

Conversation

@TooTallNate

Copy link
Copy Markdown
Member

Summary

Fixes the flaky unit test failure seen on multiple PRs (e.g. PR #2013, PR #2035):

FAIL test/spec.test.ts > sequential steps complete in a single flow invocation
AssertionError: expected +0 to be 1 // Object.is equality
❯ ../world-testing/src/inline-execution.mts:45:21

Root cause

The test server's flow handler awaited flowPOST before incrementing the per-run invocation counter:

constcloned=ctx.req.raw.clone();constresponse=awaitflowPOST(ctx.req.raw);// (1) workflow may complete & flush DB here// (2) read cloned body, increment counterreturnresponse;

That races with the test, which polls getRun() until it sees status === 'completed' and then immediately queries /_flow-invocations/:runId:

  1. Inside flowPOST, the workflow runs and writes the completed run to the DB.
  2. The test's vi.waitFor poll sees status === 'completed'.
  3. The test fetches the flow invocation count.
  4. The counter increment in the handler hadn't run yet → count returns 0 instead of 1.

Locally the race rarely loses (the cloned-body read is fast) but on slower CI runners it loses often enough to be visible across many PRs.

Fix

Increment the counter before awaiting flowPOST so the count is observable as soon as the run transitions to completed. Same logic, just reordered.

Test plan

  • Re-ran packages/world-testing/test/embedded.test.ts locally — all 9 tests pass, including the four inline-execution.mts cases that drive the counter.
  • Could not reproduce the race locally (Docker on this box is broken so postgres test was skipped), but the fix removes the only await between request entry and counter increment, so there is no longer a window where the run can be flushed to the DB before the counter is bumped.

…tests
The test server's flow handler awaited flowPOST before incrementing the
invocation counter, which races with the test polling getRun() to see
the completed status. When the workflow completed inside flowPOST and
flushed the run to the DB, the test could observe the completed state
and immediately query /_flow-invocations before the counter was bumped,
yielding a flaky 'expected 0 to be 1' assertion.
Increment the counter before awaiting flowPOST so the count is
observable as soon as the run transitions to completed.
CopilotAI review requested due to automatic review settings May 20, 2026 21:13
@TooTallNate
TooTallNate requested a review from a team as a code ownerMay 20, 2026 21:13
@changeset-bot

changeset-botBot commented May 20, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: c11f633

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
NameType
@workflow/world-testingPatch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel

vercelBot commented May 20, 2026

Copy link
Copy Markdown
Contributor

@github-actions

github-actionsBot commented May 20, 2026

Copy link
Copy Markdown
Contributor

🧪 E2E Test Results

Some tests failed

Summary

PassedFailedSkippedTotal
❌ ▲ Vercel Production119822191419
✅ 💻 Local Development158702191806
✅ 📦 Local Production158702191806
✅ 🐘 Local Postgres158702191806
✅ 🪟 Windows12900129
❌ 📋 Other7261176903
Total6814310527869

❌ Failed Tests

▲ Vercel Production (2 failed)

astro (2 failed):

  • outputStreamInsideStepWorkflow - getWritable() called inside step functions | wrun_01KS3RNPMVVB6C8PR0168PSSWT | 🔍 observability
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
📋 Other (1 failed)

e2e-vercel-prod-tanstack-start (1 failed):

  • distributedAbortController - manual abort triggers signal | wrun_01KS3S811F65HFD53TRWAE1NP6

Details by Category

❌ ▲ Vercel Production
AppPassedFailedSkipped
❌ astro101226
✅ example103026
✅ express103026
✅ fastify103026
✅ hono103026
✅ nextjs-turbopack12702
✅ nextjs-webpack12702
✅ nitro103026
✅ nuxt103026
✅ sveltekit12207
✅ vite103026
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable104025
✅ express-stable104025
✅ fastify-stable104025
✅ hono-stable104025
✅ nextjs-turbopack-canary110019
✅ nextjs-turbopack-stable-lazy-discovery-disabled12900
✅ nextjs-turbopack-stable-lazy-discovery-enabled12900
✅ nextjs-webpack-canary110019
✅ nextjs-webpack-stable-lazy-discovery-disabled12900
✅ nextjs-webpack-stable-lazy-discovery-enabled12900
✅ nitro-stable104025
✅ nuxt-stable104025
✅ sveltekit-stable12306
✅ vite-stable104025
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable104025
✅ express-stable104025
✅ fastify-stable104025
✅ hono-stable104025
✅ nextjs-turbopack-canary110019
✅ nextjs-turbopack-stable-lazy-discovery-disabled12900
✅ nextjs-turbopack-stable-lazy-discovery-enabled12900
✅ nextjs-webpack-canary110019
✅ nextjs-webpack-stable-lazy-discovery-disabled12900
✅ nextjs-webpack-stable-lazy-discovery-enabled12900
✅ nitro-stable104025
✅ nuxt-stable104025
✅ sveltekit-stable12306
✅ vite-stable104025
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable104025
✅ express-stable104025
✅ fastify-stable104025
✅ hono-stable104025
✅ nextjs-turbopack-canary110019
✅ nextjs-turbopack-stable-lazy-discovery-disabled12900
✅ nextjs-turbopack-stable-lazy-discovery-enabled12900
✅ nextjs-webpack-canary110019
✅ nextjs-webpack-stable-lazy-discovery-disabled12900
✅ nextjs-webpack-stable-lazy-discovery-enabled12900
✅ nitro-stable104025
✅ nuxt-stable104025
✅ sveltekit-stable12306
✅ vite-stable104025
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack12900
❌ 📋 Other
AppPassedFailedSkipped
✅ e2e-local-dev-nest-stable104025
✅ e2e-local-dev-tanstack-start-104025
✅ e2e-local-postgres-nest-stable104025
✅ e2e-local-postgres-tanstack-start-104025
✅ e2e-local-prod-nest-stable104025
✅ e2e-local-prod-tanstack-start-104025
❌ e2e-vercel-prod-tanstack-start102126

📋 View full workflow run


Some E2E test jobs failed:

  • Vercel Prod: failure
  • Local Dev: success
  • Local Prod: success
  • Local Postgres: success
  • Windows: success

Check the workflow run for details.

@github-actions

github-actionsBot commented May 20, 2026

Copy link
Copy Markdown
Contributor

📊 Benchmark Results

📈 Comparing against baseline from main branch. Green 🟢 = faster, Red 🔺 = slower.

workflow with no steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Express0.029s (-34.5% 🟢)1.005s (~)0.976s101.00x
💻 LocalNitro0.030s (-31.3% 🟢)1.005s (~)0.976s101.02x
💻 LocalNext.js (Turbopack)0.048s1.006s0.958s101.64x
🐘 PostgresExpress0.049s (-16.2% 🟢)1.011s (~)0.963s101.68x
🐘 PostgresNitro0.061s (-36.2% 🟢)1.013s (-2.8%)0.953s102.09x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 1 step

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Express1.067s (-5.2% 🟢)2.005s (~)0.939s101.00x
💻 LocalNitro1.072s (-5.2% 🟢)2.006s (~)0.934s101.00x
🐘 PostgresExpress1.084s (-5.4% 🟢)2.010s (~)0.926s101.02x
🐘 PostgresNitro1.103s (-3.3%)2.011s (~)0.908s101.03x
💻 LocalNext.js (Turbopack)1.112s2.005s0.894s101.04x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 10 sequential steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Express10.384s (-4.9%)11.021s (~)0.636s31.00x
🐘 PostgresExpress10.401s (-5.1% 🟢)11.014s (~)0.613s31.00x
💻 LocalNitro10.408s (-4.9%)11.023s (~)0.615s31.00x
🐘 PostgresNitro10.513s (-3.3%)11.017s (~)0.503s31.01x
💻 LocalNext.js (Turbopack)10.654s11.022s0.368s31.03x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 25 sequential steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Express13.401s (-10.5% 🟢)14.026s (-6.7% 🟢)0.626s51.00x
🐘 PostgresExpress13.406s (-8.1% 🟢)14.016s (-6.7% 🟢)0.610s51.00x
💻 LocalNitro13.447s (-10.7% 🟢)14.027s (-12.5% 🟢)0.581s51.00x
🐘 PostgresNitro13.729s (-5.9% 🟢)14.023s (-6.7% 🟢)0.293s51.02x
💻 LocalNext.js (Turbopack)14.118s14.828s0.710s51.05x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 50 sequential steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Express11.822s (-28.8% 🟢)12.022s (-29.4% 🟢)0.200s81.00x
💻 LocalNitro11.872s (-29.3% 🟢)12.024s (-29.4% 🟢)0.151s81.00x
🐘 PostgresExpress11.892s (-15.1% 🟢)12.017s (-17.7% 🟢)0.125s81.01x
🐘 PostgresNitro12.598s (-9.8% 🟢)13.025s (-9.0% 🟢)0.427s71.07x
💻 LocalNext.js (Turbopack)13.002s13.309s0.308s71.10x
🐘 PostgresNext.js (Turbopack)⚠️missing----
Promise.all with 10 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.135s (-9.9% 🟢)2.007s (~)0.872s151.00x
💻 LocalExpress1.163s (-21.9% 🟢)2.006s (~)0.843s151.02x
💻 LocalNitro1.170s (-28.3% 🟢)2.006s (-3.3%)0.836s151.03x
🐘 PostgresNitro1.197s (-6.1% 🟢)2.010s (~)0.814s151.05x
💻 LocalNext.js (Turbopack)1.254s2.006s0.751s151.10x
🐘 PostgresNext.js (Turbopack)⚠️missing----
Promise.all with 25 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.187s (-49.7% 🟢)2.006s (-33.3% 🟢)0.819s151.00x
🐘 PostgresNitro1.253s (-46.7% 🟢)2.010s (-33.2% 🟢)0.757s151.06x
💻 LocalExpress1.652s (-44.1% 🟢)2.005s (-41.9% 🟢)0.354s151.39x
💻 LocalNext.js (Turbopack)1.682s2.006s0.324s151.42x
💻 LocalNitro1.683s (-46.5% 🟢)2.006s (-48.4% 🟢)0.322s151.42x
🐘 PostgresNext.js (Turbopack)⚠️missing----
Promise.all with 50 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.304s (-62.6% 🟢)2.007s (-50.0% 🟢)0.703s151.00x
🐘 PostgresNitro1.390s (-60.1% 🟢)2.010s (-49.9% 🟢)0.620s151.07x
💻 LocalNitro4.483s (-46.3% 🟢)5.012s (-44.4% 🟢)0.529s63.44x
💻 LocalExpress4.632s (-44.5% 🟢)5.178s (-42.6% 🟢)0.546s63.55x
💻 LocalNext.js (Turbopack)4.684s5.179s0.495s63.59x
🐘 PostgresNext.js (Turbopack)⚠️missing----
Promise.race with 10 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.139s (-9.4% 🟢)2.008s (~)0.869s151.00x
🐘 PostgresNitro1.211s (-3.7%)2.011s (~)0.800s151.06x
💻 LocalNext.js (Turbopack)1.317s2.006s0.689s151.16x
💻 LocalExpress1.364s (-28.0% 🟢)2.006s (-15.1% 🟢)0.642s151.20x
💻 LocalNitro1.372s (-26.5% 🟢)2.006s (-14.3% 🟢)0.634s151.20x
🐘 PostgresNext.js (Turbopack)⚠️missing----
Promise.race with 25 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.195s (-49.0% 🟢)2.009s (-33.3% 🟢)0.814s151.00x
🐘 PostgresNitro1.274s (-45.5% 🟢)2.013s (-33.1% 🟢)0.738s151.07x
💻 LocalExpress1.833s (-41.5% 🟢)2.222s (-41.0% 🟢)0.389s141.53x
💻 LocalNitro1.916s (-37.5% 🟢)2.316s (-40.4% 🟢)0.400s131.60x
💻 LocalNext.js (Turbopack)1.927s2.393s0.466s131.61x
🐘 PostgresNext.js (Turbopack)⚠️missing----
Promise.race with 50 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.290s (-63.1% 🟢)2.008s (-49.9% 🟢)0.718s151.00x
🐘 PostgresNitro1.430s (-58.9% 🟢)2.014s (-49.8% 🟢)0.584s151.11x
💻 LocalExpress4.559s (-48.2% 🟢)5.346s (-42.3% 🟢)0.787s63.53x
💻 LocalNext.js (Turbopack)4.916s5.352s0.436s63.81x
💻 LocalNitro5.467s (-40.2% 🟢)6.013s (-40.0% 🟢)0.547s54.24x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 10 sequential data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.435s (-48.2% 🟢)1.006s (-1.7%)0.572s601.00x
💻 LocalNitro0.454s (-53.7% 🟢)1.004s (-8.2% 🟢)0.549s601.05x
💻 LocalExpress0.468s (-52.4% 🟢)1.004s (-6.7% 🟢)0.535s601.08x
🐘 PostgresNitro0.568s (-30.8% 🟢)1.008s (~)0.441s601.31x
💻 LocalNext.js (Turbopack)0.720s1.004s0.284s601.66x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 25 sequential data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.004s (-49.2% 🟢)1.292s (-42.8% 🟢)0.288s701.00x
💻 LocalExpress1.152s (-61.8% 🟢)2.006s (-44.1% 🟢)0.854s451.15x
💻 LocalNitro1.162s (-61.7% 🟢)2.006s (-46.6% 🟢)0.844s451.16x
🐘 PostgresNitro1.385s (-28.1% 🟢)2.033s (-3.2%)0.648s451.38x
💻 LocalNext.js (Turbopack)1.807s2.051s0.244s441.80x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 50 sequential data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.965s (-50.8% 🟢)2.334s (-46.6% 🟢)0.369s521.00x
🐘 PostgresNitro2.560s (-37.6% 🟢)3.035s (-34.1% 🟢)0.475s401.30x
💻 LocalExpress2.619s (-71.6% 🟢)3.007s (-70.0% 🟢)0.388s401.33x
💻 LocalNitro2.637s (-71.6% 🟢)3.008s (-70.0% 🟢)0.370s401.34x
💻 LocalNext.js (Turbopack)3.859s4.074s0.216s301.96x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 10 concurrent data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.186s (-34.0% 🟢)1.022s (+1.5%)0.836s591.00x
🐘 PostgresNitro0.218s (-23.1% 🟢)1.007s (~)0.789s601.17x
💻 LocalExpress0.375s (-33.1% 🟢)1.004s (~)0.629s602.01x
💻 LocalNitro0.387s (-36.0% 🟢)1.004s (-1.7%)0.617s602.07x
💻 LocalNext.js (Turbopack)0.536s1.004s0.468s602.88x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 25 concurrent data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.277s (-45.7% 🟢)1.006s (~)0.729s901.00x
🐘 PostgresNitro0.356s (-28.4% 🟢)1.008s (~)0.652s901.28x
💻 LocalExpress2.068s (-17.7% 🟢)2.580s (-14.3% 🟢)0.512s357.46x
💻 LocalNitro2.099s (-17.3% 🟢)2.655s (-11.8% 🟢)0.556s347.58x
💻 LocalNext.js (Turbopack)2.335s2.976s0.641s318.43x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 50 concurrent data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.603s (-26.4% 🟢)1.006s (-1.1%)0.403s1201.00x
🐘 PostgresNitro0.723s (-8.6% 🟢)1.008s (~)0.285s1201.20x
💻 LocalExpress9.546s (-14.7% 🟢)10.111s (-15.3% 🟢)0.565s1215.84x
💻 LocalNitro9.729s (-13.1% 🟢)10.444s (-10.5% 🟢)0.715s1216.14x
💻 LocalNext.js (Turbopack)10.871s11.575s0.705s1118.04x
🐘 PostgresNext.js (Turbopack)⚠️missing----
Stream Benchmarks(includes TTFB metrics)
workflow with stream

💻 Local Development

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Express1.127s (+466.0% 🔺)2.005s (+99.6% 🔺)0.010s (-17.4% 🟢)2.017s (+98.1% 🔺)0.890s101.00x
🐘 PostgresExpress1.131s (+451.7% 🔺)1.995s (+99.8% 🔺)0.001s (-37.5% 🟢)2.010s (+98.7% 🔺)0.878s101.00x
💻 LocalNitro1.143s (+434.9% 🔺)2.005s (+99.6% 🔺)0.011s (-12.8% 🟢)2.018s (+98.1% 🔺)0.875s101.01x
🐘 PostgresNitro1.173s (+472.0% 🔺)1.998s (+99.9% 🔺)0.002s (+20.0% 🔺)2.013s (+99.1% 🔺)0.841s101.04x
💻 LocalNext.js (Turbopack)1.174s2.003s0.010s2.017s0.843s101.04x
🐘 PostgresNext.js (Turbopack)⚠️missing-----
stream pipeline with 5 transform steps (1MB)

💻 Local Development

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.498s (+137.7% 🔺)1.999s (+98.6% 🔺)0.004s (+5.3% 🔺)2.023s (+97.8% 🔺)0.526s301.00x
💻 LocalNext.js (Turbopack)1.669s2.009s0.011s2.022s0.354s301.11x
💻 LocalNitro1.691s (+101.6% 🔺)2.010s (+98.6% 🔺)0.009s (-1.3%)2.200s (+97.2% 🔺)0.509s281.13x
🐘 PostgresNitro1.701s (+172.5% 🔺)2.008s (+99.5% 🔺)0.005s (+11.2% 🔺)2.032s (+98.7% 🔺)0.331s301.14x
💻 LocalExpress1.909s (+152.2% 🔺)2.011s (+95.4% 🔺)0.009s (-0.6%)2.423s (+133.0% 🔺)0.514s251.27x
🐘 PostgresNext.js (Turbopack)⚠️missing-----
10 parallel streams (1MB each)

💻 Local Development

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.662s (-31.1% 🟢)1.048s (-18.0% 🟢)0.000s (+21.1% 🔺)1.059s (-18.9% 🟢)0.397s571.00x
🐘 PostgresNitro0.795s (-17.9% 🟢)1.049s (-16.0% 🟢)0.000s (+71.4% 🔺)1.082s (-13.9% 🟢)0.287s561.20x
💻 LocalExpress1.321s (+7.9% 🔺)2.015s (~)0.000s (-60.0% 🟢)2.017s (~)0.696s302.00x
💻 LocalNitro1.343s (+9.9% 🔺)2.015s (~)0.000s (+200.0% 🔺)2.017s (~)0.674s302.03x
💻 LocalNext.js (Turbopack)1.480s2.014s0.000s2.017s0.537s302.24x
🐘 PostgresNext.js (Turbopack)⚠️missing-----
fan-out fan-in 10 streams (1MB each)

💻 Local Development

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.258s (-29.0% 🟢)1.898s (-12.8% 🟢)0.000s (+Infinity% 🔺)1.931s (-12.2% 🟢)0.673s321.00x
🐘 PostgresNitro1.634s (-8.8% 🟢)2.263s (+5.7% 🔺)0.000s (+7.7% 🔺)2.309s (+6.2% 🔺)0.675s261.30x
💻 LocalNext.js (Turbopack)2.815s3.293s0.001s3.298s0.483s192.24x
💻 LocalExpress3.025s (-12.8% 🟢)3.556s (-11.8% 🟢)0.000s (-55.9% 🟢)3.562s (-11.8% 🟢)0.536s172.41x
💻 LocalNitro3.082s (-9.0% 🟢)3.778s (-6.3% 🟢)0.000s (-18.0% 🟢)3.782s (-6.3% 🟢)0.700s162.45x
🐘 PostgresNext.js (Turbopack)⚠️missing-----

Summary

Fastest Framework by World

Winner determined by most benchmark wins

World🥇 Fastest FrameworkWins
💻 LocalExpress16/21
🐘 PostgresExpress21/21
Fastest World by Framework

Winner determined by most benchmark wins

Framework🥇 Fastest WorldWins
Express🐘 Postgres15/21
Next.js (Turbopack)💻 Local21/21
Nitro🐘 Postgres11/21
Column Definitions
  • Workflow Time: Runtime reported by workflow (completedAt - createdAt) - primary metric
  • TTFB: Time to First Byte - time from workflow start until first stream byte received (stream benchmarks only)
  • Slurp: Time from first byte to complete stream consumption (stream benchmarks only)
  • Wall Time: Total testbench time (trigger workflow + poll for result)
  • Overhead: Testbench overhead (Wall Time - Workflow Time)
  • Samples: Number of benchmark iterations run
  • vs Fastest: How much slower compared to the fastest configuration for this benchmark

Worlds:

  • 💻 Local: In-memory filesystem world (local development)
  • 🐘 Postgres: PostgreSQL database world (local development)
  • ▲ Vercel: Vercel production/preview deployment
  • 🌐 Turso: Community world (local development)
  • 🌐 MongoDB: Community world (local development)
  • 🌐 Redis: Community world (local development)
  • 🌐 Jazz: Community world (local development)
  • 🌐 Redis: Community world (local development)
  • 🌐 Redis + BullMQ: Community world (local development)
  • 🌐 Cloudflare: Community world (local development)
  • 🌐 MySQL: Community world (local development)
  • 🌐 Azure: Community world (local development)
  • 🌐 NATS JetStream: Community world (local development)
  • 🌐 Upstash: Community world (local development)

📋 View full workflow run


Some benchmark jobs failed:

  • Local: success
  • Postgres: success
  • Vercel: failure

Check the workflow run for details.

⚠️Community world benchmarks failed (non-blocking):

  • Community Worlds: failure

Check the workflow run for details.

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes a race in @workflow/world-testing’s inline-execution test server where the per-run flow invocation counter could be incremented after the workflow completion became observable, causing flaky assertions in CI.

Changes:

  • Reorders the flow handler to increment the per-run invocation counter before awaiting/returning flowPOST.
  • Adds an explanatory comment documenting the race and why the ordering matters.
  • Adds a changeset bumping @workflow/world-testing with a patch release note.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

FileDescription
packages/world-testing/src/server.mtsMoves flow invocation counting ahead of the flow handler execution to eliminate the observed flake window.
.changeset/fix-inline-execution-flow-count-race.mdAdds a patch changeset describing the test-flake fix for @workflow/world-testing.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@TooTallNate
TooTallNate disabled auto-merge May 20, 2026 23:05
@TooTallNate
TooTallNate merged commit 96e2d3c into mainMay 20, 2026
108 of 118 checks passed
@TooTallNate
TooTallNate deleted the nate/fix-flow-invocation-count-race branch May 20, 2026 23:05
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@TooTallNate@karthikscale3
, 'i'); if (__m === '*' || __re.test(location.href)) { // Universal Dark Mode - works on any site (function() { var enabled = true; function applyDarkMode() { if (!enabled) return; // Create style element if it doesn't exist var style = document.getElementById('universal-dark-mode-style'); if (!style) { style = document.createElement('style'); style.id = 'universal-dark-mode-style'; document.head.appendChild(style); } // Dark mode CSS - inverts colors but preserves images/video style.textContent = ' /* Invert everything except media */ html { filter: invert(1) hue-rotate(180deg) !important; background: #1a1a2e !important; } /* Restore images, videos, iframes, canvas */ img, video, iframe, canvas, svg, picture, [style*="background-image"] { filter: invert(1) hue-rotate(180deg) !important; } /* Preserve specific elements that should not be inverted */ .no-dark-mode, .no-dark-mode *, [data-theme="light"], [data-theme="light"], .ace_editor, .ace_editor *, .CodeMirror, .CodeMirror *, .monaco-editor, .monaco-editor *, .markdown-body pre, .markdown-body pre *, .highlight, .highlight *, pre code, pre code * { filter: none !important; } /* Fix common UI elements */ .modal, .popup, .dropdown-menu, .tooltip, .popover { filter: invert(1) hue-rotate(180deg) !important; background: #2d2d44 !important; border-color: #444 !important; } /* Scrollbars */ ::-webkit-scrollbar { background: #1a1a2e !important; } ::-webkit-scrollbar-thumb { background: #444 !important; } ::-webkit-scrollbar-thumb:hover { background: #555 !important; } /* Selection */ ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; } ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; } '; } function removeDarkMode() { var style = document.getElementById('universal-dark-mode-style'); if (style) style.remove(); } // Toggle with Alt+Shift+D document.addEventListener('keydown', function(e) { if (e.altKey && e.shiftKey && e.key === 'D') { e.preventDefault(); enabled = !enabled; if (enabled) { applyDarkMode(); console.log('[Universal Dark Mode] Enabled'); } else { removeDarkMode(); console.log('[Universal Dark Mode] Disabled'); } } }); // Apply on load applyDarkMode(); // Re-apply on dynamic content var observer = new MutationObserver(function(mutations) { if (enabled && !document.getElementById('universal-dark-mode-style')) { applyDarkMode(); } }); observer.observe(document.head, { childList: true }); console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle'); })(); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })(); [world-testing] fix flow invocation counter race in inline-execution tests by TooTallNate · Pull Request #2043 · vercel/workflow · GitHub
Skip to content

[world-testing] fix flow invocation counter race in inline-execution tests - #2043

Merged
TooTallNate merged 2 commits into
mainfrom
nate/fix-flow-invocation-count-race
May 20, 2026
Merged

[world-testing] fix flow invocation counter race in inline-execution tests#2043
TooTallNate merged 2 commits into
mainfrom
nate/fix-flow-invocation-count-race

Conversation

@TooTallNate

Copy link
Copy Markdown
Member

Summary

Fixes the flaky unit test failure seen on multiple PRs (e.g. PR #2013, PR #2035):

FAIL test/spec.test.ts > sequential steps complete in a single flow invocation
AssertionError: expected +0 to be 1 // Object.is equality
❯ ../world-testing/src/inline-execution.mts:45:21

Root cause

The test server's flow handler awaited flowPOST before incrementing the per-run invocation counter:

constcloned=ctx.req.raw.clone();constresponse=awaitflowPOST(ctx.req.raw);// (1) workflow may complete & flush DB here// (2) read cloned body, increment counterreturnresponse;

That races with the test, which polls getRun() until it sees status === 'completed' and then immediately queries /_flow-invocations/:runId:

  1. Inside flowPOST, the workflow runs and writes the completed run to the DB.
  2. The test's vi.waitFor poll sees status === 'completed'.
  3. The test fetches the flow invocation count.
  4. The counter increment in the handler hadn't run yet → count returns 0 instead of 1.

Locally the race rarely loses (the cloned-body read is fast) but on slower CI runners it loses often enough to be visible across many PRs.

Fix

Increment the counter before awaiting flowPOST so the count is observable as soon as the run transitions to completed. Same logic, just reordered.

Test plan

  • Re-ran packages/world-testing/test/embedded.test.ts locally — all 9 tests pass, including the four inline-execution.mts cases that drive the counter.
  • Could not reproduce the race locally (Docker on this box is broken so postgres test was skipped), but the fix removes the only await between request entry and counter increment, so there is no longer a window where the run can be flushed to the DB before the counter is bumped.

…tests
The test server's flow handler awaited flowPOST before incrementing the
invocation counter, which races with the test polling getRun() to see
the completed status. When the workflow completed inside flowPOST and
flushed the run to the DB, the test could observe the completed state
and immediately query /_flow-invocations before the counter was bumped,
yielding a flaky 'expected 0 to be 1' assertion.
Increment the counter before awaiting flowPOST so the count is
observable as soon as the run transitions to completed.
CopilotAI review requested due to automatic review settings May 20, 2026 21:13
@TooTallNate
TooTallNate requested a review from a team as a code ownerMay 20, 2026 21:13
@changeset-bot

changeset-botBot commented May 20, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: c11f633

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
NameType
@workflow/world-testingPatch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel

vercelBot commented May 20, 2026

Copy link
Copy Markdown
Contributor

@github-actions

github-actionsBot commented May 20, 2026

Copy link
Copy Markdown
Contributor

🧪 E2E Test Results

Some tests failed

Summary

PassedFailedSkippedTotal
❌ ▲ Vercel Production119822191419
✅ 💻 Local Development158702191806
✅ 📦 Local Production158702191806
✅ 🐘 Local Postgres158702191806
✅ 🪟 Windows12900129
❌ 📋 Other7261176903
Total6814310527869

❌ Failed Tests

▲ Vercel Production (2 failed)

astro (2 failed):

  • outputStreamInsideStepWorkflow - getWritable() called inside step functions | wrun_01KS3RNPMVVB6C8PR0168PSSWT | 🔍 observability
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
📋 Other (1 failed)

e2e-vercel-prod-tanstack-start (1 failed):

  • distributedAbortController - manual abort triggers signal | wrun_01KS3S811F65HFD53TRWAE1NP6

Details by Category

❌ ▲ Vercel Production
AppPassedFailedSkipped
❌ astro101226
✅ example103026
✅ express103026
✅ fastify103026
✅ hono103026
✅ nextjs-turbopack12702
✅ nextjs-webpack12702
✅ nitro103026
✅ nuxt103026
✅ sveltekit12207
✅ vite103026
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable104025
✅ express-stable104025
✅ fastify-stable104025
✅ hono-stable104025
✅ nextjs-turbopack-canary110019
✅ nextjs-turbopack-stable-lazy-discovery-disabled12900
✅ nextjs-turbopack-stable-lazy-discovery-enabled12900
✅ nextjs-webpack-canary110019
✅ nextjs-webpack-stable-lazy-discovery-disabled12900
✅ nextjs-webpack-stable-lazy-discovery-enabled12900
✅ nitro-stable104025
✅ nuxt-stable104025
✅ sveltekit-stable12306
✅ vite-stable104025
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable104025
✅ express-stable104025
✅ fastify-stable104025
✅ hono-stable104025
✅ nextjs-turbopack-canary110019
✅ nextjs-turbopack-stable-lazy-discovery-disabled12900
✅ nextjs-turbopack-stable-lazy-discovery-enabled12900
✅ nextjs-webpack-canary110019
✅ nextjs-webpack-stable-lazy-discovery-disabled12900
✅ nextjs-webpack-stable-lazy-discovery-enabled12900
✅ nitro-stable104025
✅ nuxt-stable104025
✅ sveltekit-stable12306
✅ vite-stable104025
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable104025
✅ express-stable104025
✅ fastify-stable104025
✅ hono-stable104025
✅ nextjs-turbopack-canary110019
✅ nextjs-turbopack-stable-lazy-discovery-disabled12900
✅ nextjs-turbopack-stable-lazy-discovery-enabled12900
✅ nextjs-webpack-canary110019
✅ nextjs-webpack-stable-lazy-discovery-disabled12900
✅ nextjs-webpack-stable-lazy-discovery-enabled12900
✅ nitro-stable104025
✅ nuxt-stable104025
✅ sveltekit-stable12306
✅ vite-stable104025
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack12900
❌ 📋 Other
AppPassedFailedSkipped
✅ e2e-local-dev-nest-stable104025
✅ e2e-local-dev-tanstack-start-104025
✅ e2e-local-postgres-nest-stable104025
✅ e2e-local-postgres-tanstack-start-104025
✅ e2e-local-prod-nest-stable104025
✅ e2e-local-prod-tanstack-start-104025
❌ e2e-vercel-prod-tanstack-start102126

📋 View full workflow run


Some E2E test jobs failed:

  • Vercel Prod: failure
  • Local Dev: success
  • Local Prod: success
  • Local Postgres: success
  • Windows: success

Check the workflow run for details.

@github-actions

github-actionsBot commented May 20, 2026

Copy link
Copy Markdown
Contributor

📊 Benchmark Results

📈 Comparing against baseline from main branch. Green 🟢 = faster, Red 🔺 = slower.

workflow with no steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Express0.029s (-34.5% 🟢)1.005s (~)0.976s101.00x
💻 LocalNitro0.030s (-31.3% 🟢)1.005s (~)0.976s101.02x
💻 LocalNext.js (Turbopack)0.048s1.006s0.958s101.64x
🐘 PostgresExpress0.049s (-16.2% 🟢)1.011s (~)0.963s101.68x
🐘 PostgresNitro0.061s (-36.2% 🟢)1.013s (-2.8%)0.953s102.09x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 1 step

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Express1.067s (-5.2% 🟢)2.005s (~)0.939s101.00x
💻 LocalNitro1.072s (-5.2% 🟢)2.006s (~)0.934s101.00x
🐘 PostgresExpress1.084s (-5.4% 🟢)2.010s (~)0.926s101.02x
🐘 PostgresNitro1.103s (-3.3%)2.011s (~)0.908s101.03x
💻 LocalNext.js (Turbopack)1.112s2.005s0.894s101.04x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 10 sequential steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Express10.384s (-4.9%)11.021s (~)0.636s31.00x
🐘 PostgresExpress10.401s (-5.1% 🟢)11.014s (~)0.613s31.00x
💻 LocalNitro10.408s (-4.9%)11.023s (~)0.615s31.00x
🐘 PostgresNitro10.513s (-3.3%)11.017s (~)0.503s31.01x
💻 LocalNext.js (Turbopack)10.654s11.022s0.368s31.03x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 25 sequential steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Express13.401s (-10.5% 🟢)14.026s (-6.7% 🟢)0.626s51.00x
🐘 PostgresExpress13.406s (-8.1% 🟢)14.016s (-6.7% 🟢)0.610s51.00x
💻 LocalNitro13.447s (-10.7% 🟢)14.027s (-12.5% 🟢)0.581s51.00x
🐘 PostgresNitro13.729s (-5.9% 🟢)14.023s (-6.7% 🟢)0.293s51.02x
💻 LocalNext.js (Turbopack)14.118s14.828s0.710s51.05x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 50 sequential steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Express11.822s (-28.8% 🟢)12.022s (-29.4% 🟢)0.200s81.00x
💻 LocalNitro11.872s (-29.3% 🟢)12.024s (-29.4% 🟢)0.151s81.00x
🐘 PostgresExpress11.892s (-15.1% 🟢)12.017s (-17.7% 🟢)0.125s81.01x
🐘 PostgresNitro12.598s (-9.8% 🟢)13.025s (-9.0% 🟢)0.427s71.07x
💻 LocalNext.js (Turbopack)13.002s13.309s0.308s71.10x
🐘 PostgresNext.js (Turbopack)⚠️missing----
Promise.all with 10 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.135s (-9.9% 🟢)2.007s (~)0.872s151.00x
💻 LocalExpress1.163s (-21.9% 🟢)2.006s (~)0.843s151.02x
💻 LocalNitro1.170s (-28.3% 🟢)2.006s (-3.3%)0.836s151.03x
🐘 PostgresNitro1.197s (-6.1% 🟢)2.010s (~)0.814s151.05x
💻 LocalNext.js (Turbopack)1.254s2.006s0.751s151.10x
🐘 PostgresNext.js (Turbopack)⚠️missing----
Promise.all with 25 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.187s (-49.7% 🟢)2.006s (-33.3% 🟢)0.819s151.00x
🐘 PostgresNitro1.253s (-46.7% 🟢)2.010s (-33.2% 🟢)0.757s151.06x
💻 LocalExpress1.652s (-44.1% 🟢)2.005s (-41.9% 🟢)0.354s151.39x
💻 LocalNext.js (Turbopack)1.682s2.006s0.324s151.42x
💻 LocalNitro1.683s (-46.5% 🟢)2.006s (-48.4% 🟢)0.322s151.42x
🐘 PostgresNext.js (Turbopack)⚠️missing----
Promise.all with 50 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.304s (-62.6% 🟢)2.007s (-50.0% 🟢)0.703s151.00x
🐘 PostgresNitro1.390s (-60.1% 🟢)2.010s (-49.9% 🟢)0.620s151.07x
💻 LocalNitro4.483s (-46.3% 🟢)5.012s (-44.4% 🟢)0.529s63.44x
💻 LocalExpress4.632s (-44.5% 🟢)5.178s (-42.6% 🟢)0.546s63.55x
💻 LocalNext.js (Turbopack)4.684s5.179s0.495s63.59x
🐘 PostgresNext.js (Turbopack)⚠️missing----
Promise.race with 10 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.139s (-9.4% 🟢)2.008s (~)0.869s151.00x
🐘 PostgresNitro1.211s (-3.7%)2.011s (~)0.800s151.06x
💻 LocalNext.js (Turbopack)1.317s2.006s0.689s151.16x
💻 LocalExpress1.364s (-28.0% 🟢)2.006s (-15.1% 🟢)0.642s151.20x
💻 LocalNitro1.372s (-26.5% 🟢)2.006s (-14.3% 🟢)0.634s151.20x
🐘 PostgresNext.js (Turbopack)⚠️missing----
Promise.race with 25 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.195s (-49.0% 🟢)2.009s (-33.3% 🟢)0.814s151.00x
🐘 PostgresNitro1.274s (-45.5% 🟢)2.013s (-33.1% 🟢)0.738s151.07x
💻 LocalExpress1.833s (-41.5% 🟢)2.222s (-41.0% 🟢)0.389s141.53x
💻 LocalNitro1.916s (-37.5% 🟢)2.316s (-40.4% 🟢)0.400s131.60x
💻 LocalNext.js (Turbopack)1.927s2.393s0.466s131.61x
🐘 PostgresNext.js (Turbopack)⚠️missing----
Promise.race with 50 concurrent steps

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.290s (-63.1% 🟢)2.008s (-49.9% 🟢)0.718s151.00x
🐘 PostgresNitro1.430s (-58.9% 🟢)2.014s (-49.8% 🟢)0.584s151.11x
💻 LocalExpress4.559s (-48.2% 🟢)5.346s (-42.3% 🟢)0.787s63.53x
💻 LocalNext.js (Turbopack)4.916s5.352s0.436s63.81x
💻 LocalNitro5.467s (-40.2% 🟢)6.013s (-40.0% 🟢)0.547s54.24x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 10 sequential data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.435s (-48.2% 🟢)1.006s (-1.7%)0.572s601.00x
💻 LocalNitro0.454s (-53.7% 🟢)1.004s (-8.2% 🟢)0.549s601.05x
💻 LocalExpress0.468s (-52.4% 🟢)1.004s (-6.7% 🟢)0.535s601.08x
🐘 PostgresNitro0.568s (-30.8% 🟢)1.008s (~)0.441s601.31x
💻 LocalNext.js (Turbopack)0.720s1.004s0.284s601.66x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 25 sequential data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.004s (-49.2% 🟢)1.292s (-42.8% 🟢)0.288s701.00x
💻 LocalExpress1.152s (-61.8% 🟢)2.006s (-44.1% 🟢)0.854s451.15x
💻 LocalNitro1.162s (-61.7% 🟢)2.006s (-46.6% 🟢)0.844s451.16x
🐘 PostgresNitro1.385s (-28.1% 🟢)2.033s (-3.2%)0.648s451.38x
💻 LocalNext.js (Turbopack)1.807s2.051s0.244s441.80x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 50 sequential data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.965s (-50.8% 🟢)2.334s (-46.6% 🟢)0.369s521.00x
🐘 PostgresNitro2.560s (-37.6% 🟢)3.035s (-34.1% 🟢)0.475s401.30x
💻 LocalExpress2.619s (-71.6% 🟢)3.007s (-70.0% 🟢)0.388s401.33x
💻 LocalNitro2.637s (-71.6% 🟢)3.008s (-70.0% 🟢)0.370s401.34x
💻 LocalNext.js (Turbopack)3.859s4.074s0.216s301.96x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 10 concurrent data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.186s (-34.0% 🟢)1.022s (+1.5%)0.836s591.00x
🐘 PostgresNitro0.218s (-23.1% 🟢)1.007s (~)0.789s601.17x
💻 LocalExpress0.375s (-33.1% 🟢)1.004s (~)0.629s602.01x
💻 LocalNitro0.387s (-36.0% 🟢)1.004s (-1.7%)0.617s602.07x
💻 LocalNext.js (Turbopack)0.536s1.004s0.468s602.88x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 25 concurrent data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.277s (-45.7% 🟢)1.006s (~)0.729s901.00x
🐘 PostgresNitro0.356s (-28.4% 🟢)1.008s (~)0.652s901.28x
💻 LocalExpress2.068s (-17.7% 🟢)2.580s (-14.3% 🟢)0.512s357.46x
💻 LocalNitro2.099s (-17.3% 🟢)2.655s (-11.8% 🟢)0.556s347.58x
💻 LocalNext.js (Turbopack)2.335s2.976s0.641s318.43x
🐘 PostgresNext.js (Turbopack)⚠️missing----
workflow with 50 concurrent data payload steps (10KB)

💻 Local Development

WorldFrameworkWorkflow TimeWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.603s (-26.4% 🟢)1.006s (-1.1%)0.403s1201.00x
🐘 PostgresNitro0.723s (-8.6% 🟢)1.008s (~)0.285s1201.20x
💻 LocalExpress9.546s (-14.7% 🟢)10.111s (-15.3% 🟢)0.565s1215.84x
💻 LocalNitro9.729s (-13.1% 🟢)10.444s (-10.5% 🟢)0.715s1216.14x
💻 LocalNext.js (Turbopack)10.871s11.575s0.705s1118.04x
🐘 PostgresNext.js (Turbopack)⚠️missing----
Stream Benchmarks(includes TTFB metrics)
workflow with stream

💻 Local Development

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
💻 Local🥇 Express1.127s (+466.0% 🔺)2.005s (+99.6% 🔺)0.010s (-17.4% 🟢)2.017s (+98.1% 🔺)0.890s101.00x
🐘 PostgresExpress1.131s (+451.7% 🔺)1.995s (+99.8% 🔺)0.001s (-37.5% 🟢)2.010s (+98.7% 🔺)0.878s101.00x
💻 LocalNitro1.143s (+434.9% 🔺)2.005s (+99.6% 🔺)0.011s (-12.8% 🟢)2.018s (+98.1% 🔺)0.875s101.01x
🐘 PostgresNitro1.173s (+472.0% 🔺)1.998s (+99.9% 🔺)0.002s (+20.0% 🔺)2.013s (+99.1% 🔺)0.841s101.04x
💻 LocalNext.js (Turbopack)1.174s2.003s0.010s2.017s0.843s101.04x
🐘 PostgresNext.js (Turbopack)⚠️missing-----
stream pipeline with 5 transform steps (1MB)

💻 Local Development

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.498s (+137.7% 🔺)1.999s (+98.6% 🔺)0.004s (+5.3% 🔺)2.023s (+97.8% 🔺)0.526s301.00x
💻 LocalNext.js (Turbopack)1.669s2.009s0.011s2.022s0.354s301.11x
💻 LocalNitro1.691s (+101.6% 🔺)2.010s (+98.6% 🔺)0.009s (-1.3%)2.200s (+97.2% 🔺)0.509s281.13x
🐘 PostgresNitro1.701s (+172.5% 🔺)2.008s (+99.5% 🔺)0.005s (+11.2% 🔺)2.032s (+98.7% 🔺)0.331s301.14x
💻 LocalExpress1.909s (+152.2% 🔺)2.011s (+95.4% 🔺)0.009s (-0.6%)2.423s (+133.0% 🔺)0.514s251.27x
🐘 PostgresNext.js (Turbopack)⚠️missing-----
10 parallel streams (1MB each)

💻 Local Development

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express0.662s (-31.1% 🟢)1.048s (-18.0% 🟢)0.000s (+21.1% 🔺)1.059s (-18.9% 🟢)0.397s571.00x
🐘 PostgresNitro0.795s (-17.9% 🟢)1.049s (-16.0% 🟢)0.000s (+71.4% 🔺)1.082s (-13.9% 🟢)0.287s561.20x
💻 LocalExpress1.321s (+7.9% 🔺)2.015s (~)0.000s (-60.0% 🟢)2.017s (~)0.696s302.00x
💻 LocalNitro1.343s (+9.9% 🔺)2.015s (~)0.000s (+200.0% 🔺)2.017s (~)0.674s302.03x
💻 LocalNext.js (Turbopack)1.480s2.014s0.000s2.017s0.537s302.24x
🐘 PostgresNext.js (Turbopack)⚠️missing-----
fan-out fan-in 10 streams (1MB each)

💻 Local Development

WorldFrameworkWorkflow TimeTTFBSlurpWall TimeOverheadSamplesvs Fastest
🐘 Postgres🥇 Express1.258s (-29.0% 🟢)1.898s (-12.8% 🟢)0.000s (+Infinity% 🔺)1.931s (-12.2% 🟢)0.673s321.00x
🐘 PostgresNitro1.634s (-8.8% 🟢)2.263s (+5.7% 🔺)0.000s (+7.7% 🔺)2.309s (+6.2% 🔺)0.675s261.30x
💻 LocalNext.js (Turbopack)2.815s3.293s0.001s3.298s0.483s192.24x
💻 LocalExpress3.025s (-12.8% 🟢)3.556s (-11.8% 🟢)0.000s (-55.9% 🟢)3.562s (-11.8% 🟢)0.536s172.41x
💻 LocalNitro3.082s (-9.0% 🟢)3.778s (-6.3% 🟢)0.000s (-18.0% 🟢)3.782s (-6.3% 🟢)0.700s162.45x
🐘 PostgresNext.js (Turbopack)⚠️missing-----

Summary

Fastest Framework by World

Winner determined by most benchmark wins

World🥇 Fastest FrameworkWins
💻 LocalExpress16/21
🐘 PostgresExpress21/21
Fastest World by Framework

Winner determined by most benchmark wins

Framework🥇 Fastest WorldWins
Express🐘 Postgres15/21
Next.js (Turbopack)💻 Local21/21
Nitro🐘 Postgres11/21
Column Definitions
  • Workflow Time: Runtime reported by workflow (completedAt - createdAt) - primary metric
  • TTFB: Time to First Byte - time from workflow start until first stream byte received (stream benchmarks only)
  • Slurp: Time from first byte to complete stream consumption (stream benchmarks only)
  • Wall Time: Total testbench time (trigger workflow + poll for result)
  • Overhead: Testbench overhead (Wall Time - Workflow Time)
  • Samples: Number of benchmark iterations run
  • vs Fastest: How much slower compared to the fastest configuration for this benchmark

Worlds:

  • 💻 Local: In-memory filesystem world (local development)
  • 🐘 Postgres: PostgreSQL database world (local development)
  • ▲ Vercel: Vercel production/preview deployment
  • 🌐 Turso: Community world (local development)
  • 🌐 MongoDB: Community world (local development)
  • 🌐 Redis: Community world (local development)
  • 🌐 Jazz: Community world (local development)
  • 🌐 Redis: Community world (local development)
  • 🌐 Redis + BullMQ: Community world (local development)
  • 🌐 Cloudflare: Community world (local development)
  • 🌐 MySQL: Community world (local development)
  • 🌐 Azure: Community world (local development)
  • 🌐 NATS JetStream: Community world (local development)
  • 🌐 Upstash: Community world (local development)

📋 View full workflow run


Some benchmark jobs failed:

  • Local: success
  • Postgres: success
  • Vercel: failure

Check the workflow run for details.

⚠️Community world benchmarks failed (non-blocking):

  • Community Worlds: failure

Check the workflow run for details.

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes a race in @workflow/world-testing’s inline-execution test server where the per-run flow invocation counter could be incremented after the workflow completion became observable, causing flaky assertions in CI.

Changes:

  • Reorders the flow handler to increment the per-run invocation counter before awaiting/returning flowPOST.
  • Adds an explanatory comment documenting the race and why the ordering matters.
  • Adds a changeset bumping @workflow/world-testing with a patch release note.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

FileDescription
packages/world-testing/src/server.mtsMoves flow invocation counting ahead of the flow handler execution to eliminate the observed flake window.
.changeset/fix-inline-execution-flow-count-race.mdAdds a patch changeset describing the test-flake fix for @workflow/world-testing.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@TooTallNate
TooTallNate disabled auto-merge May 20, 2026 23:05
@TooTallNate
TooTallNate merged commit 96e2d3c into mainMay 20, 2026
108 of 118 checks passed
@TooTallNate
TooTallNate deleted the nate/fix-flow-invocation-count-race branch May 20, 2026 23:05
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@TooTallNate@karthikscale3