Skip to content

[e2e] Add Python e2e Test - #3369

Merged
fantix merged 3 commits into
mainfrom
fantix/python-conformance
Aug 14, 2026
Merged

[e2e] Add Python e2e Test#3369
fantix merged 3 commits into
mainfrom
fantix/python-conformance

Conversation

@fantix

@fantixfantix commented Aug 6, 2026•
edited
Loading

Copy link
Copy Markdown
Member

Description

Turns packages/core/e2e/e2e.test.ts into a cross-language conformance suite and adds workbench/python as its first non-JavaScript subject.

The gate. Tests whose subject is the JavaScript implementation itself — the compiler, @workflow/next, the CLI's TS codepaths — are marked testJsOnly / describeJsOnly. Everything else is in scope for any language and is gated on a per-app e2e-conformance.json:

fieldmeaning
languageanything but javascript skips the JS-only tests
fixtureswhich workflows/99_e2e fixtures the app implements
unsupportedindividual tests the app's runtime cannot pass yet, mapped to why

Both axes are ratchets, which is the point — a fixture listed but missing from the deployed manifest is a hard failure, not a skip, and an unsupported entry naming a test that no longer exists fails too. An app can only move forward by editing the declaration, and the declaration is the thing review looks at.

Apps that ship no e2e-conformance.json — every existing one — are reported as "JavaScript, all fixtures present", so this is a no-op for the JS matrix. Format reference: packages/core/e2e/e2e-conformance.example.json.

The Python app.workbench/python is a bare ASGI app on vercel-py, pinned by git rev rather than a release so the suite reports on what is actually being built. It serves the .well-known/workflow/v1 surface itself — the Python SDK does not claim to — and runs against both world-local (over HTTP) and a real Vercel deployment (over the CBOR queue transport, where the only HTTP route the driver touches is manifest.json).

Six fixtures are ported as a demo of the framework to start with: addTen, nullByte, parallelSleep, promiseAll, sleepInLoop, sleeping.

How did you test your changes?

Two CI lanes, both added here:

laneresult
E2E Python Conformance (world-local)8 passed, 129 skipped, of 137
E2E Vercel Prod Tests (python - node)7 of 156 when the race below goes our way, 6 when it does not

One baseline, not two: the extra 19 collected on Vercel are e2e-agent.test.ts, which that lane picks up and skips whole, and the eighth local pass is deploymentId: 'latest' is a no-op in non-Vercel worlds.

E2E Python Conformance is deliberately its own job rather than a getTestMatrix row. Every step in the two local lanes is JS toolchain — pnpm dev / pnpm start, the canary next patch, dev.test.ts and local-build.test.ts, the per-framework DEV_TEST_CONFIG of compiler-generated paths — and none of it has a Python counterpart. The e2e-vercel-prod lane, whose rows only need a URL, does carry python as an ordinary matrix row.

PR Checklist - Required to merge

  • 📦 pnpm changeset was run to create a changelog for this PR
  • 🔒 DCO sign-off passes
  • 📝 Ping @vercel/workflow in a comment once the PR is ready, and the above checklist is complete

@changeset-bot

changeset-botBot commented Aug 6, 2026•
edited
Loading

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: e4569c0

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 0 packages

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel

vercelBot commented Aug 6, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreviewAug 14, 2026 5:23pm
example-nextjs-workflow-webpackReadyReadyPreviewAug 14, 2026 5:23pm
example-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-astro-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-express-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-fastify-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-hono-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-nestjs-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-nitro-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-nuxt-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-python-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-sveltekit-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-tanstack-start-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-vite-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workflow-swc-playgroundReadyReadyPreviewAug 14, 2026 5:23pm
workflow-tarballsReadyReadyPreviewAug 14, 2026 5:23pm
workflow-webReadyReadyPreviewAug 14, 2026 5:23pm
1 Skipped Deployment
ProjectDeploymentActionsUpdated (UTC)
workflow-docsSkippedSkippedAug 14, 2026 5:23pm

@github-actions

github-actionsBot commented Aug 6, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

🧪 E2E Test Results

✅ All tests passed

E2E Test Summary

Summary
PassedFailedSkippedTotal
✅ ▲ Vercel Production347407384212
✅ 💻 Local Development381005584368
✅ 📦 Local Production381005584368
✅ 🐘 Local Postgres381005584368
✅ 🪟 Windows31200312
✅ 🌐 Cross-language Conformance90128137
✅ vercel-multi-region270027
Total152520254017792
Details by Category

✅ ▲ Vercel Production

AppPassedFailedSkipped
✅ astro-node128028
✅ astro-quickjs128028
✅ example-node128028
✅ example-quickjs128028
✅ express-node128028
✅ express-quickjs128028
✅ fastify-node128028
✅ fastify-quickjs128028
✅ hono-node128028
✅ hono-quickjs128028
✅ nest-node128028
✅ nest-quickjs128028
✅ nextjs-turbopack-node15303
✅ nextjs-turbopack-quickjs15303
✅ nextjs-webpack-node15303
✅ nextjs-webpack-quickjs15303
✅ nitro-node128028
✅ nitro-quickjs128028
✅ nuxt-node128028
✅ nuxt-quickjs128028
✅ python-node80148
✅ sveltekit-node14709
✅ sveltekit-quickjs14709
✅ tanstack-start-node128028
✅ tanstack-start-quickjs128028
✅ vite-node128028
✅ vite-quickjs128028

✅ 💻 Local Development

AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026

✅ 📦 Local Production

AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026

✅ 🐘 Local Postgres

AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026

✅ 🪟 Windows

AppPassedFailedSkipped
✅ nextjs-turbopack-node15600
✅ nextjs-turbopack-quickjs15600

✅ 🌐 Cross-language Conformance

AppPassedFailedSkipped
✅ python90128

✅ vercel-multi-region

AppPassedFailedSkipped
✅ nextjs-turbopack2700

📋 View full workflow run

@github-actions

github-actionsBot commented Aug 6, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

📊 Workflow Benchmarks

commit e4569c0 · Fri, 14 Aug 2026 17:40:20 GMT · run logs

Backend: vercel · app: nextjs-turbopack

MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep392 (+62%) 🔻1359 🔴 (+17%) 🔻1368 🔴 (+13%)1554 🔴 (+20%) 🔻30
TTFSstream1298 (+409%) 🔻1372 🔴 (+21%) 🔻1398 🔴 (+18%) 🔻1425 🔴 (+15%) 🔻30
TTFShook + stream501 (+14%)1664 🔴 (+18%) 🔻1751 🔴 (+21%) 🔻1980 🔴 (+28%) 🔻30
Fan-out TTFSPromise.all(100 steps)8855 (-1.1%)9050 (-12%)9256 (-16%) 💚10345 (-91%) 💚10
Fan-out TTLSPromise.all(100 steps)17348 (-2.3%)17542 (-8.1%)17682 (-10%)20103 (-84%) 💚10
STSO1020 steps (inline)117 (-17%) 💚171 (-19%) 💚194 (-23%) 💚281 (-50%) 💚1019
WO1020 steps169733 (-21%) 💚169733 (-21%) 💚169733 (-21%) 💚169733 (-21%) 💚1
SLstream latency89 (-17%) 💚126 🔴 (-16%) 💚140 🔴 (-22%) 💚152 🔴 (-82%) 💚30
SOstream overhead (text)110 (-13%)251 🔴 (-11%)477 (+14%)4679 🔴 (+426%) 🔻30
SOstream overhead (structured)100 (-28%) 💚160 (-56%) 💚168 (-75%) 💚260 (-77%) 💚30
📈 STSO distribution vs main (inline / queue-hop histograms)

1020 steps (inline)

Cumulative STSO time: main 214388ms → this run 169546ms (Δ -44842ms, -21%)

 100-150 ms █░░░┃ main 11 this 171 +160
150-200 ms ████████████████████░░░┃ main 653 this 766 +113
200-250 ms █┃██████ main 248 this 64 -184
250-300 ms ┃█ main 54 this 9 -45
300-350 ms ┃ main 17 this 5 -12
350-400 ms ┃ main 14 this 3 -11
400-450 ms ┃ main 4 this 1 -3
450-500 ms ┃ main 4 this 0 -4
500-550 ms ┃ main 3 this 0 -3
550-600 ms ┃ main 3 this 0 -3
600-650 ms ┃ main 2 this 0 -2
650-700 ms ┃ main 2 this 0 -2
750-800 ms ┃ main 1 this 0 -1
850-900 ms ┃ main 1 this 0 -1
1000-1050 ms ┃ main 1 this 0 -1
5100-5150 ms ┃ main 1 this 0 -1
📜 Previous results (1)

cbf7209

Fri, 14 Aug 2026 17:06:46 GMT · run logs

vercel / nextjs-turbopack

MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep1385 (+96%) 🔻1511 🔴 (+50%) 🔻1534 🔴 (+51%) 🔻1665 🔴 (+10%)30
TTFSstream1357 (+42%) 🔻1437 🔴 (+45%) 🔻1469 🔴 (+45%) 🔻1529 🔴 (+45%) 🔻30
TTFShook + stream1601 (+32%) 🔻1713 🔴 (+33%) 🔻1750 🔴 (+31%) 🔻2367 🔴 (+47%) 🔻30
Fan-out TTFSPromise.all(100 steps)891510634107101469310
Fan-out TTLSPromise.all(100 steps)1735019396194152452310
STSO1020 steps (inline)1351832133951019
WO1020 steps182912 (-53%) 💚182912 (-53%) 💚182912 (-53%) 💚182912 (-53%) 💚1
SLstream latency98 (+21%) 🔻134 🔴 (+1.5%)154 🔴 (+10%)293 🔴 (+64%) 🔻30
SOstream overhead (text)125 (+24%) 🔻186 (+3.3%)210 (+4.0%)275 (+10%)30
SOstream overhead (structured)126 (+27%) 🔻198 (+22%) 🔻273 (+40%) 🔻465 (+114%) 🔻30
ℹ️ Metric definitions & methodology

The collapsed STSO distribution section above buckets every step gap of the sequential-steps run (not a sampled window), split by whether the step ending the gap ran inline — in the same warm process as the step before it, so the gap is pure framework overhead — or after a queue-hop — the first step of a fresh process, which pays queue dispatch, client reinit and event-log replay. Bars overlay the two runs: █ is main, ┃ marks where this run lands, ░ bridges the gap when this run has more samples in a bucket.

Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.

Metrics — TTFS: time to first step body (in-deployment start() → first step body, deployment clocks) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · SL: stream latency (in-deployment write → read propagation, readAt - writtenAt) · SO: stream overhead (end-to-end write+consume time beyond the modelled generation window)

Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · stream latency: parallel reader/writer steps on a dedicated stream; SL is the in-deployment write->read propagation (readAt - writtenAt) · stream overhead (text): writer streams 300 variable-length text token deltas paced at 100/s for 3s (a haiku-size LLM's token throughput) while a parallel reader drains the whole stream; SO is the end-to-end write+consume time beyond the 3s generation window (overhead/backpressure) · stream overhead (structured): same workload as stream overhead (text), but each delta is an AI-SDK-style structured object ({ type: 'text-delta', id, text }) instead of a raw string, so the SO gap vs the text scenario is the added serialization cost

🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · SO 250/500/1000

All metrics are measured from deployment-side timestamps only. Runs are triggered by an in-deployment route that stamps the anchor (clientStart) right before start(), so the CI runner’s request and its path through api.vercel.com sit outside every measured window. TTFS = in-deployment start() → first step body (turbo uses the in-process fast path, non-turbo the dispatch path), and includes the VQS dispatch hop plus any /flow cold start. Fan-out TTFS/TTLS are the first and last step completions of a single Promise.all over trivial steps, from the same anchor, so the gap between the two rows is the spread the runtime adds across the fan-out. STSO/WO are measured between step bodies on the deployment. SL is measured inside the workflow (parallel reader/writer steps), so it no longer includes the api.vercel.com read path.

Cold starts are kept in the numbers on purpose — they are part of real bursty-workload latency. The workbench deployment cold-starts the /flow invocation for a large fraction of runs, inflating P75+; the Best column shows the fastest (warm-start) sample for comparison.

@socket-security

socket-securityBot commented Aug 6, 2026•
edited
Loading

Copy link
Copy Markdown

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

DiffPackageSupply Chain
Security		VulnerabilityQualityMaintenanceLicense
Addedpypi/​vercel@​0.10.097100100100100
Addedpypi/​uvicorn@​0.52.398100100100100

View full report

@vercel
vercelBottemporarily deployed to Preview – workflow-docs August 6, 2026 03:00 Inactive
vercel[bot]
vercelBot reviewed Aug 11, 2026
View reviewed changes
Comment threadworkbench/python/uv.lock Outdated
@github-actions

github-actionsBot commented Aug 11, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

Sim World

Simulated world deterministic testing for races. Traces

🟠 Mint-ordered log — 3 fail of 41 total

log=mint-ordered · fence=per-spec

scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
❌ in-flight-before-decisionfailed91.0mMISMATCH1
❌ in-flight-before-decision-countedfailed91.0mMISMATCH1
❌ in-flight-after-decisionfailed91.0mMISMATCH1
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0

Full trace: world-sim-mint.txt

🟢 Append-only log — 0 fail of 41 total

log=append-only · fence=per-spec

scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
✅ in-flight-before-decisioncompleted171.0mok0
✅ in-flight-before-decision-countedcompleted171.0mok0
✅ in-flight-after-decisioncompleted192.0mok0
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0

Full trace: world-sim-append-only.txt

@fantix@claude
[e2e] Run the e2e suite against a Python app
672209f
Turns `packages/core/e2e/e2e.test.ts` into a cross-language conformance
suite and adds `workbench/python` as its first non-JavaScript subject.
Tests whose subject is the JavaScript implementation are marked
`testJsOnly` / `describeJsOnly`; everything else is gated on a per-app
`e2e-conformance.json` declaring which `workflows/99_e2e` fixtures the
app implements, plus any individual tests its runtime cannot pass yet.
Apps without that file — every existing one — are unaffected, so this
is a no-op for the JS matrix.
The Python app is built on vercel-py, pinned by git rev rather than a
release so the suite reports on what is actually being built. It runs
against both `world-local` and a real Vercel deployment, the latter
through the CBOR queue transport rather than HTTP. Baseline is 8 of 137
passing locally and 7 of 156 on Vercel; the rest skip, and the point of
the lane is that they keep skipping for the declared reason.
`workbench/python/README.md` records what is and is not ported, and the
four gaps found on the way: the CBOR queue transport (vercel-py#265,
#266), the Trusted Sources bypass header (vercel-py#278), `encr`
payload decryption (vercel-py#279), and reading the slot-numbered
`specVersion: 6` this repo's world-vercel now stamps (#3389). All four
are fixed in the pinned rev.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@fantix

fantix commented Aug 12, 2026

Copy link
Copy Markdown
MemberAuthor

@vercel/workflow this PR is ready.

One test is flaky due to the missing implementation of resilient start in Python, which is being added in vercel/vercel-py#284. This PR should be slightly updated after that fix, but I think the majority of this PR is ready for review.

@fantixfantix mentioned this pull request Aug 12, 2026
Draft
VaguelySerious
VaguelySerious approved these changes Aug 12, 2026
View reviewed changes

@VaguelySeriousVaguelySerious left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

@fantix
Merge remote-tracking branch 'origin/main' into fantix/python-conform…
b5b690e
…ance
@fantix
[e2e] Track the vercel-py runtime's move to its own package
e4569c0
vercel-py#299 split the workflow runtime out of `vercel` into
`vercel-workflow`, so the pin now names one package instead of two —
`vercel-queue` 0.8.0 is released and carries the CBOR queue codec that
forced the second entry. The import namespace is unchanged; the
`_internal` paths are not (`vercel.workflow._internal`). `cryptography`
comes along outright rather than behind an `encryption` extra this app had
to remember to ask for.
What the app *declares* deliberately does not follow the split. Depending
on `vercel-workflow` — the only thing imported here — takes the Vercel lane
down: `@vercel/python` picks queue mode over workers mode by looking for
the literal name `vercel` in `[project].dependencies` and then reading
`importlib.metadata.version("vercel")`, and no released builder has heard
of the new name (checked through 6.56.2). Falling back writes the trigger
for consumer `app_registry` on `__wkf_*` instead of `default` on
`__wkf_workflow_*`, which is quieter than it sounds: deliveries do not
500, they never arrive, and all 8 conformance tests time out having logged
zero requests. So the umbrella package stays declared and pulls the
runtime (`vercel-workflow>=0.9.0,<1.1.0`) until the builder learns the
split. The cost is that the runtime resolves from PyPI rather than the
rev; at this rev that is exact, and the note above the source entry says
how to re-check it on the next bump.
Two API changes come with the bump. `HTTPRequest` now also requires
`method` and `url`, which is what lets `workflow_entrypoint` answer the
`?__health` probe itself (vercel-py#292, matching `withHealthCheck` in
`@workflow/core` down to the CORS headers) — so the hand-rolled
`{"status": "ok"}` here is gone and `app.py` only routes. The three
health-check tests stay JS-only, but the reason has changed: vercel-py
implements both probes and omits `workflowCoreVersion`, which the driver
asserts, on the grounds that it names a JavaScript package's version.
The bump also empties `unsupported`. vercel-py#282 carries `runInput`
through, #284 bootstraps the run from `run_started` instead of reading the
row first, and #283 gives `world-local` the same resilient start, so
*resilient start: addTenWorkflow completes when run_created returns 500*
passes and the local baseline is 9 of 137. That was never confined to the
one test: `start()` writes `run_created` and pushes the queue message in
parallel on purpose, so a consumer that 500s on a missing row loses an
arbitrary run per suite on Vercel — a different fixture each time, which
is why it read as flakiness for two rounds. The README records it as the
last of the Vercel-lane gaps.
@github-actions

github-actionsBot commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

No backport to stable for 09b299a (AI decision).

This adds new capability rather than fixing a defect: a cross-language conformance gating mechanism (e2e-conformance.json, testJsOnly/describeJsOnly, new @vitest/runner dependency), a brand-new workbench/python app, and two new CI lanes. It is explicitly a no-op for the existing JavaScript workbench apps, so it fixes nothing that is broken on stable and only expands test surface area on a maintenance branch.

To override, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:

09b299a03e4c4cac3b5ad630625f4f83c1dd908b

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vercelvercel[bot]vercel[bot] left review comments
@VaguelySeriousVaguelySeriousVaguelySerious approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@fantix@VaguelySerious
, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all 
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
[e2e] Add Python e2e Test by fantix · Pull Request #3369 · vercel/workflow · GitHub
Skip to content
[e2e] Add Python e2e Test - #3369
Merged
fantix merged 3 commits into
mainfrom 
fantix/python-conformance
Aug 14, 2026
Merged
[e2e] Add Python e2e Test#3369
fantix merged 3 commits into
mainfrom 
fantix/python-conformance
Conversation
@fantix
@fantixfantix
commented
Aug 6, 2026•

edited

Loading

Copy link


Copy Markdown

Member
Description
Turns packages/core/e2e/e2e.test.ts into a cross-language conformance suite and adds workbench/python as its first non-JavaScript subject.
The gate. Tests whose subject is the JavaScript implementation itself — the compiler, @workflow/next, the CLI's TS codepaths — are marked testJsOnly / describeJsOnly. Everything else is in scope for any language and is gated on a per-app e2e-conformance.json:
fieldmeaning
languageanything but javascript skips the JS-only tests
fixtureswhich workflows/99_e2e fixtures the app implements
unsupportedindividual tests the app's runtime cannot pass yet, mapped to why
Both axes are ratchets, which is the point — a fixture listed but missing from the deployed manifest is a hard failure, not a skip, and an unsupported entry naming a test that no longer exists fails too. An app can only move forward by editing the declaration, and the declaration is the thing review looks at.
Apps that ship no e2e-conformance.json — every existing one — are reported as "JavaScript, all fixtures present", so this is a no-op for the JS matrix. Format reference: packages/core/e2e/e2e-conformance.example.json.
The Python app.workbench/python is a bare ASGI app on vercel-py, pinned by git rev rather than a release so the suite reports on what is actually being built. It serves the .well-known/workflow/v1 surface itself — the Python SDK does not claim to — and runs against both world-local (over HTTP) and a real Vercel deployment (over the CBOR queue transport, where the only HTTP route the driver touches is manifest.json).
Six fixtures are ported as a demo of the framework to start with: addTen, nullByte, parallelSleep, promiseAll, sleepInLoop, sleeping.
How did you test your changes?
Two CI lanes, both added here:
laneresult
E2E Python Conformance (world-local)8 passed, 129 skipped, of 137
E2E Vercel Prod Tests (python - node)7 of 156 when the race below goes our way, 6 when it does not
One baseline, not two: the extra 19 collected on Vercel are e2e-agent.test.ts, which that lane picks up and skips whole, and the eighth local pass is deploymentId: 'latest' is a no-op in non-Vercel worlds.
E2E Python Conformance is deliberately its own job rather than a getTestMatrix row. Every step in the two local lanes is JS toolchain — pnpm dev / pnpm start, the canary next patch, dev.test.ts and local-build.test.ts, the per-framework DEV_TEST_CONFIG of compiler-generated paths — and none of it has a Python counterpart. The e2e-vercel-prod lane, whose rows only need a URL, does carry python as an ordinary matrix row.
PR Checklist - Required to merge
  •  📦 pnpm changeset was run to create a changelog for this PR
  •  🔒 DCO sign-off passes
  •  📝 Ping @vercel/workflow in a comment once the PR is ready, and the above checklist is complete
@changeset-bot
changeset-botBot
commented
Aug 6, 2026•

edited

Loading

Copy link


Copy Markdown

🦋 Changeset detected
Latest commit: e4569c0
The changes in this PR will be included in the next version bump.
This PR includes changesets to release 0 packages
When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types
Not sure what this means? Click here to learn what changesets are.
Click here if you're a maintainer who wants to add another changeset to this PR
@vercel
vercelBot
commented
Aug 6, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
The latest updates on your projects. Learn more about Vercel for GitHub.
ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreviewAug 14, 2026 5:23pm
example-nextjs-workflow-webpackReadyReadyPreviewAug 14, 2026 5:23pm
example-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-astro-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-express-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-fastify-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-hono-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-nestjs-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-nitro-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-nuxt-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-python-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-sveltekit-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-tanstack-start-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-vite-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workflow-swc-playgroundReadyReadyPreviewAug 14, 2026 5:23pm
workflow-tarballsReadyReadyPreviewAug 14, 2026 5:23pm
workflow-webReadyReadyPreviewAug 14, 2026 5:23pm
1 Skipped Deployment
ProjectDeploymentActionsUpdated (UTC)
workflow-docsSkippedSkippedAug 14, 2026 5:23pm
@github-actions
github-actionsBot
commented
Aug 6, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
🧪 E2E Test Results
✅ All tests passed
E2E Test Summary
Summary
PassedFailedSkippedTotal
✅ ▲ Vercel Production347407384212
✅ 💻 Local Development381005584368
✅ 📦 Local Production381005584368
✅ 🐘 Local Postgres381005584368
✅ 🪟 Windows31200312
✅ 🌐 Cross-language Conformance90128137
✅ vercel-multi-region270027
Total152520254017792
Details by Category
✅ ▲ Vercel Production
AppPassedFailedSkipped
✅ astro-node128028
✅ astro-quickjs128028
✅ example-node128028
✅ example-quickjs128028
✅ express-node128028
✅ express-quickjs128028
✅ fastify-node128028
✅ fastify-quickjs128028
✅ hono-node128028
✅ hono-quickjs128028
✅ nest-node128028
✅ nest-quickjs128028
✅ nextjs-turbopack-node15303
✅ nextjs-turbopack-quickjs15303
✅ nextjs-webpack-node15303
✅ nextjs-webpack-quickjs15303
✅ nitro-node128028
✅ nitro-quickjs128028
✅ nuxt-node128028
✅ nuxt-quickjs128028
✅ python-node80148
✅ sveltekit-node14709
✅ sveltekit-quickjs14709
✅ tanstack-start-node128028
✅ tanstack-start-quickjs128028
✅ vite-node128028
✅ vite-quickjs128028
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack-node15600
✅ nextjs-turbopack-quickjs15600
✅ 🌐 Cross-language Conformance
AppPassedFailedSkipped
✅ python90128
✅ vercel-multi-region
AppPassedFailedSkipped
✅ nextjs-turbopack2700
📋 View full workflow run
@github-actions
github-actionsBot
commented
Aug 6, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
📊 Workflow Benchmarks
commit e4569c0 · Fri, 14 Aug 2026 17:40:20 GMT · run logs
Backend: vercel · app: nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep392 (+62%) 🔻1359 🔴 (+17%) 🔻1368 🔴 (+13%)1554 🔴 (+20%) 🔻30
TTFSstream1298 (+409%) 🔻1372 🔴 (+21%) 🔻1398 🔴 (+18%) 🔻1425 🔴 (+15%) 🔻30
TTFShook + stream501 (+14%)1664 🔴 (+18%) 🔻1751 🔴 (+21%) 🔻1980 🔴 (+28%) 🔻30
Fan-out TTFSPromise.all(100 steps)8855 (-1.1%)9050 (-12%)9256 (-16%) 💚10345 (-91%) 💚10
Fan-out TTLSPromise.all(100 steps)17348 (-2.3%)17542 (-8.1%)17682 (-10%)20103 (-84%) 💚10
STSO1020 steps (inline)117 (-17%) 💚171 (-19%) 💚194 (-23%) 💚281 (-50%) 💚1019
WO1020 steps169733 (-21%) 💚169733 (-21%) 💚169733 (-21%) 💚169733 (-21%) 💚1
SLstream latency89 (-17%) 💚126 🔴 (-16%) 💚140 🔴 (-22%) 💚152 🔴 (-82%) 💚30
SOstream overhead (text)110 (-13%)251 🔴 (-11%)477 (+14%)4679 🔴 (+426%) 🔻30
SOstream overhead (structured)100 (-28%) 💚160 (-56%) 💚168 (-75%) 💚260 (-77%) 💚30
📈 STSO distribution vs main (inline / queue-hop histograms)
1020 steps (inline)
Cumulative STSO time: main 214388ms → this run 169546ms (Δ -44842ms, -21%)
 100-150 ms █░░░┃ main 11 this 171 +160
150-200 ms ████████████████████░░░┃ main 653 this 766 +113
200-250 ms █┃██████ main 248 this 64 -184
250-300 ms ┃█ main 54 this 9 -45
300-350 ms ┃ main 17 this 5 -12
350-400 ms ┃ main 14 this 3 -11
400-450 ms ┃ main 4 this 1 -3
450-500 ms ┃ main 4 this 0 -4
500-550 ms ┃ main 3 this 0 -3
550-600 ms ┃ main 3 this 0 -3
600-650 ms ┃ main 2 this 0 -2
650-700 ms ┃ main 2 this 0 -2
750-800 ms ┃ main 1 this 0 -1
850-900 ms ┃ main 1 this 0 -1
1000-1050 ms ┃ main 1 this 0 -1
5100-5150 ms ┃ main 1 this 0 -1

📜 Previous results (1)
cbf7209
Fri, 14 Aug 2026 17:06:46 GMT · run logs
vercel / nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep1385 (+96%) 🔻1511 🔴 (+50%) 🔻1534 🔴 (+51%) 🔻1665 🔴 (+10%)30
TTFSstream1357 (+42%) 🔻1437 🔴 (+45%) 🔻1469 🔴 (+45%) 🔻1529 🔴 (+45%) 🔻30
TTFShook + stream1601 (+32%) 🔻1713 🔴 (+33%) 🔻1750 🔴 (+31%) 🔻2367 🔴 (+47%) 🔻30
Fan-out TTFSPromise.all(100 steps)891510634107101469310
Fan-out TTLSPromise.all(100 steps)1735019396194152452310
STSO1020 steps (inline)1351832133951019
WO1020 steps182912 (-53%) 💚182912 (-53%) 💚182912 (-53%) 💚182912 (-53%) 💚1
SLstream latency98 (+21%) 🔻134 🔴 (+1.5%)154 🔴 (+10%)293 🔴 (+64%) 🔻30
SOstream overhead (text)125 (+24%) 🔻186 (+3.3%)210 (+4.0%)275 (+10%)30
SOstream overhead (structured)126 (+27%) 🔻198 (+22%) 🔻273 (+40%) 🔻465 (+114%) 🔻30
ℹ️ Metric definitions & methodology
The collapsed STSO distribution section above buckets every step gap of the sequential-steps run (not a sampled window), split by whether the step ending the gap ran inline — in the same warm process as the step before it, so the gap is pure framework overhead — or after a queue-hop — the first step of a fresh process, which pays queue dispatch, client reinit and event-log replay. Bars overlay the two runs: █ is main, ┃ marks where this run lands, ░ bridges the gap when this run has more samples in a bucket.
Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.
Metrics — TTFS: time to first step body (in-deployment start() → first step body, deployment clocks) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · SL: stream latency (in-deployment write → read propagation, readAt - writtenAt) · SO: stream overhead (end-to-end write+consume time beyond the modelled generation window)
Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · stream latency: parallel reader/writer steps on a dedicated stream; SL is the in-deployment write->read propagation (readAt - writtenAt) · stream overhead (text): writer streams 300 variable-length text token deltas paced at 100/s for 3s (a haiku-size LLM's token throughput) while a parallel reader drains the whole stream; SO is the end-to-end write+consume time beyond the 3s generation window (overhead/backpressure) · stream overhead (structured): same workload as stream overhead (text), but each delta is an AI-SDK-style structured object ({ type: 'text-delta', id, text }) instead of a raw string, so the SO gap vs the text scenario is the added serialization cost
🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · SO 250/500/1000
All metrics are measured from deployment-side timestamps only. Runs are triggered by an in-deployment route that stamps the anchor (clientStart) right before start(), so the CI runner’s request and its path through api.vercel.com sit outside every measured window. TTFS = in-deployment start() → first step body (turbo uses the in-process fast path, non-turbo the dispatch path), and includes the VQS dispatch hop plus any /flow cold start. Fan-out TTFS/TTLS are the first and last step completions of a single Promise.all over trivial steps, from the same anchor, so the gap between the two rows is the spread the runtime adds across the fan-out. STSO/WO are measured between step bodies on the deployment. SL is measured inside the workflow (parallel reader/writer steps), so it no longer includes the api.vercel.com read path.
Cold starts are kept in the numbers on purpose — they are part of real bursty-workload latency. The workbench deployment cold-starts the /flow invocation for a large fraction of runs, inflating P75+; the Best column shows the fastest (warm-start) sample for comparison.
@socket-security
socket-securityBot
commented
Aug 6, 2026•

edited

Loading

Copy link


Copy Markdown

Review the following changes in direct dependencies. Learn more about Socket for GitHub.
DiffPackageSupply Chain
Security		VulnerabilityQualityMaintenanceLicense
Addedpypi/​vercel@​0.10.097100100100100
Addedpypi/​uvicorn@​0.52.398100100100100
View full report
@vercel
vercelBottemporarily deployed
to
Preview – workflow-docs
August 6, 2026 03:00
Inactive

vercel[bot]
vercelBot
reviewed
Aug 11, 2026
View reviewed changes
Comment threadworkbench/python/uv.lock
Outdated

@github-actions
github-actionsBot
commented
Aug 11, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Sim World
Simulated world deterministic testing for races. Traces
🟠 Mint-ordered log — 3 fail of 41 total
log=mint-ordered · fence=per-spec
scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
❌ in-flight-before-decisionfailed91.0mMISMATCH1
❌ in-flight-before-decision-countedfailed91.0mMISMATCH1
❌ in-flight-after-decisionfailed91.0mMISMATCH1
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0
Full trace: world-sim-mint.txt
🟢 Append-only log — 0 fail of 41 total
log=append-only · fence=per-spec
scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
✅ in-flight-before-decisioncompleted171.0mok0
✅ in-flight-before-decision-countedcompleted171.0mok0
✅ in-flight-after-decisioncompleted192.0mok0
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0
Full trace: world-sim-append-only.txt
@fantix@claude
[e2e] Run the e2e suite against a Python app
672209f
Turns `packages/core/e2e/e2e.test.ts` into a cross-language conformance
suite and adds `workbench/python` as its first non-JavaScript subject.
Tests whose subject is the JavaScript implementation are marked
`testJsOnly` / `describeJsOnly`; everything else is gated on a per-app
`e2e-conformance.json` declaring which `workflows/99_e2e` fixtures the
app implements, plus any individual tests its runtime cannot pass yet.
Apps without that file — every existing one — are unaffected, so this
is a no-op for the JS matrix.
The Python app is built on vercel-py, pinned by git rev rather than a
release so the suite reports on what is actually being built. It runs
against both `world-local` and a real Vercel deployment, the latter
through the CBOR queue transport rather than HTTP. Baseline is 8 of 137
passing locally and 7 of 156 on Vercel; the rest skip, and the point of
the lane is that they keep skipping for the declared reason.
`workbench/python/README.md` records what is and is not ported, and the
four gaps found on the way: the CBOR queue transport (vercel-py#265,
#266), the Trusted Sources bypass header (vercel-py#278), `encr`
payload decryption (vercel-py#279), and reading the slot-numbered
`specVersion: 6` this repo's world-vercel now stamps (#3389). All four
are fixed in the pinned rev.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@fantix
fantix
commented
Aug 12, 2026

Copy link


Copy Markdown

MemberAuthor
@vercel/workflow this PR is ready.
One test is flaky due to the missing implementation of resilient start in Python, which is being added in vercel/vercel-py#284. This PR should be slightly updated after that fix, but I think the majority of this PR is ready for review.
@fantixfantix
mentioned this pull request
Aug 12, 2026
 Draft

VaguelySerious
VaguelySerious
approved these changes
Aug 12, 2026
View reviewed changes
@VaguelySeriousVaguelySerious
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!
@fantix
Merge remote-tracking branch 'origin/main' into fantix/python-conform…
b5b690e
…ance
@fantix
[e2e] Track the vercel-py runtime's move to its own package
e4569c0
vercel-py#299 split the workflow runtime out of `vercel` into
`vercel-workflow`, so the pin now names one package instead of two —
`vercel-queue` 0.8.0 is released and carries the CBOR queue codec that
forced the second entry. The import namespace is unchanged; the
`_internal` paths are not (`vercel.workflow._internal`). `cryptography`
comes along outright rather than behind an `encryption` extra this app had
to remember to ask for.
What the app *declares* deliberately does not follow the split. Depending
on `vercel-workflow` — the only thing imported here — takes the Vercel lane
down: `@vercel/python` picks queue mode over workers mode by looking for
the literal name `vercel` in `[project].dependencies` and then reading
`importlib.metadata.version("vercel")`, and no released builder has heard
of the new name (checked through 6.56.2). Falling back writes the trigger
for consumer `app_registry` on `__wkf_*` instead of `default` on
`__wkf_workflow_*`, which is quieter than it sounds: deliveries do not
500, they never arrive, and all 8 conformance tests time out having logged
zero requests. So the umbrella package stays declared and pulls the
runtime (`vercel-workflow>=0.9.0,<1.1.0`) until the builder learns the
split. The cost is that the runtime resolves from PyPI rather than the
rev; at this rev that is exact, and the note above the source entry says
how to re-check it on the next bump.
Two API changes come with the bump. `HTTPRequest` now also requires
`method` and `url`, which is what lets `workflow_entrypoint` answer the
`?__health` probe itself (vercel-py#292, matching `withHealthCheck` in
`@workflow/core` down to the CORS headers) — so the hand-rolled
`{"status": "ok"}` here is gone and `app.py` only routes. The three
health-check tests stay JS-only, but the reason has changed: vercel-py
implements both probes and omits `workflowCoreVersion`, which the driver
asserts, on the grounds that it names a JavaScript package's version.
The bump also empties `unsupported`. vercel-py#282 carries `runInput`
through, #284 bootstraps the run from `run_started` instead of reading the
row first, and #283 gives `world-local` the same resilient start, so
*resilient start: addTenWorkflow completes when run_created returns 500*
passes and the local baseline is 9 of 137. That was never confined to the
one test: `start()` writes `run_created` and pushes the queue message in
parallel on purpose, so a consumer that 500s on a missing row loses an
arbitrary run per suite on Vercel — a different fixture each time, which
is why it read as flakiness for two rounds. The README records it as the
last of the Vercel-lane gaps.
@github-actions
github-actionsBot
commented
Aug 14, 2026

Copy link


Copy Markdown

Contributor
No backport to stable for 09b299a (AI decision).
This adds new capability rather than fixing a defect: a cross-language conformance gating mechanism (e2e-conformance.json, testJsOnly/describeJsOnly, new @vitest/runner dependency), a brand-new workbench/python app, and two new CI lanes. It is explicitly a no-op for the existing JavaScript workbench apps, so it fixes nothing that is broken on stable and only expands test surface area on a maintenance branch.
To override, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:
09b299a03e4c4cac3b5ad630625f4f83c1dd908b

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@vercelvercel[bot]vercel[bot] left review comments
@VaguelySeriousVaguelySeriousVaguelySerious approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@fantix@VaguelySerious

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Force GitHub README to respect dark mode
(function() {
var style = document.createElement('style');
style.textContent = '
.markdown-body {
color-scheme: dark light;
}
.markdown-body pre { background: #161b22 !important; }
.markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }
.markdown-body table th, .markdown-body table td { border-color: #30363d !important; }
.markdown-body img { background: #0d1117; }
.markdown-body blockquote { border-left-color: #8b949e; }
.markdown-body hr { border-color: #30363d; }
';
document.head.appendChild(style);
})();
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
[e2e] Add Python e2e Test by fantix · Pull Request #3369 · vercel/workflow · GitHub
Skip to content
[e2e] Add Python e2e Test - #3369
Merged
fantix merged 3 commits into
mainfrom 
fantix/python-conformance
Aug 14, 2026
Merged
[e2e] Add Python e2e Test#3369
fantix merged 3 commits into
mainfrom 
fantix/python-conformance
Conversation
@fantix
@fantixfantix
commented
Aug 6, 2026•

edited

Loading

Copy link


Copy Markdown

Member
Description
Turns packages/core/e2e/e2e.test.ts into a cross-language conformance suite and adds workbench/python as its first non-JavaScript subject.
The gate. Tests whose subject is the JavaScript implementation itself — the compiler, @workflow/next, the CLI's TS codepaths — are marked testJsOnly / describeJsOnly. Everything else is in scope for any language and is gated on a per-app e2e-conformance.json:
fieldmeaning
languageanything but javascript skips the JS-only tests
fixtureswhich workflows/99_e2e fixtures the app implements
unsupportedindividual tests the app's runtime cannot pass yet, mapped to why
Both axes are ratchets, which is the point — a fixture listed but missing from the deployed manifest is a hard failure, not a skip, and an unsupported entry naming a test that no longer exists fails too. An app can only move forward by editing the declaration, and the declaration is the thing review looks at.
Apps that ship no e2e-conformance.json — every existing one — are reported as "JavaScript, all fixtures present", so this is a no-op for the JS matrix. Format reference: packages/core/e2e/e2e-conformance.example.json.
The Python app.workbench/python is a bare ASGI app on vercel-py, pinned by git rev rather than a release so the suite reports on what is actually being built. It serves the .well-known/workflow/v1 surface itself — the Python SDK does not claim to — and runs against both world-local (over HTTP) and a real Vercel deployment (over the CBOR queue transport, where the only HTTP route the driver touches is manifest.json).
Six fixtures are ported as a demo of the framework to start with: addTen, nullByte, parallelSleep, promiseAll, sleepInLoop, sleeping.
How did you test your changes?
Two CI lanes, both added here:
laneresult
E2E Python Conformance (world-local)8 passed, 129 skipped, of 137
E2E Vercel Prod Tests (python - node)7 of 156 when the race below goes our way, 6 when it does not
One baseline, not two: the extra 19 collected on Vercel are e2e-agent.test.ts, which that lane picks up and skips whole, and the eighth local pass is deploymentId: 'latest' is a no-op in non-Vercel worlds.
E2E Python Conformance is deliberately its own job rather than a getTestMatrix row. Every step in the two local lanes is JS toolchain — pnpm dev / pnpm start, the canary next patch, dev.test.ts and local-build.test.ts, the per-framework DEV_TEST_CONFIG of compiler-generated paths — and none of it has a Python counterpart. The e2e-vercel-prod lane, whose rows only need a URL, does carry python as an ordinary matrix row.
PR Checklist - Required to merge
  •  📦 pnpm changeset was run to create a changelog for this PR
  •  🔒 DCO sign-off passes
  •  📝 Ping @vercel/workflow in a comment once the PR is ready, and the above checklist is complete
@changeset-bot
changeset-botBot
commented
Aug 6, 2026•

edited

Loading

Copy link


Copy Markdown

🦋 Changeset detected
Latest commit: e4569c0
The changes in this PR will be included in the next version bump.
This PR includes changesets to release 0 packages
When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types
Not sure what this means? Click here to learn what changesets are.
Click here if you're a maintainer who wants to add another changeset to this PR
@vercel
vercelBot
commented
Aug 6, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
The latest updates on your projects. Learn more about Vercel for GitHub.
ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreviewAug 14, 2026 5:23pm
example-nextjs-workflow-webpackReadyReadyPreviewAug 14, 2026 5:23pm
example-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-astro-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-express-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-fastify-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-hono-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-nestjs-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-nitro-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-nuxt-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-python-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-sveltekit-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-tanstack-start-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-vite-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workflow-swc-playgroundReadyReadyPreviewAug 14, 2026 5:23pm
workflow-tarballsReadyReadyPreviewAug 14, 2026 5:23pm
workflow-webReadyReadyPreviewAug 14, 2026 5:23pm
1 Skipped Deployment
ProjectDeploymentActionsUpdated (UTC)
workflow-docsSkippedSkippedAug 14, 2026 5:23pm
@github-actions
github-actionsBot
commented
Aug 6, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
🧪 E2E Test Results
✅ All tests passed
E2E Test Summary
Summary
PassedFailedSkippedTotal
✅ ▲ Vercel Production347407384212
✅ 💻 Local Development381005584368
✅ 📦 Local Production381005584368
✅ 🐘 Local Postgres381005584368
✅ 🪟 Windows31200312
✅ 🌐 Cross-language Conformance90128137
✅ vercel-multi-region270027
Total152520254017792
Details by Category
✅ ▲ Vercel Production
AppPassedFailedSkipped
✅ astro-node128028
✅ astro-quickjs128028
✅ example-node128028
✅ example-quickjs128028
✅ express-node128028
✅ express-quickjs128028
✅ fastify-node128028
✅ fastify-quickjs128028
✅ hono-node128028
✅ hono-quickjs128028
✅ nest-node128028
✅ nest-quickjs128028
✅ nextjs-turbopack-node15303
✅ nextjs-turbopack-quickjs15303
✅ nextjs-webpack-node15303
✅ nextjs-webpack-quickjs15303
✅ nitro-node128028
✅ nitro-quickjs128028
✅ nuxt-node128028
✅ nuxt-quickjs128028
✅ python-node80148
✅ sveltekit-node14709
✅ sveltekit-quickjs14709
✅ tanstack-start-node128028
✅ tanstack-start-quickjs128028
✅ vite-node128028
✅ vite-quickjs128028
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack-node15600
✅ nextjs-turbopack-quickjs15600
✅ 🌐 Cross-language Conformance
AppPassedFailedSkipped
✅ python90128
✅ vercel-multi-region
AppPassedFailedSkipped
✅ nextjs-turbopack2700
📋 View full workflow run
@github-actions
github-actionsBot
commented
Aug 6, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
📊 Workflow Benchmarks
commit e4569c0 · Fri, 14 Aug 2026 17:40:20 GMT · run logs
Backend: vercel · app: nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep392 (+62%) 🔻1359 🔴 (+17%) 🔻1368 🔴 (+13%)1554 🔴 (+20%) 🔻30
TTFSstream1298 (+409%) 🔻1372 🔴 (+21%) 🔻1398 🔴 (+18%) 🔻1425 🔴 (+15%) 🔻30
TTFShook + stream501 (+14%)1664 🔴 (+18%) 🔻1751 🔴 (+21%) 🔻1980 🔴 (+28%) 🔻30
Fan-out TTFSPromise.all(100 steps)8855 (-1.1%)9050 (-12%)9256 (-16%) 💚10345 (-91%) 💚10
Fan-out TTLSPromise.all(100 steps)17348 (-2.3%)17542 (-8.1%)17682 (-10%)20103 (-84%) 💚10
STSO1020 steps (inline)117 (-17%) 💚171 (-19%) 💚194 (-23%) 💚281 (-50%) 💚1019
WO1020 steps169733 (-21%) 💚169733 (-21%) 💚169733 (-21%) 💚169733 (-21%) 💚1
SLstream latency89 (-17%) 💚126 🔴 (-16%) 💚140 🔴 (-22%) 💚152 🔴 (-82%) 💚30
SOstream overhead (text)110 (-13%)251 🔴 (-11%)477 (+14%)4679 🔴 (+426%) 🔻30
SOstream overhead (structured)100 (-28%) 💚160 (-56%) 💚168 (-75%) 💚260 (-77%) 💚30
📈 STSO distribution vs main (inline / queue-hop histograms)
1020 steps (inline)
Cumulative STSO time: main 214388ms → this run 169546ms (Δ -44842ms, -21%)
 100-150 ms █░░░┃ main 11 this 171 +160
150-200 ms ████████████████████░░░┃ main 653 this 766 +113
200-250 ms █┃██████ main 248 this 64 -184
250-300 ms ┃█ main 54 this 9 -45
300-350 ms ┃ main 17 this 5 -12
350-400 ms ┃ main 14 this 3 -11
400-450 ms ┃ main 4 this 1 -3
450-500 ms ┃ main 4 this 0 -4
500-550 ms ┃ main 3 this 0 -3
550-600 ms ┃ main 3 this 0 -3
600-650 ms ┃ main 2 this 0 -2
650-700 ms ┃ main 2 this 0 -2
750-800 ms ┃ main 1 this 0 -1
850-900 ms ┃ main 1 this 0 -1
1000-1050 ms ┃ main 1 this 0 -1
5100-5150 ms ┃ main 1 this 0 -1

📜 Previous results (1)
cbf7209
Fri, 14 Aug 2026 17:06:46 GMT · run logs
vercel / nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep1385 (+96%) 🔻1511 🔴 (+50%) 🔻1534 🔴 (+51%) 🔻1665 🔴 (+10%)30
TTFSstream1357 (+42%) 🔻1437 🔴 (+45%) 🔻1469 🔴 (+45%) 🔻1529 🔴 (+45%) 🔻30
TTFShook + stream1601 (+32%) 🔻1713 🔴 (+33%) 🔻1750 🔴 (+31%) 🔻2367 🔴 (+47%) 🔻30
Fan-out TTFSPromise.all(100 steps)891510634107101469310
Fan-out TTLSPromise.all(100 steps)1735019396194152452310
STSO1020 steps (inline)1351832133951019
WO1020 steps182912 (-53%) 💚182912 (-53%) 💚182912 (-53%) 💚182912 (-53%) 💚1
SLstream latency98 (+21%) 🔻134 🔴 (+1.5%)154 🔴 (+10%)293 🔴 (+64%) 🔻30
SOstream overhead (text)125 (+24%) 🔻186 (+3.3%)210 (+4.0%)275 (+10%)30
SOstream overhead (structured)126 (+27%) 🔻198 (+22%) 🔻273 (+40%) 🔻465 (+114%) 🔻30
ℹ️ Metric definitions & methodology
The collapsed STSO distribution section above buckets every step gap of the sequential-steps run (not a sampled window), split by whether the step ending the gap ran inline — in the same warm process as the step before it, so the gap is pure framework overhead — or after a queue-hop — the first step of a fresh process, which pays queue dispatch, client reinit and event-log replay. Bars overlay the two runs: █ is main, ┃ marks where this run lands, ░ bridges the gap when this run has more samples in a bucket.
Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.
Metrics — TTFS: time to first step body (in-deployment start() → first step body, deployment clocks) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · SL: stream latency (in-deployment write → read propagation, readAt - writtenAt) · SO: stream overhead (end-to-end write+consume time beyond the modelled generation window)
Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · stream latency: parallel reader/writer steps on a dedicated stream; SL is the in-deployment write->read propagation (readAt - writtenAt) · stream overhead (text): writer streams 300 variable-length text token deltas paced at 100/s for 3s (a haiku-size LLM's token throughput) while a parallel reader drains the whole stream; SO is the end-to-end write+consume time beyond the 3s generation window (overhead/backpressure) · stream overhead (structured): same workload as stream overhead (text), but each delta is an AI-SDK-style structured object ({ type: 'text-delta', id, text }) instead of a raw string, so the SO gap vs the text scenario is the added serialization cost
🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · SO 250/500/1000
All metrics are measured from deployment-side timestamps only. Runs are triggered by an in-deployment route that stamps the anchor (clientStart) right before start(), so the CI runner’s request and its path through api.vercel.com sit outside every measured window. TTFS = in-deployment start() → first step body (turbo uses the in-process fast path, non-turbo the dispatch path), and includes the VQS dispatch hop plus any /flow cold start. Fan-out TTFS/TTLS are the first and last step completions of a single Promise.all over trivial steps, from the same anchor, so the gap between the two rows is the spread the runtime adds across the fan-out. STSO/WO are measured between step bodies on the deployment. SL is measured inside the workflow (parallel reader/writer steps), so it no longer includes the api.vercel.com read path.
Cold starts are kept in the numbers on purpose — they are part of real bursty-workload latency. The workbench deployment cold-starts the /flow invocation for a large fraction of runs, inflating P75+; the Best column shows the fastest (warm-start) sample for comparison.
@socket-security
socket-securityBot
commented
Aug 6, 2026•

edited

Loading

Copy link


Copy Markdown

Review the following changes in direct dependencies. Learn more about Socket for GitHub.
DiffPackageSupply Chain
Security		VulnerabilityQualityMaintenanceLicense
Addedpypi/​vercel@​0.10.097100100100100
Addedpypi/​uvicorn@​0.52.398100100100100
View full report
@vercel
vercelBottemporarily deployed
to
Preview – workflow-docs
August 6, 2026 03:00
Inactive

vercel[bot]
vercelBot
reviewed
Aug 11, 2026
View reviewed changes
Comment threadworkbench/python/uv.lock
Outdated

@github-actions
github-actionsBot
commented
Aug 11, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Sim World
Simulated world deterministic testing for races. Traces
🟠 Mint-ordered log — 3 fail of 41 total
log=mint-ordered · fence=per-spec
scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
❌ in-flight-before-decisionfailed91.0mMISMATCH1
❌ in-flight-before-decision-countedfailed91.0mMISMATCH1
❌ in-flight-after-decisionfailed91.0mMISMATCH1
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0
Full trace: world-sim-mint.txt
🟢 Append-only log — 0 fail of 41 total
log=append-only · fence=per-spec
scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
✅ in-flight-before-decisioncompleted171.0mok0
✅ in-flight-before-decision-countedcompleted171.0mok0
✅ in-flight-after-decisioncompleted192.0mok0
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0
Full trace: world-sim-append-only.txt
@fantix@claude
[e2e] Run the e2e suite against a Python app
672209f
Turns `packages/core/e2e/e2e.test.ts` into a cross-language conformance
suite and adds `workbench/python` as its first non-JavaScript subject.
Tests whose subject is the JavaScript implementation are marked
`testJsOnly` / `describeJsOnly`; everything else is gated on a per-app
`e2e-conformance.json` declaring which `workflows/99_e2e` fixtures the
app implements, plus any individual tests its runtime cannot pass yet.
Apps without that file — every existing one — are unaffected, so this
is a no-op for the JS matrix.
The Python app is built on vercel-py, pinned by git rev rather than a
release so the suite reports on what is actually being built. It runs
against both `world-local` and a real Vercel deployment, the latter
through the CBOR queue transport rather than HTTP. Baseline is 8 of 137
passing locally and 7 of 156 on Vercel; the rest skip, and the point of
the lane is that they keep skipping for the declared reason.
`workbench/python/README.md` records what is and is not ported, and the
four gaps found on the way: the CBOR queue transport (vercel-py#265,
#266), the Trusted Sources bypass header (vercel-py#278), `encr`
payload decryption (vercel-py#279), and reading the slot-numbered
`specVersion: 6` this repo's world-vercel now stamps (#3389). All four
are fixed in the pinned rev.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@fantix
fantix
commented
Aug 12, 2026

Copy link


Copy Markdown

MemberAuthor
@vercel/workflow this PR is ready.
One test is flaky due to the missing implementation of resilient start in Python, which is being added in vercel/vercel-py#284. This PR should be slightly updated after that fix, but I think the majority of this PR is ready for review.
@fantixfantix
mentioned this pull request
Aug 12, 2026
 Draft

VaguelySerious
VaguelySerious
approved these changes
Aug 12, 2026
View reviewed changes
@VaguelySeriousVaguelySerious
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!
@fantix
Merge remote-tracking branch 'origin/main' into fantix/python-conform…
b5b690e
…ance
@fantix
[e2e] Track the vercel-py runtime's move to its own package
e4569c0
vercel-py#299 split the workflow runtime out of `vercel` into
`vercel-workflow`, so the pin now names one package instead of two —
`vercel-queue` 0.8.0 is released and carries the CBOR queue codec that
forced the second entry. The import namespace is unchanged; the
`_internal` paths are not (`vercel.workflow._internal`). `cryptography`
comes along outright rather than behind an `encryption` extra this app had
to remember to ask for.
What the app *declares* deliberately does not follow the split. Depending
on `vercel-workflow` — the only thing imported here — takes the Vercel lane
down: `@vercel/python` picks queue mode over workers mode by looking for
the literal name `vercel` in `[project].dependencies` and then reading
`importlib.metadata.version("vercel")`, and no released builder has heard
of the new name (checked through 6.56.2). Falling back writes the trigger
for consumer `app_registry` on `__wkf_*` instead of `default` on
`__wkf_workflow_*`, which is quieter than it sounds: deliveries do not
500, they never arrive, and all 8 conformance tests time out having logged
zero requests. So the umbrella package stays declared and pulls the
runtime (`vercel-workflow>=0.9.0,<1.1.0`) until the builder learns the
split. The cost is that the runtime resolves from PyPI rather than the
rev; at this rev that is exact, and the note above the source entry says
how to re-check it on the next bump.
Two API changes come with the bump. `HTTPRequest` now also requires
`method` and `url`, which is what lets `workflow_entrypoint` answer the
`?__health` probe itself (vercel-py#292, matching `withHealthCheck` in
`@workflow/core` down to the CORS headers) — so the hand-rolled
`{"status": "ok"}` here is gone and `app.py` only routes. The three
health-check tests stay JS-only, but the reason has changed: vercel-py
implements both probes and omits `workflowCoreVersion`, which the driver
asserts, on the grounds that it names a JavaScript package's version.
The bump also empties `unsupported`. vercel-py#282 carries `runInput`
through, #284 bootstraps the run from `run_started` instead of reading the
row first, and #283 gives `world-local` the same resilient start, so
*resilient start: addTenWorkflow completes when run_created returns 500*
passes and the local baseline is 9 of 137. That was never confined to the
one test: `start()` writes `run_created` and pushes the queue message in
parallel on purpose, so a consumer that 500s on a missing row loses an
arbitrary run per suite on Vercel — a different fixture each time, which
is why it read as flakiness for two rounds. The README records it as the
last of the Vercel-lane gaps.
@github-actions
github-actionsBot
commented
Aug 14, 2026

Copy link


Copy Markdown

Contributor
No backport to stable for 09b299a (AI decision).
This adds new capability rather than fixing a defect: a cross-language conformance gating mechanism (e2e-conformance.json, testJsOnly/describeJsOnly, new @vitest/runner dependency), a brand-new workbench/python app, and two new CI lanes. It is explicitly a no-op for the existing JavaScript workbench apps, so it fixes nothing that is broken on stable and only expands test surface area on a maintenance branch.
To override, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:
09b299a03e4c4cac3b5ad630625f4f83c1dd908b

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@vercelvercel[bot]vercel[bot] left review comments
@VaguelySeriousVaguelySeriousVaguelySerious approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@fantix@VaguelySerious

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Highlight search terms from Google/DuckDuckGo/Bing referrer
(function() {
var ref = document.referrer;
var terms = [];
if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {
var url = new URL(ref);
var q = url.searchParams.get('q') || url.searchParams.get('p');
if (q) {
terms = q.split(/\s+/).filter(function(t) { return t.length > 2; });
}
}
if (terms.length === 0) return;
var style = document.createElement('style');
style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';
document.head.appendChild(style);
function highlight(node) {
if (node.nodeType === 3) { // text node
var text = node.textContent;
var found = false;
terms.forEach(function(term) {
var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi');
if (regex.test(text)) {
found = true;
var frag = document.createDocumentFragment();
var parts = text.split(regex);
parts.forEach(function(part, i) {
if (i % 2 === 0) {
frag.appendChild(document.createTextNode(part));
} else {
var span = document.createElement('span');
span.className = 'userscript-highlight';
span.textContent = part;
frag.appendChild(span);
}
});
node.parentNode.replaceChild(frag, node);
}
});
} else if (node.nodeType === 1 && node.childNodes) { // element
var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];
if (!skipTags.includes(node.tagName)) {
Array.from(node.childNodes).forEach(highlight);
}
}
}
highlight(document.body);
// Re-highlight on dynamic content
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1 || node.nodeType === 3) highlight(node);
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
[e2e] Add Python e2e Test by fantix · Pull Request #3369 · vercel/workflow · GitHub
Skip to content
[e2e] Add Python e2e Test - #3369
Merged
fantix merged 3 commits into
mainfrom 
fantix/python-conformance
Aug 14, 2026
Merged
[e2e] Add Python e2e Test#3369
fantix merged 3 commits into
mainfrom 
fantix/python-conformance
Conversation
@fantix
@fantixfantix
commented
Aug 6, 2026•

edited

Loading

Copy link


Copy Markdown

Member
Description
Turns packages/core/e2e/e2e.test.ts into a cross-language conformance suite and adds workbench/python as its first non-JavaScript subject.
The gate. Tests whose subject is the JavaScript implementation itself — the compiler, @workflow/next, the CLI's TS codepaths — are marked testJsOnly / describeJsOnly. Everything else is in scope for any language and is gated on a per-app e2e-conformance.json:
fieldmeaning
languageanything but javascript skips the JS-only tests
fixtureswhich workflows/99_e2e fixtures the app implements
unsupportedindividual tests the app's runtime cannot pass yet, mapped to why
Both axes are ratchets, which is the point — a fixture listed but missing from the deployed manifest is a hard failure, not a skip, and an unsupported entry naming a test that no longer exists fails too. An app can only move forward by editing the declaration, and the declaration is the thing review looks at.
Apps that ship no e2e-conformance.json — every existing one — are reported as "JavaScript, all fixtures present", so this is a no-op for the JS matrix. Format reference: packages/core/e2e/e2e-conformance.example.json.
The Python app.workbench/python is a bare ASGI app on vercel-py, pinned by git rev rather than a release so the suite reports on what is actually being built. It serves the .well-known/workflow/v1 surface itself — the Python SDK does not claim to — and runs against both world-local (over HTTP) and a real Vercel deployment (over the CBOR queue transport, where the only HTTP route the driver touches is manifest.json).
Six fixtures are ported as a demo of the framework to start with: addTen, nullByte, parallelSleep, promiseAll, sleepInLoop, sleeping.
How did you test your changes?
Two CI lanes, both added here:
laneresult
E2E Python Conformance (world-local)8 passed, 129 skipped, of 137
E2E Vercel Prod Tests (python - node)7 of 156 when the race below goes our way, 6 when it does not
One baseline, not two: the extra 19 collected on Vercel are e2e-agent.test.ts, which that lane picks up and skips whole, and the eighth local pass is deploymentId: 'latest' is a no-op in non-Vercel worlds.
E2E Python Conformance is deliberately its own job rather than a getTestMatrix row. Every step in the two local lanes is JS toolchain — pnpm dev / pnpm start, the canary next patch, dev.test.ts and local-build.test.ts, the per-framework DEV_TEST_CONFIG of compiler-generated paths — and none of it has a Python counterpart. The e2e-vercel-prod lane, whose rows only need a URL, does carry python as an ordinary matrix row.
PR Checklist - Required to merge
  •  📦 pnpm changeset was run to create a changelog for this PR
  •  🔒 DCO sign-off passes
  •  📝 Ping @vercel/workflow in a comment once the PR is ready, and the above checklist is complete
@changeset-bot
changeset-botBot
commented
Aug 6, 2026•

edited

Loading

Copy link


Copy Markdown

🦋 Changeset detected
Latest commit: e4569c0
The changes in this PR will be included in the next version bump.
This PR includes changesets to release 0 packages
When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types
Not sure what this means? Click here to learn what changesets are.
Click here if you're a maintainer who wants to add another changeset to this PR
@vercel
vercelBot
commented
Aug 6, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
The latest updates on your projects. Learn more about Vercel for GitHub.
ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreviewAug 14, 2026 5:23pm
example-nextjs-workflow-webpackReadyReadyPreviewAug 14, 2026 5:23pm
example-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-astro-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-express-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-fastify-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-hono-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-nestjs-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-nitro-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-nuxt-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-python-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-sveltekit-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-tanstack-start-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-vite-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workflow-swc-playgroundReadyReadyPreviewAug 14, 2026 5:23pm
workflow-tarballsReadyReadyPreviewAug 14, 2026 5:23pm
workflow-webReadyReadyPreviewAug 14, 2026 5:23pm
1 Skipped Deployment
ProjectDeploymentActionsUpdated (UTC)
workflow-docsSkippedSkippedAug 14, 2026 5:23pm
@github-actions
github-actionsBot
commented
Aug 6, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
🧪 E2E Test Results
✅ All tests passed
E2E Test Summary
Summary
PassedFailedSkippedTotal
✅ ▲ Vercel Production347407384212
✅ 💻 Local Development381005584368
✅ 📦 Local Production381005584368
✅ 🐘 Local Postgres381005584368
✅ 🪟 Windows31200312
✅ 🌐 Cross-language Conformance90128137
✅ vercel-multi-region270027
Total152520254017792
Details by Category
✅ ▲ Vercel Production
AppPassedFailedSkipped
✅ astro-node128028
✅ astro-quickjs128028
✅ example-node128028
✅ example-quickjs128028
✅ express-node128028
✅ express-quickjs128028
✅ fastify-node128028
✅ fastify-quickjs128028
✅ hono-node128028
✅ hono-quickjs128028
✅ nest-node128028
✅ nest-quickjs128028
✅ nextjs-turbopack-node15303
✅ nextjs-turbopack-quickjs15303
✅ nextjs-webpack-node15303
✅ nextjs-webpack-quickjs15303
✅ nitro-node128028
✅ nitro-quickjs128028
✅ nuxt-node128028
✅ nuxt-quickjs128028
✅ python-node80148
✅ sveltekit-node14709
✅ sveltekit-quickjs14709
✅ tanstack-start-node128028
✅ tanstack-start-quickjs128028
✅ vite-node128028
✅ vite-quickjs128028
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack-node15600
✅ nextjs-turbopack-quickjs15600
✅ 🌐 Cross-language Conformance
AppPassedFailedSkipped
✅ python90128
✅ vercel-multi-region
AppPassedFailedSkipped
✅ nextjs-turbopack2700
📋 View full workflow run
@github-actions
github-actionsBot
commented
Aug 6, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
📊 Workflow Benchmarks
commit e4569c0 · Fri, 14 Aug 2026 17:40:20 GMT · run logs
Backend: vercel · app: nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep392 (+62%) 🔻1359 🔴 (+17%) 🔻1368 🔴 (+13%)1554 🔴 (+20%) 🔻30
TTFSstream1298 (+409%) 🔻1372 🔴 (+21%) 🔻1398 🔴 (+18%) 🔻1425 🔴 (+15%) 🔻30
TTFShook + stream501 (+14%)1664 🔴 (+18%) 🔻1751 🔴 (+21%) 🔻1980 🔴 (+28%) 🔻30
Fan-out TTFSPromise.all(100 steps)8855 (-1.1%)9050 (-12%)9256 (-16%) 💚10345 (-91%) 💚10
Fan-out TTLSPromise.all(100 steps)17348 (-2.3%)17542 (-8.1%)17682 (-10%)20103 (-84%) 💚10
STSO1020 steps (inline)117 (-17%) 💚171 (-19%) 💚194 (-23%) 💚281 (-50%) 💚1019
WO1020 steps169733 (-21%) 💚169733 (-21%) 💚169733 (-21%) 💚169733 (-21%) 💚1
SLstream latency89 (-17%) 💚126 🔴 (-16%) 💚140 🔴 (-22%) 💚152 🔴 (-82%) 💚30
SOstream overhead (text)110 (-13%)251 🔴 (-11%)477 (+14%)4679 🔴 (+426%) 🔻30
SOstream overhead (structured)100 (-28%) 💚160 (-56%) 💚168 (-75%) 💚260 (-77%) 💚30
📈 STSO distribution vs main (inline / queue-hop histograms)
1020 steps (inline)
Cumulative STSO time: main 214388ms → this run 169546ms (Δ -44842ms, -21%)
 100-150 ms █░░░┃ main 11 this 171 +160
150-200 ms ████████████████████░░░┃ main 653 this 766 +113
200-250 ms █┃██████ main 248 this 64 -184
250-300 ms ┃█ main 54 this 9 -45
300-350 ms ┃ main 17 this 5 -12
350-400 ms ┃ main 14 this 3 -11
400-450 ms ┃ main 4 this 1 -3
450-500 ms ┃ main 4 this 0 -4
500-550 ms ┃ main 3 this 0 -3
550-600 ms ┃ main 3 this 0 -3
600-650 ms ┃ main 2 this 0 -2
650-700 ms ┃ main 2 this 0 -2
750-800 ms ┃ main 1 this 0 -1
850-900 ms ┃ main 1 this 0 -1
1000-1050 ms ┃ main 1 this 0 -1
5100-5150 ms ┃ main 1 this 0 -1

📜 Previous results (1)
cbf7209
Fri, 14 Aug 2026 17:06:46 GMT · run logs
vercel / nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep1385 (+96%) 🔻1511 🔴 (+50%) 🔻1534 🔴 (+51%) 🔻1665 🔴 (+10%)30
TTFSstream1357 (+42%) 🔻1437 🔴 (+45%) 🔻1469 🔴 (+45%) 🔻1529 🔴 (+45%) 🔻30
TTFShook + stream1601 (+32%) 🔻1713 🔴 (+33%) 🔻1750 🔴 (+31%) 🔻2367 🔴 (+47%) 🔻30
Fan-out TTFSPromise.all(100 steps)891510634107101469310
Fan-out TTLSPromise.all(100 steps)1735019396194152452310
STSO1020 steps (inline)1351832133951019
WO1020 steps182912 (-53%) 💚182912 (-53%) 💚182912 (-53%) 💚182912 (-53%) 💚1
SLstream latency98 (+21%) 🔻134 🔴 (+1.5%)154 🔴 (+10%)293 🔴 (+64%) 🔻30
SOstream overhead (text)125 (+24%) 🔻186 (+3.3%)210 (+4.0%)275 (+10%)30
SOstream overhead (structured)126 (+27%) 🔻198 (+22%) 🔻273 (+40%) 🔻465 (+114%) 🔻30
ℹ️ Metric definitions & methodology
The collapsed STSO distribution section above buckets every step gap of the sequential-steps run (not a sampled window), split by whether the step ending the gap ran inline — in the same warm process as the step before it, so the gap is pure framework overhead — or after a queue-hop — the first step of a fresh process, which pays queue dispatch, client reinit and event-log replay. Bars overlay the two runs: █ is main, ┃ marks where this run lands, ░ bridges the gap when this run has more samples in a bucket.
Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.
Metrics — TTFS: time to first step body (in-deployment start() → first step body, deployment clocks) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · SL: stream latency (in-deployment write → read propagation, readAt - writtenAt) · SO: stream overhead (end-to-end write+consume time beyond the modelled generation window)
Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · stream latency: parallel reader/writer steps on a dedicated stream; SL is the in-deployment write->read propagation (readAt - writtenAt) · stream overhead (text): writer streams 300 variable-length text token deltas paced at 100/s for 3s (a haiku-size LLM's token throughput) while a parallel reader drains the whole stream; SO is the end-to-end write+consume time beyond the 3s generation window (overhead/backpressure) · stream overhead (structured): same workload as stream overhead (text), but each delta is an AI-SDK-style structured object ({ type: 'text-delta', id, text }) instead of a raw string, so the SO gap vs the text scenario is the added serialization cost
🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · SO 250/500/1000
All metrics are measured from deployment-side timestamps only. Runs are triggered by an in-deployment route that stamps the anchor (clientStart) right before start(), so the CI runner’s request and its path through api.vercel.com sit outside every measured window. TTFS = in-deployment start() → first step body (turbo uses the in-process fast path, non-turbo the dispatch path), and includes the VQS dispatch hop plus any /flow cold start. Fan-out TTFS/TTLS are the first and last step completions of a single Promise.all over trivial steps, from the same anchor, so the gap between the two rows is the spread the runtime adds across the fan-out. STSO/WO are measured between step bodies on the deployment. SL is measured inside the workflow (parallel reader/writer steps), so it no longer includes the api.vercel.com read path.
Cold starts are kept in the numbers on purpose — they are part of real bursty-workload latency. The workbench deployment cold-starts the /flow invocation for a large fraction of runs, inflating P75+; the Best column shows the fastest (warm-start) sample for comparison.
@socket-security
socket-securityBot
commented
Aug 6, 2026•

edited

Loading

Copy link


Copy Markdown

Review the following changes in direct dependencies. Learn more about Socket for GitHub.
DiffPackageSupply Chain
Security		VulnerabilityQualityMaintenanceLicense
Addedpypi/​vercel@​0.10.097100100100100
Addedpypi/​uvicorn@​0.52.398100100100100
View full report
@vercel
vercelBottemporarily deployed
to
Preview – workflow-docs
August 6, 2026 03:00
Inactive

vercel[bot]
vercelBot
reviewed
Aug 11, 2026
View reviewed changes
Comment threadworkbench/python/uv.lock
Outdated

@github-actions
github-actionsBot
commented
Aug 11, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Sim World
Simulated world deterministic testing for races. Traces
🟠 Mint-ordered log — 3 fail of 41 total
log=mint-ordered · fence=per-spec
scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
❌ in-flight-before-decisionfailed91.0mMISMATCH1
❌ in-flight-before-decision-countedfailed91.0mMISMATCH1
❌ in-flight-after-decisionfailed91.0mMISMATCH1
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0
Full trace: world-sim-mint.txt
🟢 Append-only log — 0 fail of 41 total
log=append-only · fence=per-spec
scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
✅ in-flight-before-decisioncompleted171.0mok0
✅ in-flight-before-decision-countedcompleted171.0mok0
✅ in-flight-after-decisioncompleted192.0mok0
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0
Full trace: world-sim-append-only.txt
@fantix@claude
[e2e] Run the e2e suite against a Python app
672209f
Turns `packages/core/e2e/e2e.test.ts` into a cross-language conformance
suite and adds `workbench/python` as its first non-JavaScript subject.
Tests whose subject is the JavaScript implementation are marked
`testJsOnly` / `describeJsOnly`; everything else is gated on a per-app
`e2e-conformance.json` declaring which `workflows/99_e2e` fixtures the
app implements, plus any individual tests its runtime cannot pass yet.
Apps without that file — every existing one — are unaffected, so this
is a no-op for the JS matrix.
The Python app is built on vercel-py, pinned by git rev rather than a
release so the suite reports on what is actually being built. It runs
against both `world-local` and a real Vercel deployment, the latter
through the CBOR queue transport rather than HTTP. Baseline is 8 of 137
passing locally and 7 of 156 on Vercel; the rest skip, and the point of
the lane is that they keep skipping for the declared reason.
`workbench/python/README.md` records what is and is not ported, and the
four gaps found on the way: the CBOR queue transport (vercel-py#265,
#266), the Trusted Sources bypass header (vercel-py#278), `encr`
payload decryption (vercel-py#279), and reading the slot-numbered
`specVersion: 6` this repo's world-vercel now stamps (#3389). All four
are fixed in the pinned rev.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@fantix
fantix
commented
Aug 12, 2026

Copy link


Copy Markdown

MemberAuthor
@vercel/workflow this PR is ready.
One test is flaky due to the missing implementation of resilient start in Python, which is being added in vercel/vercel-py#284. This PR should be slightly updated after that fix, but I think the majority of this PR is ready for review.
@fantixfantix
mentioned this pull request
Aug 12, 2026
 Draft

VaguelySerious
VaguelySerious
approved these changes
Aug 12, 2026
View reviewed changes
@VaguelySeriousVaguelySerious
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!
@fantix
Merge remote-tracking branch 'origin/main' into fantix/python-conform…
b5b690e
…ance
@fantix
[e2e] Track the vercel-py runtime's move to its own package
e4569c0
vercel-py#299 split the workflow runtime out of `vercel` into
`vercel-workflow`, so the pin now names one package instead of two —
`vercel-queue` 0.8.0 is released and carries the CBOR queue codec that
forced the second entry. The import namespace is unchanged; the
`_internal` paths are not (`vercel.workflow._internal`). `cryptography`
comes along outright rather than behind an `encryption` extra this app had
to remember to ask for.
What the app *declares* deliberately does not follow the split. Depending
on `vercel-workflow` — the only thing imported here — takes the Vercel lane
down: `@vercel/python` picks queue mode over workers mode by looking for
the literal name `vercel` in `[project].dependencies` and then reading
`importlib.metadata.version("vercel")`, and no released builder has heard
of the new name (checked through 6.56.2). Falling back writes the trigger
for consumer `app_registry` on `__wkf_*` instead of `default` on
`__wkf_workflow_*`, which is quieter than it sounds: deliveries do not
500, they never arrive, and all 8 conformance tests time out having logged
zero requests. So the umbrella package stays declared and pulls the
runtime (`vercel-workflow>=0.9.0,<1.1.0`) until the builder learns the
split. The cost is that the runtime resolves from PyPI rather than the
rev; at this rev that is exact, and the note above the source entry says
how to re-check it on the next bump.
Two API changes come with the bump. `HTTPRequest` now also requires
`method` and `url`, which is what lets `workflow_entrypoint` answer the
`?__health` probe itself (vercel-py#292, matching `withHealthCheck` in
`@workflow/core` down to the CORS headers) — so the hand-rolled
`{"status": "ok"}` here is gone and `app.py` only routes. The three
health-check tests stay JS-only, but the reason has changed: vercel-py
implements both probes and omits `workflowCoreVersion`, which the driver
asserts, on the grounds that it names a JavaScript package's version.
The bump also empties `unsupported`. vercel-py#282 carries `runInput`
through, #284 bootstraps the run from `run_started` instead of reading the
row first, and #283 gives `world-local` the same resilient start, so
*resilient start: addTenWorkflow completes when run_created returns 500*
passes and the local baseline is 9 of 137. That was never confined to the
one test: `start()` writes `run_created` and pushes the queue message in
parallel on purpose, so a consumer that 500s on a missing row loses an
arbitrary run per suite on Vercel — a different fixture each time, which
is why it read as flakiness for two rounds. The README records it as the
last of the Vercel-lane gaps.
@github-actions
github-actionsBot
commented
Aug 14, 2026

Copy link


Copy Markdown

Contributor
No backport to stable for 09b299a (AI decision).
This adds new capability rather than fixing a defect: a cross-language conformance gating mechanism (e2e-conformance.json, testJsOnly/describeJsOnly, new @vitest/runner dependency), a brand-new workbench/python app, and two new CI lanes. It is explicitly a no-op for the existing JavaScript workbench apps, so it fixes nothing that is broken on stable and only expands test surface area on a maintenance branch.
To override, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:
09b299a03e4c4cac3b5ad630625f4f83c1dd908b

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@vercelvercel[bot]vercel[bot] left review comments
@VaguelySeriousVaguelySeriousVaguelySerious approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@fantix@VaguelySerious

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Strip utm_, fbclid, gclid, etc. from all links on page
(function() {
var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',
'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',
'ref', 'ref_src', 'source', 'medium', 'campaign'];
function cleanUrl(url) {
try {
var u = new URL(url, window.location.origin);
var changed = false;
trackingParams.forEach(function(p) {
if (u.searchParams.has(p)) {
u.searchParams.delete(p);
changed = true;
}
});
return changed ? u.toString() : url;
} catch (e) {
return url;
}
}
function cleanLinks() {
document.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
cleanLinks();
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1) {
if (node.tagName === 'A') cleanLinks();
node.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '
[e2e] Add Python e2e Test by fantix · Pull Request #3369 · vercel/workflow · GitHub
Skip to content
[e2e] Add Python e2e Test - #3369
Merged
fantix merged 3 commits into
mainfrom 
fantix/python-conformance
Aug 14, 2026
Merged
[e2e] Add Python e2e Test#3369
fantix merged 3 commits into
mainfrom 
fantix/python-conformance
Conversation
@fantix
@fantixfantix
commented
Aug 6, 2026•

edited

Loading

Copy link


Copy Markdown

Member
Description
Turns packages/core/e2e/e2e.test.ts into a cross-language conformance suite and adds workbench/python as its first non-JavaScript subject.
The gate. Tests whose subject is the JavaScript implementation itself — the compiler, @workflow/next, the CLI's TS codepaths — are marked testJsOnly / describeJsOnly. Everything else is in scope for any language and is gated on a per-app e2e-conformance.json:
fieldmeaning
languageanything but javascript skips the JS-only tests
fixtureswhich workflows/99_e2e fixtures the app implements
unsupportedindividual tests the app's runtime cannot pass yet, mapped to why
Both axes are ratchets, which is the point — a fixture listed but missing from the deployed manifest is a hard failure, not a skip, and an unsupported entry naming a test that no longer exists fails too. An app can only move forward by editing the declaration, and the declaration is the thing review looks at.
Apps that ship no e2e-conformance.json — every existing one — are reported as "JavaScript, all fixtures present", so this is a no-op for the JS matrix. Format reference: packages/core/e2e/e2e-conformance.example.json.
The Python app.workbench/python is a bare ASGI app on vercel-py, pinned by git rev rather than a release so the suite reports on what is actually being built. It serves the .well-known/workflow/v1 surface itself — the Python SDK does not claim to — and runs against both world-local (over HTTP) and a real Vercel deployment (over the CBOR queue transport, where the only HTTP route the driver touches is manifest.json).
Six fixtures are ported as a demo of the framework to start with: addTen, nullByte, parallelSleep, promiseAll, sleepInLoop, sleeping.
How did you test your changes?
Two CI lanes, both added here:
laneresult
E2E Python Conformance (world-local)8 passed, 129 skipped, of 137
E2E Vercel Prod Tests (python - node)7 of 156 when the race below goes our way, 6 when it does not
One baseline, not two: the extra 19 collected on Vercel are e2e-agent.test.ts, which that lane picks up and skips whole, and the eighth local pass is deploymentId: 'latest' is a no-op in non-Vercel worlds.
E2E Python Conformance is deliberately its own job rather than a getTestMatrix row. Every step in the two local lanes is JS toolchain — pnpm dev / pnpm start, the canary next patch, dev.test.ts and local-build.test.ts, the per-framework DEV_TEST_CONFIG of compiler-generated paths — and none of it has a Python counterpart. The e2e-vercel-prod lane, whose rows only need a URL, does carry python as an ordinary matrix row.
PR Checklist - Required to merge
  •  📦 pnpm changeset was run to create a changelog for this PR
  •  🔒 DCO sign-off passes
  •  📝 Ping @vercel/workflow in a comment once the PR is ready, and the above checklist is complete
@changeset-bot
changeset-botBot
commented
Aug 6, 2026•

edited

Loading

Copy link


Copy Markdown

🦋 Changeset detected
Latest commit: e4569c0
The changes in this PR will be included in the next version bump.
This PR includes changesets to release 0 packages
When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types
Not sure what this means? Click here to learn what changesets are.
Click here if you're a maintainer who wants to add another changeset to this PR
@vercel
vercelBot
commented
Aug 6, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
The latest updates on your projects. Learn more about Vercel for GitHub.
ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreviewAug 14, 2026 5:23pm
example-nextjs-workflow-webpackReadyReadyPreviewAug 14, 2026 5:23pm
example-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-astro-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-express-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-fastify-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-hono-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-nestjs-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-nitro-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-nuxt-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-python-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-sveltekit-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-tanstack-start-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-vite-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workflow-swc-playgroundReadyReadyPreviewAug 14, 2026 5:23pm
workflow-tarballsReadyReadyPreviewAug 14, 2026 5:23pm
workflow-webReadyReadyPreviewAug 14, 2026 5:23pm
1 Skipped Deployment
ProjectDeploymentActionsUpdated (UTC)
workflow-docsSkippedSkippedAug 14, 2026 5:23pm
@github-actions
github-actionsBot
commented
Aug 6, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
🧪 E2E Test Results
✅ All tests passed
E2E Test Summary
Summary
PassedFailedSkippedTotal
✅ ▲ Vercel Production347407384212
✅ 💻 Local Development381005584368
✅ 📦 Local Production381005584368
✅ 🐘 Local Postgres381005584368
✅ 🪟 Windows31200312
✅ 🌐 Cross-language Conformance90128137
✅ vercel-multi-region270027
Total152520254017792
Details by Category
✅ ▲ Vercel Production
AppPassedFailedSkipped
✅ astro-node128028
✅ astro-quickjs128028
✅ example-node128028
✅ example-quickjs128028
✅ express-node128028
✅ express-quickjs128028
✅ fastify-node128028
✅ fastify-quickjs128028
✅ hono-node128028
✅ hono-quickjs128028
✅ nest-node128028
✅ nest-quickjs128028
✅ nextjs-turbopack-node15303
✅ nextjs-turbopack-quickjs15303
✅ nextjs-webpack-node15303
✅ nextjs-webpack-quickjs15303
✅ nitro-node128028
✅ nitro-quickjs128028
✅ nuxt-node128028
✅ nuxt-quickjs128028
✅ python-node80148
✅ sveltekit-node14709
✅ sveltekit-quickjs14709
✅ tanstack-start-node128028
✅ tanstack-start-quickjs128028
✅ vite-node128028
✅ vite-quickjs128028
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack-node15600
✅ nextjs-turbopack-quickjs15600
✅ 🌐 Cross-language Conformance
AppPassedFailedSkipped
✅ python90128
✅ vercel-multi-region
AppPassedFailedSkipped
✅ nextjs-turbopack2700
📋 View full workflow run
@github-actions
github-actionsBot
commented
Aug 6, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
📊 Workflow Benchmarks
commit e4569c0 · Fri, 14 Aug 2026 17:40:20 GMT · run logs
Backend: vercel · app: nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep392 (+62%) 🔻1359 🔴 (+17%) 🔻1368 🔴 (+13%)1554 🔴 (+20%) 🔻30
TTFSstream1298 (+409%) 🔻1372 🔴 (+21%) 🔻1398 🔴 (+18%) 🔻1425 🔴 (+15%) 🔻30
TTFShook + stream501 (+14%)1664 🔴 (+18%) 🔻1751 🔴 (+21%) 🔻1980 🔴 (+28%) 🔻30
Fan-out TTFSPromise.all(100 steps)8855 (-1.1%)9050 (-12%)9256 (-16%) 💚10345 (-91%) 💚10
Fan-out TTLSPromise.all(100 steps)17348 (-2.3%)17542 (-8.1%)17682 (-10%)20103 (-84%) 💚10
STSO1020 steps (inline)117 (-17%) 💚171 (-19%) 💚194 (-23%) 💚281 (-50%) 💚1019
WO1020 steps169733 (-21%) 💚169733 (-21%) 💚169733 (-21%) 💚169733 (-21%) 💚1
SLstream latency89 (-17%) 💚126 🔴 (-16%) 💚140 🔴 (-22%) 💚152 🔴 (-82%) 💚30
SOstream overhead (text)110 (-13%)251 🔴 (-11%)477 (+14%)4679 🔴 (+426%) 🔻30
SOstream overhead (structured)100 (-28%) 💚160 (-56%) 💚168 (-75%) 💚260 (-77%) 💚30
📈 STSO distribution vs main (inline / queue-hop histograms)
1020 steps (inline)
Cumulative STSO time: main 214388ms → this run 169546ms (Δ -44842ms, -21%)
 100-150 ms █░░░┃ main 11 this 171 +160
150-200 ms ████████████████████░░░┃ main 653 this 766 +113
200-250 ms █┃██████ main 248 this 64 -184
250-300 ms ┃█ main 54 this 9 -45
300-350 ms ┃ main 17 this 5 -12
350-400 ms ┃ main 14 this 3 -11
400-450 ms ┃ main 4 this 1 -3
450-500 ms ┃ main 4 this 0 -4
500-550 ms ┃ main 3 this 0 -3
550-600 ms ┃ main 3 this 0 -3
600-650 ms ┃ main 2 this 0 -2
650-700 ms ┃ main 2 this 0 -2
750-800 ms ┃ main 1 this 0 -1
850-900 ms ┃ main 1 this 0 -1
1000-1050 ms ┃ main 1 this 0 -1
5100-5150 ms ┃ main 1 this 0 -1

📜 Previous results (1)
cbf7209
Fri, 14 Aug 2026 17:06:46 GMT · run logs
vercel / nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep1385 (+96%) 🔻1511 🔴 (+50%) 🔻1534 🔴 (+51%) 🔻1665 🔴 (+10%)30
TTFSstream1357 (+42%) 🔻1437 🔴 (+45%) 🔻1469 🔴 (+45%) 🔻1529 🔴 (+45%) 🔻30
TTFShook + stream1601 (+32%) 🔻1713 🔴 (+33%) 🔻1750 🔴 (+31%) 🔻2367 🔴 (+47%) 🔻30
Fan-out TTFSPromise.all(100 steps)891510634107101469310
Fan-out TTLSPromise.all(100 steps)1735019396194152452310
STSO1020 steps (inline)1351832133951019
WO1020 steps182912 (-53%) 💚182912 (-53%) 💚182912 (-53%) 💚182912 (-53%) 💚1
SLstream latency98 (+21%) 🔻134 🔴 (+1.5%)154 🔴 (+10%)293 🔴 (+64%) 🔻30
SOstream overhead (text)125 (+24%) 🔻186 (+3.3%)210 (+4.0%)275 (+10%)30
SOstream overhead (structured)126 (+27%) 🔻198 (+22%) 🔻273 (+40%) 🔻465 (+114%) 🔻30
ℹ️ Metric definitions & methodology
The collapsed STSO distribution section above buckets every step gap of the sequential-steps run (not a sampled window), split by whether the step ending the gap ran inline — in the same warm process as the step before it, so the gap is pure framework overhead — or after a queue-hop — the first step of a fresh process, which pays queue dispatch, client reinit and event-log replay. Bars overlay the two runs: █ is main, ┃ marks where this run lands, ░ bridges the gap when this run has more samples in a bucket.
Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.
Metrics — TTFS: time to first step body (in-deployment start() → first step body, deployment clocks) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · SL: stream latency (in-deployment write → read propagation, readAt - writtenAt) · SO: stream overhead (end-to-end write+consume time beyond the modelled generation window)
Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · stream latency: parallel reader/writer steps on a dedicated stream; SL is the in-deployment write->read propagation (readAt - writtenAt) · stream overhead (text): writer streams 300 variable-length text token deltas paced at 100/s for 3s (a haiku-size LLM's token throughput) while a parallel reader drains the whole stream; SO is the end-to-end write+consume time beyond the 3s generation window (overhead/backpressure) · stream overhead (structured): same workload as stream overhead (text), but each delta is an AI-SDK-style structured object ({ type: 'text-delta', id, text }) instead of a raw string, so the SO gap vs the text scenario is the added serialization cost
🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · SO 250/500/1000
All metrics are measured from deployment-side timestamps only. Runs are triggered by an in-deployment route that stamps the anchor (clientStart) right before start(), so the CI runner’s request and its path through api.vercel.com sit outside every measured window. TTFS = in-deployment start() → first step body (turbo uses the in-process fast path, non-turbo the dispatch path), and includes the VQS dispatch hop plus any /flow cold start. Fan-out TTFS/TTLS are the first and last step completions of a single Promise.all over trivial steps, from the same anchor, so the gap between the two rows is the spread the runtime adds across the fan-out. STSO/WO are measured between step bodies on the deployment. SL is measured inside the workflow (parallel reader/writer steps), so it no longer includes the api.vercel.com read path.
Cold starts are kept in the numbers on purpose — they are part of real bursty-workload latency. The workbench deployment cold-starts the /flow invocation for a large fraction of runs, inflating P75+; the Best column shows the fastest (warm-start) sample for comparison.
@socket-security
socket-securityBot
commented
Aug 6, 2026•

edited

Loading

Copy link


Copy Markdown

Review the following changes in direct dependencies. Learn more about Socket for GitHub.
DiffPackageSupply Chain
Security		VulnerabilityQualityMaintenanceLicense
Addedpypi/​vercel@​0.10.097100100100100
Addedpypi/​uvicorn@​0.52.398100100100100
View full report
@vercel
vercelBottemporarily deployed
to
Preview – workflow-docs
August 6, 2026 03:00
Inactive

vercel[bot]
vercelBot
reviewed
Aug 11, 2026
View reviewed changes
Comment threadworkbench/python/uv.lock
Outdated

@github-actions
github-actionsBot
commented
Aug 11, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Sim World
Simulated world deterministic testing for races. Traces
🟠 Mint-ordered log — 3 fail of 41 total
log=mint-ordered · fence=per-spec
scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
❌ in-flight-before-decisionfailed91.0mMISMATCH1
❌ in-flight-before-decision-countedfailed91.0mMISMATCH1
❌ in-flight-after-decisionfailed91.0mMISMATCH1
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0
Full trace: world-sim-mint.txt
🟢 Append-only log — 0 fail of 41 total
log=append-only · fence=per-spec
scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
✅ in-flight-before-decisioncompleted171.0mok0
✅ in-flight-before-decision-countedcompleted171.0mok0
✅ in-flight-after-decisioncompleted192.0mok0
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0
Full trace: world-sim-append-only.txt
@fantix@claude
[e2e] Run the e2e suite against a Python app
672209f
Turns `packages/core/e2e/e2e.test.ts` into a cross-language conformance
suite and adds `workbench/python` as its first non-JavaScript subject.
Tests whose subject is the JavaScript implementation are marked
`testJsOnly` / `describeJsOnly`; everything else is gated on a per-app
`e2e-conformance.json` declaring which `workflows/99_e2e` fixtures the
app implements, plus any individual tests its runtime cannot pass yet.
Apps without that file — every existing one — are unaffected, so this
is a no-op for the JS matrix.
The Python app is built on vercel-py, pinned by git rev rather than a
release so the suite reports on what is actually being built. It runs
against both `world-local` and a real Vercel deployment, the latter
through the CBOR queue transport rather than HTTP. Baseline is 8 of 137
passing locally and 7 of 156 on Vercel; the rest skip, and the point of
the lane is that they keep skipping for the declared reason.
`workbench/python/README.md` records what is and is not ported, and the
four gaps found on the way: the CBOR queue transport (vercel-py#265,
#266), the Trusted Sources bypass header (vercel-py#278), `encr`
payload decryption (vercel-py#279), and reading the slot-numbered
`specVersion: 6` this repo's world-vercel now stamps (#3389). All four
are fixed in the pinned rev.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@fantix
fantix
commented
Aug 12, 2026

Copy link


Copy Markdown

MemberAuthor
@vercel/workflow this PR is ready.
One test is flaky due to the missing implementation of resilient start in Python, which is being added in vercel/vercel-py#284. This PR should be slightly updated after that fix, but I think the majority of this PR is ready for review.
@fantixfantix
mentioned this pull request
Aug 12, 2026
 Draft

VaguelySerious
VaguelySerious
approved these changes
Aug 12, 2026
View reviewed changes
@VaguelySeriousVaguelySerious
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!
@fantix
Merge remote-tracking branch 'origin/main' into fantix/python-conform…
b5b690e
…ance
@fantix
[e2e] Track the vercel-py runtime's move to its own package
e4569c0
vercel-py#299 split the workflow runtime out of `vercel` into
`vercel-workflow`, so the pin now names one package instead of two —
`vercel-queue` 0.8.0 is released and carries the CBOR queue codec that
forced the second entry. The import namespace is unchanged; the
`_internal` paths are not (`vercel.workflow._internal`). `cryptography`
comes along outright rather than behind an `encryption` extra this app had
to remember to ask for.
What the app *declares* deliberately does not follow the split. Depending
on `vercel-workflow` — the only thing imported here — takes the Vercel lane
down: `@vercel/python` picks queue mode over workers mode by looking for
the literal name `vercel` in `[project].dependencies` and then reading
`importlib.metadata.version("vercel")`, and no released builder has heard
of the new name (checked through 6.56.2). Falling back writes the trigger
for consumer `app_registry` on `__wkf_*` instead of `default` on
`__wkf_workflow_*`, which is quieter than it sounds: deliveries do not
500, they never arrive, and all 8 conformance tests time out having logged
zero requests. So the umbrella package stays declared and pulls the
runtime (`vercel-workflow>=0.9.0,<1.1.0`) until the builder learns the
split. The cost is that the runtime resolves from PyPI rather than the
rev; at this rev that is exact, and the note above the source entry says
how to re-check it on the next bump.
Two API changes come with the bump. `HTTPRequest` now also requires
`method` and `url`, which is what lets `workflow_entrypoint` answer the
`?__health` probe itself (vercel-py#292, matching `withHealthCheck` in
`@workflow/core` down to the CORS headers) — so the hand-rolled
`{"status": "ok"}` here is gone and `app.py` only routes. The three
health-check tests stay JS-only, but the reason has changed: vercel-py
implements both probes and omits `workflowCoreVersion`, which the driver
asserts, on the grounds that it names a JavaScript package's version.
The bump also empties `unsupported`. vercel-py#282 carries `runInput`
through, #284 bootstraps the run from `run_started` instead of reading the
row first, and #283 gives `world-local` the same resilient start, so
*resilient start: addTenWorkflow completes when run_created returns 500*
passes and the local baseline is 9 of 137. That was never confined to the
one test: `start()` writes `run_created` and pushes the queue message in
parallel on purpose, so a consumer that 500s on a missing row loses an
arbitrary run per suite on Vercel — a different fixture each time, which
is why it read as flakiness for two rounds. The README records it as the
last of the Vercel-lane gaps.
@github-actions
github-actionsBot
commented
Aug 14, 2026

Copy link


Copy Markdown

Contributor
No backport to stable for 09b299a (AI decision).
This adds new capability rather than fixing a defect: a cross-language conformance gating mechanism (e2e-conformance.json, testJsOnly/describeJsOnly, new @vitest/runner dependency), a brand-new workbench/python app, and two new CI lanes. It is explicitly a no-op for the existing JavaScript workbench apps, so it fixes nothing that is broken on stable and only expands test surface area on a maintenance branch.
To override, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:
09b299a03e4c4cac3b5ad630625f4f83c1dd908b

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@vercelvercel[bot]vercel[bot] left review comments
@VaguelySeriousVaguelySeriousVaguelySerious approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@fantix@VaguelySerious

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Auto-enable theater mode on YouTube
(function() {
function tryTheater() {
var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]');
if (btn && !btn.classList.contains('activated')) {
btn.click();
}
}
// Try immediately
tryTheater();
// Try after navigation (SPA)
var lastUrl = location.href;
setInterval(function() {
if (location.href !== lastUrl) {
lastUrl = location.href;
setTimeout(tryTheater, 500);
}
}, 1000);
// Also try on player load
var observer = new MutationObserver(tryTheater);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
[e2e] Add Python e2e Test by fantix · Pull Request #3369 · vercel/workflow · GitHub
Skip to content
[e2e] Add Python e2e Test - #3369
Merged
fantix merged 3 commits into
mainfrom 
fantix/python-conformance
Aug 14, 2026
Merged
[e2e] Add Python e2e Test#3369
fantix merged 3 commits into
mainfrom 
fantix/python-conformance
Conversation
@fantix
@fantixfantix
commented
Aug 6, 2026•

edited

Loading

Copy link


Copy Markdown

Member
Description
Turns packages/core/e2e/e2e.test.ts into a cross-language conformance suite and adds workbench/python as its first non-JavaScript subject.
The gate. Tests whose subject is the JavaScript implementation itself — the compiler, @workflow/next, the CLI's TS codepaths — are marked testJsOnly / describeJsOnly. Everything else is in scope for any language and is gated on a per-app e2e-conformance.json:
fieldmeaning
languageanything but javascript skips the JS-only tests
fixtureswhich workflows/99_e2e fixtures the app implements
unsupportedindividual tests the app's runtime cannot pass yet, mapped to why
Both axes are ratchets, which is the point — a fixture listed but missing from the deployed manifest is a hard failure, not a skip, and an unsupported entry naming a test that no longer exists fails too. An app can only move forward by editing the declaration, and the declaration is the thing review looks at.
Apps that ship no e2e-conformance.json — every existing one — are reported as "JavaScript, all fixtures present", so this is a no-op for the JS matrix. Format reference: packages/core/e2e/e2e-conformance.example.json.
The Python app.workbench/python is a bare ASGI app on vercel-py, pinned by git rev rather than a release so the suite reports on what is actually being built. It serves the .well-known/workflow/v1 surface itself — the Python SDK does not claim to — and runs against both world-local (over HTTP) and a real Vercel deployment (over the CBOR queue transport, where the only HTTP route the driver touches is manifest.json).
Six fixtures are ported as a demo of the framework to start with: addTen, nullByte, parallelSleep, promiseAll, sleepInLoop, sleeping.
How did you test your changes?
Two CI lanes, both added here:
laneresult
E2E Python Conformance (world-local)8 passed, 129 skipped, of 137
E2E Vercel Prod Tests (python - node)7 of 156 when the race below goes our way, 6 when it does not
One baseline, not two: the extra 19 collected on Vercel are e2e-agent.test.ts, which that lane picks up and skips whole, and the eighth local pass is deploymentId: 'latest' is a no-op in non-Vercel worlds.
E2E Python Conformance is deliberately its own job rather than a getTestMatrix row. Every step in the two local lanes is JS toolchain — pnpm dev / pnpm start, the canary next patch, dev.test.ts and local-build.test.ts, the per-framework DEV_TEST_CONFIG of compiler-generated paths — and none of it has a Python counterpart. The e2e-vercel-prod lane, whose rows only need a URL, does carry python as an ordinary matrix row.
PR Checklist - Required to merge
  •  📦 pnpm changeset was run to create a changelog for this PR
  •  🔒 DCO sign-off passes
  •  📝 Ping @vercel/workflow in a comment once the PR is ready, and the above checklist is complete
@changeset-bot
changeset-botBot
commented
Aug 6, 2026•

edited

Loading

Copy link


Copy Markdown

🦋 Changeset detected
Latest commit: e4569c0
The changes in this PR will be included in the next version bump.
This PR includes changesets to release 0 packages
When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types
Not sure what this means? Click here to learn what changesets are.
Click here if you're a maintainer who wants to add another changeset to this PR
@vercel
vercelBot
commented
Aug 6, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
The latest updates on your projects. Learn more about Vercel for GitHub.
ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreviewAug 14, 2026 5:23pm
example-nextjs-workflow-webpackReadyReadyPreviewAug 14, 2026 5:23pm
example-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-astro-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-express-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-fastify-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-hono-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-nestjs-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-nitro-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-nuxt-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-python-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-sveltekit-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-tanstack-start-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-vite-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workflow-swc-playgroundReadyReadyPreviewAug 14, 2026 5:23pm
workflow-tarballsReadyReadyPreviewAug 14, 2026 5:23pm
workflow-webReadyReadyPreviewAug 14, 2026 5:23pm
1 Skipped Deployment
ProjectDeploymentActionsUpdated (UTC)
workflow-docsSkippedSkippedAug 14, 2026 5:23pm
@github-actions
github-actionsBot
commented
Aug 6, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
🧪 E2E Test Results
✅ All tests passed
E2E Test Summary
Summary
PassedFailedSkippedTotal
✅ ▲ Vercel Production347407384212
✅ 💻 Local Development381005584368
✅ 📦 Local Production381005584368
✅ 🐘 Local Postgres381005584368
✅ 🪟 Windows31200312
✅ 🌐 Cross-language Conformance90128137
✅ vercel-multi-region270027
Total152520254017792
Details by Category
✅ ▲ Vercel Production
AppPassedFailedSkipped
✅ astro-node128028
✅ astro-quickjs128028
✅ example-node128028
✅ example-quickjs128028
✅ express-node128028
✅ express-quickjs128028
✅ fastify-node128028
✅ fastify-quickjs128028
✅ hono-node128028
✅ hono-quickjs128028
✅ nest-node128028
✅ nest-quickjs128028
✅ nextjs-turbopack-node15303
✅ nextjs-turbopack-quickjs15303
✅ nextjs-webpack-node15303
✅ nextjs-webpack-quickjs15303
✅ nitro-node128028
✅ nitro-quickjs128028
✅ nuxt-node128028
✅ nuxt-quickjs128028
✅ python-node80148
✅ sveltekit-node14709
✅ sveltekit-quickjs14709
✅ tanstack-start-node128028
✅ tanstack-start-quickjs128028
✅ vite-node128028
✅ vite-quickjs128028
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack-node15600
✅ nextjs-turbopack-quickjs15600
✅ 🌐 Cross-language Conformance
AppPassedFailedSkipped
✅ python90128
✅ vercel-multi-region
AppPassedFailedSkipped
✅ nextjs-turbopack2700
📋 View full workflow run
@github-actions
github-actionsBot
commented
Aug 6, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
📊 Workflow Benchmarks
commit e4569c0 · Fri, 14 Aug 2026 17:40:20 GMT · run logs
Backend: vercel · app: nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep392 (+62%) 🔻1359 🔴 (+17%) 🔻1368 🔴 (+13%)1554 🔴 (+20%) 🔻30
TTFSstream1298 (+409%) 🔻1372 🔴 (+21%) 🔻1398 🔴 (+18%) 🔻1425 🔴 (+15%) 🔻30
TTFShook + stream501 (+14%)1664 🔴 (+18%) 🔻1751 🔴 (+21%) 🔻1980 🔴 (+28%) 🔻30
Fan-out TTFSPromise.all(100 steps)8855 (-1.1%)9050 (-12%)9256 (-16%) 💚10345 (-91%) 💚10
Fan-out TTLSPromise.all(100 steps)17348 (-2.3%)17542 (-8.1%)17682 (-10%)20103 (-84%) 💚10
STSO1020 steps (inline)117 (-17%) 💚171 (-19%) 💚194 (-23%) 💚281 (-50%) 💚1019
WO1020 steps169733 (-21%) 💚169733 (-21%) 💚169733 (-21%) 💚169733 (-21%) 💚1
SLstream latency89 (-17%) 💚126 🔴 (-16%) 💚140 🔴 (-22%) 💚152 🔴 (-82%) 💚30
SOstream overhead (text)110 (-13%)251 🔴 (-11%)477 (+14%)4679 🔴 (+426%) 🔻30
SOstream overhead (structured)100 (-28%) 💚160 (-56%) 💚168 (-75%) 💚260 (-77%) 💚30
📈 STSO distribution vs main (inline / queue-hop histograms)
1020 steps (inline)
Cumulative STSO time: main 214388ms → this run 169546ms (Δ -44842ms, -21%)
 100-150 ms █░░░┃ main 11 this 171 +160
150-200 ms ████████████████████░░░┃ main 653 this 766 +113
200-250 ms █┃██████ main 248 this 64 -184
250-300 ms ┃█ main 54 this 9 -45
300-350 ms ┃ main 17 this 5 -12
350-400 ms ┃ main 14 this 3 -11
400-450 ms ┃ main 4 this 1 -3
450-500 ms ┃ main 4 this 0 -4
500-550 ms ┃ main 3 this 0 -3
550-600 ms ┃ main 3 this 0 -3
600-650 ms ┃ main 2 this 0 -2
650-700 ms ┃ main 2 this 0 -2
750-800 ms ┃ main 1 this 0 -1
850-900 ms ┃ main 1 this 0 -1
1000-1050 ms ┃ main 1 this 0 -1
5100-5150 ms ┃ main 1 this 0 -1

📜 Previous results (1)
cbf7209
Fri, 14 Aug 2026 17:06:46 GMT · run logs
vercel / nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep1385 (+96%) 🔻1511 🔴 (+50%) 🔻1534 🔴 (+51%) 🔻1665 🔴 (+10%)30
TTFSstream1357 (+42%) 🔻1437 🔴 (+45%) 🔻1469 🔴 (+45%) 🔻1529 🔴 (+45%) 🔻30
TTFShook + stream1601 (+32%) 🔻1713 🔴 (+33%) 🔻1750 🔴 (+31%) 🔻2367 🔴 (+47%) 🔻30
Fan-out TTFSPromise.all(100 steps)891510634107101469310
Fan-out TTLSPromise.all(100 steps)1735019396194152452310
STSO1020 steps (inline)1351832133951019
WO1020 steps182912 (-53%) 💚182912 (-53%) 💚182912 (-53%) 💚182912 (-53%) 💚1
SLstream latency98 (+21%) 🔻134 🔴 (+1.5%)154 🔴 (+10%)293 🔴 (+64%) 🔻30
SOstream overhead (text)125 (+24%) 🔻186 (+3.3%)210 (+4.0%)275 (+10%)30
SOstream overhead (structured)126 (+27%) 🔻198 (+22%) 🔻273 (+40%) 🔻465 (+114%) 🔻30
ℹ️ Metric definitions & methodology
The collapsed STSO distribution section above buckets every step gap of the sequential-steps run (not a sampled window), split by whether the step ending the gap ran inline — in the same warm process as the step before it, so the gap is pure framework overhead — or after a queue-hop — the first step of a fresh process, which pays queue dispatch, client reinit and event-log replay. Bars overlay the two runs: █ is main, ┃ marks where this run lands, ░ bridges the gap when this run has more samples in a bucket.
Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.
Metrics — TTFS: time to first step body (in-deployment start() → first step body, deployment clocks) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · SL: stream latency (in-deployment write → read propagation, readAt - writtenAt) · SO: stream overhead (end-to-end write+consume time beyond the modelled generation window)
Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · stream latency: parallel reader/writer steps on a dedicated stream; SL is the in-deployment write->read propagation (readAt - writtenAt) · stream overhead (text): writer streams 300 variable-length text token deltas paced at 100/s for 3s (a haiku-size LLM's token throughput) while a parallel reader drains the whole stream; SO is the end-to-end write+consume time beyond the 3s generation window (overhead/backpressure) · stream overhead (structured): same workload as stream overhead (text), but each delta is an AI-SDK-style structured object ({ type: 'text-delta', id, text }) instead of a raw string, so the SO gap vs the text scenario is the added serialization cost
🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · SO 250/500/1000
All metrics are measured from deployment-side timestamps only. Runs are triggered by an in-deployment route that stamps the anchor (clientStart) right before start(), so the CI runner’s request and its path through api.vercel.com sit outside every measured window. TTFS = in-deployment start() → first step body (turbo uses the in-process fast path, non-turbo the dispatch path), and includes the VQS dispatch hop plus any /flow cold start. Fan-out TTFS/TTLS are the first and last step completions of a single Promise.all over trivial steps, from the same anchor, so the gap between the two rows is the spread the runtime adds across the fan-out. STSO/WO are measured between step bodies on the deployment. SL is measured inside the workflow (parallel reader/writer steps), so it no longer includes the api.vercel.com read path.
Cold starts are kept in the numbers on purpose — they are part of real bursty-workload latency. The workbench deployment cold-starts the /flow invocation for a large fraction of runs, inflating P75+; the Best column shows the fastest (warm-start) sample for comparison.
@socket-security
socket-securityBot
commented
Aug 6, 2026•

edited

Loading

Copy link


Copy Markdown

Review the following changes in direct dependencies. Learn more about Socket for GitHub.
DiffPackageSupply Chain
Security		VulnerabilityQualityMaintenanceLicense
Addedpypi/​vercel@​0.10.097100100100100
Addedpypi/​uvicorn@​0.52.398100100100100
View full report
@vercel
vercelBottemporarily deployed
to
Preview – workflow-docs
August 6, 2026 03:00
Inactive

vercel[bot]
vercelBot
reviewed
Aug 11, 2026
View reviewed changes
Comment threadworkbench/python/uv.lock
Outdated

@github-actions
github-actionsBot
commented
Aug 11, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Sim World
Simulated world deterministic testing for races. Traces
🟠 Mint-ordered log — 3 fail of 41 total
log=mint-ordered · fence=per-spec
scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
❌ in-flight-before-decisionfailed91.0mMISMATCH1
❌ in-flight-before-decision-countedfailed91.0mMISMATCH1
❌ in-flight-after-decisionfailed91.0mMISMATCH1
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0
Full trace: world-sim-mint.txt
🟢 Append-only log — 0 fail of 41 total
log=append-only · fence=per-spec
scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
✅ in-flight-before-decisioncompleted171.0mok0
✅ in-flight-before-decision-countedcompleted171.0mok0
✅ in-flight-after-decisioncompleted192.0mok0
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0
Full trace: world-sim-append-only.txt
@fantix@claude
[e2e] Run the e2e suite against a Python app
672209f
Turns `packages/core/e2e/e2e.test.ts` into a cross-language conformance
suite and adds `workbench/python` as its first non-JavaScript subject.
Tests whose subject is the JavaScript implementation are marked
`testJsOnly` / `describeJsOnly`; everything else is gated on a per-app
`e2e-conformance.json` declaring which `workflows/99_e2e` fixtures the
app implements, plus any individual tests its runtime cannot pass yet.
Apps without that file — every existing one — are unaffected, so this
is a no-op for the JS matrix.
The Python app is built on vercel-py, pinned by git rev rather than a
release so the suite reports on what is actually being built. It runs
against both `world-local` and a real Vercel deployment, the latter
through the CBOR queue transport rather than HTTP. Baseline is 8 of 137
passing locally and 7 of 156 on Vercel; the rest skip, and the point of
the lane is that they keep skipping for the declared reason.
`workbench/python/README.md` records what is and is not ported, and the
four gaps found on the way: the CBOR queue transport (vercel-py#265,
#266), the Trusted Sources bypass header (vercel-py#278), `encr`
payload decryption (vercel-py#279), and reading the slot-numbered
`specVersion: 6` this repo's world-vercel now stamps (#3389). All four
are fixed in the pinned rev.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@fantix
fantix
commented
Aug 12, 2026

Copy link


Copy Markdown

MemberAuthor
@vercel/workflow this PR is ready.
One test is flaky due to the missing implementation of resilient start in Python, which is being added in vercel/vercel-py#284. This PR should be slightly updated after that fix, but I think the majority of this PR is ready for review.
@fantixfantix
mentioned this pull request
Aug 12, 2026
 Draft

VaguelySerious
VaguelySerious
approved these changes
Aug 12, 2026
View reviewed changes
@VaguelySeriousVaguelySerious
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!
@fantix
Merge remote-tracking branch 'origin/main' into fantix/python-conform…
b5b690e
…ance
@fantix
[e2e] Track the vercel-py runtime's move to its own package
e4569c0
vercel-py#299 split the workflow runtime out of `vercel` into
`vercel-workflow`, so the pin now names one package instead of two —
`vercel-queue` 0.8.0 is released and carries the CBOR queue codec that
forced the second entry. The import namespace is unchanged; the
`_internal` paths are not (`vercel.workflow._internal`). `cryptography`
comes along outright rather than behind an `encryption` extra this app had
to remember to ask for.
What the app *declares* deliberately does not follow the split. Depending
on `vercel-workflow` — the only thing imported here — takes the Vercel lane
down: `@vercel/python` picks queue mode over workers mode by looking for
the literal name `vercel` in `[project].dependencies` and then reading
`importlib.metadata.version("vercel")`, and no released builder has heard
of the new name (checked through 6.56.2). Falling back writes the trigger
for consumer `app_registry` on `__wkf_*` instead of `default` on
`__wkf_workflow_*`, which is quieter than it sounds: deliveries do not
500, they never arrive, and all 8 conformance tests time out having logged
zero requests. So the umbrella package stays declared and pulls the
runtime (`vercel-workflow>=0.9.0,<1.1.0`) until the builder learns the
split. The cost is that the runtime resolves from PyPI rather than the
rev; at this rev that is exact, and the note above the source entry says
how to re-check it on the next bump.
Two API changes come with the bump. `HTTPRequest` now also requires
`method` and `url`, which is what lets `workflow_entrypoint` answer the
`?__health` probe itself (vercel-py#292, matching `withHealthCheck` in
`@workflow/core` down to the CORS headers) — so the hand-rolled
`{"status": "ok"}` here is gone and `app.py` only routes. The three
health-check tests stay JS-only, but the reason has changed: vercel-py
implements both probes and omits `workflowCoreVersion`, which the driver
asserts, on the grounds that it names a JavaScript package's version.
The bump also empties `unsupported`. vercel-py#282 carries `runInput`
through, #284 bootstraps the run from `run_started` instead of reading the
row first, and #283 gives `world-local` the same resilient start, so
*resilient start: addTenWorkflow completes when run_created returns 500*
passes and the local baseline is 9 of 137. That was never confined to the
one test: `start()` writes `run_created` and pushes the queue message in
parallel on purpose, so a consumer that 500s on a missing row loses an
arbitrary run per suite on Vercel — a different fixture each time, which
is why it read as flakiness for two rounds. The README records it as the
last of the Vercel-lane gaps.
@github-actions
github-actionsBot
commented
Aug 14, 2026

Copy link


Copy Markdown

Contributor
No backport to stable for 09b299a (AI decision).
This adds new capability rather than fixing a defect: a cross-language conformance gating mechanism (e2e-conformance.json, testJsOnly/describeJsOnly, new @vitest/runner dependency), a brand-new workbench/python app, and two new CI lanes. It is explicitly a no-op for the existing JavaScript workbench apps, so it fixes nothing that is broken on stable and only expands test surface area on a maintenance branch.
To override, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:
09b299a03e4c4cac3b5ad630625f4f83c1dd908b

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@vercelvercel[bot]vercel[bot] left review comments
@VaguelySeriousVaguelySeriousVaguelySerious approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@fantix@VaguelySerious

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Remove or un-stick sticky/fixed headers that block content
(function() {
function unstick() {
document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) {
if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {
el.style.position = 'static';
el.style.top = 'auto';
el.style.zIndex = 'auto';
}
});
}
unstick();
var observer = new MutationObserver(unstick);
observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });
})();
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
[e2e] Add Python e2e Test by fantix · Pull Request #3369 · vercel/workflow · GitHub
Skip to content
[e2e] Add Python e2e Test - #3369
Merged
fantix merged 3 commits into
mainfrom 
fantix/python-conformance
Aug 14, 2026
Merged
[e2e] Add Python e2e Test#3369
fantix merged 3 commits into
mainfrom 
fantix/python-conformance
Conversation
@fantix
@fantixfantix
commented
Aug 6, 2026•

edited

Loading

Copy link


Copy Markdown

Member
Description
Turns packages/core/e2e/e2e.test.ts into a cross-language conformance suite and adds workbench/python as its first non-JavaScript subject.
The gate. Tests whose subject is the JavaScript implementation itself — the compiler, @workflow/next, the CLI's TS codepaths — are marked testJsOnly / describeJsOnly. Everything else is in scope for any language and is gated on a per-app e2e-conformance.json:
fieldmeaning
languageanything but javascript skips the JS-only tests
fixtureswhich workflows/99_e2e fixtures the app implements
unsupportedindividual tests the app's runtime cannot pass yet, mapped to why
Both axes are ratchets, which is the point — a fixture listed but missing from the deployed manifest is a hard failure, not a skip, and an unsupported entry naming a test that no longer exists fails too. An app can only move forward by editing the declaration, and the declaration is the thing review looks at.
Apps that ship no e2e-conformance.json — every existing one — are reported as "JavaScript, all fixtures present", so this is a no-op for the JS matrix. Format reference: packages/core/e2e/e2e-conformance.example.json.
The Python app.workbench/python is a bare ASGI app on vercel-py, pinned by git rev rather than a release so the suite reports on what is actually being built. It serves the .well-known/workflow/v1 surface itself — the Python SDK does not claim to — and runs against both world-local (over HTTP) and a real Vercel deployment (over the CBOR queue transport, where the only HTTP route the driver touches is manifest.json).
Six fixtures are ported as a demo of the framework to start with: addTen, nullByte, parallelSleep, promiseAll, sleepInLoop, sleeping.
How did you test your changes?
Two CI lanes, both added here:
laneresult
E2E Python Conformance (world-local)8 passed, 129 skipped, of 137
E2E Vercel Prod Tests (python - node)7 of 156 when the race below goes our way, 6 when it does not
One baseline, not two: the extra 19 collected on Vercel are e2e-agent.test.ts, which that lane picks up and skips whole, and the eighth local pass is deploymentId: 'latest' is a no-op in non-Vercel worlds.
E2E Python Conformance is deliberately its own job rather than a getTestMatrix row. Every step in the two local lanes is JS toolchain — pnpm dev / pnpm start, the canary next patch, dev.test.ts and local-build.test.ts, the per-framework DEV_TEST_CONFIG of compiler-generated paths — and none of it has a Python counterpart. The e2e-vercel-prod lane, whose rows only need a URL, does carry python as an ordinary matrix row.
PR Checklist - Required to merge
  •  📦 pnpm changeset was run to create a changelog for this PR
  •  🔒 DCO sign-off passes
  •  📝 Ping @vercel/workflow in a comment once the PR is ready, and the above checklist is complete
@changeset-bot
changeset-botBot
commented
Aug 6, 2026•

edited

Loading

Copy link


Copy Markdown

🦋 Changeset detected
Latest commit: e4569c0
The changes in this PR will be included in the next version bump.
This PR includes changesets to release 0 packages
When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types
Not sure what this means? Click here to learn what changesets are.
Click here if you're a maintainer who wants to add another changeset to this PR
@vercel
vercelBot
commented
Aug 6, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
The latest updates on your projects. Learn more about Vercel for GitHub.
ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreviewAug 14, 2026 5:23pm
example-nextjs-workflow-webpackReadyReadyPreviewAug 14, 2026 5:23pm
example-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-astro-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-express-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-fastify-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-hono-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-nestjs-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-nitro-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-nuxt-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-python-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-sveltekit-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-tanstack-start-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-vite-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workflow-swc-playgroundReadyReadyPreviewAug 14, 2026 5:23pm
workflow-tarballsReadyReadyPreviewAug 14, 2026 5:23pm
workflow-webReadyReadyPreviewAug 14, 2026 5:23pm
1 Skipped Deployment
ProjectDeploymentActionsUpdated (UTC)
workflow-docsSkippedSkippedAug 14, 2026 5:23pm
@github-actions
github-actionsBot
commented
Aug 6, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
🧪 E2E Test Results
✅ All tests passed
E2E Test Summary
Summary
PassedFailedSkippedTotal
✅ ▲ Vercel Production347407384212
✅ 💻 Local Development381005584368
✅ 📦 Local Production381005584368
✅ 🐘 Local Postgres381005584368
✅ 🪟 Windows31200312
✅ 🌐 Cross-language Conformance90128137
✅ vercel-multi-region270027
Total152520254017792
Details by Category
✅ ▲ Vercel Production
AppPassedFailedSkipped
✅ astro-node128028
✅ astro-quickjs128028
✅ example-node128028
✅ example-quickjs128028
✅ express-node128028
✅ express-quickjs128028
✅ fastify-node128028
✅ fastify-quickjs128028
✅ hono-node128028
✅ hono-quickjs128028
✅ nest-node128028
✅ nest-quickjs128028
✅ nextjs-turbopack-node15303
✅ nextjs-turbopack-quickjs15303
✅ nextjs-webpack-node15303
✅ nextjs-webpack-quickjs15303
✅ nitro-node128028
✅ nitro-quickjs128028
✅ nuxt-node128028
✅ nuxt-quickjs128028
✅ python-node80148
✅ sveltekit-node14709
✅ sveltekit-quickjs14709
✅ tanstack-start-node128028
✅ tanstack-start-quickjs128028
✅ vite-node128028
✅ vite-quickjs128028
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack-node15600
✅ nextjs-turbopack-quickjs15600
✅ 🌐 Cross-language Conformance
AppPassedFailedSkipped
✅ python90128
✅ vercel-multi-region
AppPassedFailedSkipped
✅ nextjs-turbopack2700
📋 View full workflow run
@github-actions
github-actionsBot
commented
Aug 6, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
📊 Workflow Benchmarks
commit e4569c0 · Fri, 14 Aug 2026 17:40:20 GMT · run logs
Backend: vercel · app: nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep392 (+62%) 🔻1359 🔴 (+17%) 🔻1368 🔴 (+13%)1554 🔴 (+20%) 🔻30
TTFSstream1298 (+409%) 🔻1372 🔴 (+21%) 🔻1398 🔴 (+18%) 🔻1425 🔴 (+15%) 🔻30
TTFShook + stream501 (+14%)1664 🔴 (+18%) 🔻1751 🔴 (+21%) 🔻1980 🔴 (+28%) 🔻30
Fan-out TTFSPromise.all(100 steps)8855 (-1.1%)9050 (-12%)9256 (-16%) 💚10345 (-91%) 💚10
Fan-out TTLSPromise.all(100 steps)17348 (-2.3%)17542 (-8.1%)17682 (-10%)20103 (-84%) 💚10
STSO1020 steps (inline)117 (-17%) 💚171 (-19%) 💚194 (-23%) 💚281 (-50%) 💚1019
WO1020 steps169733 (-21%) 💚169733 (-21%) 💚169733 (-21%) 💚169733 (-21%) 💚1
SLstream latency89 (-17%) 💚126 🔴 (-16%) 💚140 🔴 (-22%) 💚152 🔴 (-82%) 💚30
SOstream overhead (text)110 (-13%)251 🔴 (-11%)477 (+14%)4679 🔴 (+426%) 🔻30
SOstream overhead (structured)100 (-28%) 💚160 (-56%) 💚168 (-75%) 💚260 (-77%) 💚30
📈 STSO distribution vs main (inline / queue-hop histograms)
1020 steps (inline)
Cumulative STSO time: main 214388ms → this run 169546ms (Δ -44842ms, -21%)
 100-150 ms █░░░┃ main 11 this 171 +160
150-200 ms ████████████████████░░░┃ main 653 this 766 +113
200-250 ms █┃██████ main 248 this 64 -184
250-300 ms ┃█ main 54 this 9 -45
300-350 ms ┃ main 17 this 5 -12
350-400 ms ┃ main 14 this 3 -11
400-450 ms ┃ main 4 this 1 -3
450-500 ms ┃ main 4 this 0 -4
500-550 ms ┃ main 3 this 0 -3
550-600 ms ┃ main 3 this 0 -3
600-650 ms ┃ main 2 this 0 -2
650-700 ms ┃ main 2 this 0 -2
750-800 ms ┃ main 1 this 0 -1
850-900 ms ┃ main 1 this 0 -1
1000-1050 ms ┃ main 1 this 0 -1
5100-5150 ms ┃ main 1 this 0 -1

📜 Previous results (1)
cbf7209
Fri, 14 Aug 2026 17:06:46 GMT · run logs
vercel / nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep1385 (+96%) 🔻1511 🔴 (+50%) 🔻1534 🔴 (+51%) 🔻1665 🔴 (+10%)30
TTFSstream1357 (+42%) 🔻1437 🔴 (+45%) 🔻1469 🔴 (+45%) 🔻1529 🔴 (+45%) 🔻30
TTFShook + stream1601 (+32%) 🔻1713 🔴 (+33%) 🔻1750 🔴 (+31%) 🔻2367 🔴 (+47%) 🔻30
Fan-out TTFSPromise.all(100 steps)891510634107101469310
Fan-out TTLSPromise.all(100 steps)1735019396194152452310
STSO1020 steps (inline)1351832133951019
WO1020 steps182912 (-53%) 💚182912 (-53%) 💚182912 (-53%) 💚182912 (-53%) 💚1
SLstream latency98 (+21%) 🔻134 🔴 (+1.5%)154 🔴 (+10%)293 🔴 (+64%) 🔻30
SOstream overhead (text)125 (+24%) 🔻186 (+3.3%)210 (+4.0%)275 (+10%)30
SOstream overhead (structured)126 (+27%) 🔻198 (+22%) 🔻273 (+40%) 🔻465 (+114%) 🔻30
ℹ️ Metric definitions & methodology
The collapsed STSO distribution section above buckets every step gap of the sequential-steps run (not a sampled window), split by whether the step ending the gap ran inline — in the same warm process as the step before it, so the gap is pure framework overhead — or after a queue-hop — the first step of a fresh process, which pays queue dispatch, client reinit and event-log replay. Bars overlay the two runs: █ is main, ┃ marks where this run lands, ░ bridges the gap when this run has more samples in a bucket.
Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.
Metrics — TTFS: time to first step body (in-deployment start() → first step body, deployment clocks) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · SL: stream latency (in-deployment write → read propagation, readAt - writtenAt) · SO: stream overhead (end-to-end write+consume time beyond the modelled generation window)
Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · stream latency: parallel reader/writer steps on a dedicated stream; SL is the in-deployment write->read propagation (readAt - writtenAt) · stream overhead (text): writer streams 300 variable-length text token deltas paced at 100/s for 3s (a haiku-size LLM's token throughput) while a parallel reader drains the whole stream; SO is the end-to-end write+consume time beyond the 3s generation window (overhead/backpressure) · stream overhead (structured): same workload as stream overhead (text), but each delta is an AI-SDK-style structured object ({ type: 'text-delta', id, text }) instead of a raw string, so the SO gap vs the text scenario is the added serialization cost
🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · SO 250/500/1000
All metrics are measured from deployment-side timestamps only. Runs are triggered by an in-deployment route that stamps the anchor (clientStart) right before start(), so the CI runner’s request and its path through api.vercel.com sit outside every measured window. TTFS = in-deployment start() → first step body (turbo uses the in-process fast path, non-turbo the dispatch path), and includes the VQS dispatch hop plus any /flow cold start. Fan-out TTFS/TTLS are the first and last step completions of a single Promise.all over trivial steps, from the same anchor, so the gap between the two rows is the spread the runtime adds across the fan-out. STSO/WO are measured between step bodies on the deployment. SL is measured inside the workflow (parallel reader/writer steps), so it no longer includes the api.vercel.com read path.
Cold starts are kept in the numbers on purpose — they are part of real bursty-workload latency. The workbench deployment cold-starts the /flow invocation for a large fraction of runs, inflating P75+; the Best column shows the fastest (warm-start) sample for comparison.
@socket-security
socket-securityBot
commented
Aug 6, 2026•

edited

Loading

Copy link


Copy Markdown

Review the following changes in direct dependencies. Learn more about Socket for GitHub.
DiffPackageSupply Chain
Security		VulnerabilityQualityMaintenanceLicense
Addedpypi/​vercel@​0.10.097100100100100
Addedpypi/​uvicorn@​0.52.398100100100100
View full report
@vercel
vercelBottemporarily deployed
to
Preview – workflow-docs
August 6, 2026 03:00
Inactive

vercel[bot]
vercelBot
reviewed
Aug 11, 2026
View reviewed changes
Comment threadworkbench/python/uv.lock
Outdated

@github-actions
github-actionsBot
commented
Aug 11, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Sim World
Simulated world deterministic testing for races. Traces
🟠 Mint-ordered log — 3 fail of 41 total
log=mint-ordered · fence=per-spec
scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
❌ in-flight-before-decisionfailed91.0mMISMATCH1
❌ in-flight-before-decision-countedfailed91.0mMISMATCH1
❌ in-flight-after-decisionfailed91.0mMISMATCH1
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0
Full trace: world-sim-mint.txt
🟢 Append-only log — 0 fail of 41 total
log=append-only · fence=per-spec
scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
✅ in-flight-before-decisioncompleted171.0mok0
✅ in-flight-before-decision-countedcompleted171.0mok0
✅ in-flight-after-decisioncompleted192.0mok0
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0
Full trace: world-sim-append-only.txt
@fantix@claude
[e2e] Run the e2e suite against a Python app
672209f
Turns `packages/core/e2e/e2e.test.ts` into a cross-language conformance
suite and adds `workbench/python` as its first non-JavaScript subject.
Tests whose subject is the JavaScript implementation are marked
`testJsOnly` / `describeJsOnly`; everything else is gated on a per-app
`e2e-conformance.json` declaring which `workflows/99_e2e` fixtures the
app implements, plus any individual tests its runtime cannot pass yet.
Apps without that file — every existing one — are unaffected, so this
is a no-op for the JS matrix.
The Python app is built on vercel-py, pinned by git rev rather than a
release so the suite reports on what is actually being built. It runs
against both `world-local` and a real Vercel deployment, the latter
through the CBOR queue transport rather than HTTP. Baseline is 8 of 137
passing locally and 7 of 156 on Vercel; the rest skip, and the point of
the lane is that they keep skipping for the declared reason.
`workbench/python/README.md` records what is and is not ported, and the
four gaps found on the way: the CBOR queue transport (vercel-py#265,
#266), the Trusted Sources bypass header (vercel-py#278), `encr`
payload decryption (vercel-py#279), and reading the slot-numbered
`specVersion: 6` this repo's world-vercel now stamps (#3389). All four
are fixed in the pinned rev.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@fantix
fantix
commented
Aug 12, 2026

Copy link


Copy Markdown

MemberAuthor
@vercel/workflow this PR is ready.
One test is flaky due to the missing implementation of resilient start in Python, which is being added in vercel/vercel-py#284. This PR should be slightly updated after that fix, but I think the majority of this PR is ready for review.
@fantixfantix
mentioned this pull request
Aug 12, 2026
 Draft

VaguelySerious
VaguelySerious
approved these changes
Aug 12, 2026
View reviewed changes
@VaguelySeriousVaguelySerious
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!
@fantix
Merge remote-tracking branch 'origin/main' into fantix/python-conform…
b5b690e
…ance
@fantix
[e2e] Track the vercel-py runtime's move to its own package
e4569c0
vercel-py#299 split the workflow runtime out of `vercel` into
`vercel-workflow`, so the pin now names one package instead of two —
`vercel-queue` 0.8.0 is released and carries the CBOR queue codec that
forced the second entry. The import namespace is unchanged; the
`_internal` paths are not (`vercel.workflow._internal`). `cryptography`
comes along outright rather than behind an `encryption` extra this app had
to remember to ask for.
What the app *declares* deliberately does not follow the split. Depending
on `vercel-workflow` — the only thing imported here — takes the Vercel lane
down: `@vercel/python` picks queue mode over workers mode by looking for
the literal name `vercel` in `[project].dependencies` and then reading
`importlib.metadata.version("vercel")`, and no released builder has heard
of the new name (checked through 6.56.2). Falling back writes the trigger
for consumer `app_registry` on `__wkf_*` instead of `default` on
`__wkf_workflow_*`, which is quieter than it sounds: deliveries do not
500, they never arrive, and all 8 conformance tests time out having logged
zero requests. So the umbrella package stays declared and pulls the
runtime (`vercel-workflow>=0.9.0,<1.1.0`) until the builder learns the
split. The cost is that the runtime resolves from PyPI rather than the
rev; at this rev that is exact, and the note above the source entry says
how to re-check it on the next bump.
Two API changes come with the bump. `HTTPRequest` now also requires
`method` and `url`, which is what lets `workflow_entrypoint` answer the
`?__health` probe itself (vercel-py#292, matching `withHealthCheck` in
`@workflow/core` down to the CORS headers) — so the hand-rolled
`{"status": "ok"}` here is gone and `app.py` only routes. The three
health-check tests stay JS-only, but the reason has changed: vercel-py
implements both probes and omits `workflowCoreVersion`, which the driver
asserts, on the grounds that it names a JavaScript package's version.
The bump also empties `unsupported`. vercel-py#282 carries `runInput`
through, #284 bootstraps the run from `run_started` instead of reading the
row first, and #283 gives `world-local` the same resilient start, so
*resilient start: addTenWorkflow completes when run_created returns 500*
passes and the local baseline is 9 of 137. That was never confined to the
one test: `start()` writes `run_created` and pushes the queue message in
parallel on purpose, so a consumer that 500s on a missing row loses an
arbitrary run per suite on Vercel — a different fixture each time, which
is why it read as flakiness for two rounds. The README records it as the
last of the Vercel-lane gaps.
@github-actions
github-actionsBot
commented
Aug 14, 2026

Copy link


Copy Markdown

Contributor
No backport to stable for 09b299a (AI decision).
This adds new capability rather than fixing a defect: a cross-language conformance gating mechanism (e2e-conformance.json, testJsOnly/describeJsOnly, new @vitest/runner dependency), a brand-new workbench/python app, and two new CI lanes. It is explicitly a no-op for the existing JavaScript workbench apps, so it fixes nothing that is broken on stable and only expands test surface area on a maintenance branch.
To override, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:
09b299a03e4c4cac3b5ad630625f4f83c1dd908b

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@vercelvercel[bot]vercel[bot] left review comments
@VaguelySeriousVaguelySeriousVaguelySerious approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@fantix@VaguelySerious

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Universal Dark Mode - works on any site
(function() {
var enabled = true;
function applyDarkMode() {
if (!enabled) return;
// Create style element if it doesn't exist
var style = document.getElementById('universal-dark-mode-style');
if (!style) {
style = document.createElement('style');
style.id = 'universal-dark-mode-style';
document.head.appendChild(style);
}
// Dark mode CSS - inverts colors but preserves images/video
style.textContent = '
/* Invert everything except media */
html {
filter: invert(1) hue-rotate(180deg) !important;
background: #1a1a2e !important;
}
/* Restore images, videos, iframes, canvas */
img, video, iframe, canvas, svg, picture, [style*="background-image"] {
filter: invert(1) hue-rotate(180deg) !important;
}
/* Preserve specific elements that should not be inverted */
.no-dark-mode, .no-dark-mode *,
[data-theme="light"], [data-theme="light"],
.ace_editor, .ace_editor *,
.CodeMirror, .CodeMirror *,
.monaco-editor, .monaco-editor *,
.markdown-body pre, .markdown-body pre *,
.highlight, .highlight *,
pre code, pre code * {
filter: none !important;
}
/* Fix common UI elements */
.modal, .popup, .dropdown-menu, .tooltip, .popover {
filter: invert(1) hue-rotate(180deg) !important;
background: #2d2d44 !important;
border-color: #444 !important;
}
/* Scrollbars */
::-webkit-scrollbar { background: #1a1a2e !important; }
::-webkit-scrollbar-thumb { background: #444 !important; }
::-webkit-scrollbar-thumb:hover { background: #555 !important; }
/* Selection */
::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
';
}
function removeDarkMode() {
var style = document.getElementById('universal-dark-mode-style');
if (style) style.remove();
}
// Toggle with Alt+Shift+D
document.addEventListener('keydown', function(e) {
if (e.altKey && e.shiftKey && e.key === 'D') {
e.preventDefault();
enabled = !enabled;
if (enabled) {
applyDarkMode();
console.log('[Universal Dark Mode] Enabled');
} else {
removeDarkMode();
console.log('[Universal Dark Mode] Disabled');
}
}
});
// Apply on load
applyDarkMode();
// Re-apply on dynamic content
var observer = new MutationObserver(function(mutations) {
if (enabled && !document.getElementById('universal-dark-mode-style')) {
applyDarkMode();
}
});
observer.observe(document.head, { childList: true });
console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');
})();
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();
[e2e] Add Python e2e Test by fantix · Pull Request #3369 · vercel/workflow · GitHub
Skip to content
[e2e] Add Python e2e Test - #3369
Merged
fantix merged 3 commits into
mainfrom 
fantix/python-conformance
Aug 14, 2026
Merged
[e2e] Add Python e2e Test#3369
fantix merged 3 commits into
mainfrom 
fantix/python-conformance
Conversation
@fantix
@fantixfantix
commented
Aug 6, 2026•

edited

Loading

Copy link


Copy Markdown

Member
Description
Turns packages/core/e2e/e2e.test.ts into a cross-language conformance suite and adds workbench/python as its first non-JavaScript subject.
The gate. Tests whose subject is the JavaScript implementation itself — the compiler, @workflow/next, the CLI's TS codepaths — are marked testJsOnly / describeJsOnly. Everything else is in scope for any language and is gated on a per-app e2e-conformance.json:
fieldmeaning
languageanything but javascript skips the JS-only tests
fixtureswhich workflows/99_e2e fixtures the app implements
unsupportedindividual tests the app's runtime cannot pass yet, mapped to why
Both axes are ratchets, which is the point — a fixture listed but missing from the deployed manifest is a hard failure, not a skip, and an unsupported entry naming a test that no longer exists fails too. An app can only move forward by editing the declaration, and the declaration is the thing review looks at.
Apps that ship no e2e-conformance.json — every existing one — are reported as "JavaScript, all fixtures present", so this is a no-op for the JS matrix. Format reference: packages/core/e2e/e2e-conformance.example.json.
The Python app.workbench/python is a bare ASGI app on vercel-py, pinned by git rev rather than a release so the suite reports on what is actually being built. It serves the .well-known/workflow/v1 surface itself — the Python SDK does not claim to — and runs against both world-local (over HTTP) and a real Vercel deployment (over the CBOR queue transport, where the only HTTP route the driver touches is manifest.json).
Six fixtures are ported as a demo of the framework to start with: addTen, nullByte, parallelSleep, promiseAll, sleepInLoop, sleeping.
How did you test your changes?
Two CI lanes, both added here:
laneresult
E2E Python Conformance (world-local)8 passed, 129 skipped, of 137
E2E Vercel Prod Tests (python - node)7 of 156 when the race below goes our way, 6 when it does not
One baseline, not two: the extra 19 collected on Vercel are e2e-agent.test.ts, which that lane picks up and skips whole, and the eighth local pass is deploymentId: 'latest' is a no-op in non-Vercel worlds.
E2E Python Conformance is deliberately its own job rather than a getTestMatrix row. Every step in the two local lanes is JS toolchain — pnpm dev / pnpm start, the canary next patch, dev.test.ts and local-build.test.ts, the per-framework DEV_TEST_CONFIG of compiler-generated paths — and none of it has a Python counterpart. The e2e-vercel-prod lane, whose rows only need a URL, does carry python as an ordinary matrix row.
PR Checklist - Required to merge
  •  📦 pnpm changeset was run to create a changelog for this PR
  •  🔒 DCO sign-off passes
  •  📝 Ping @vercel/workflow in a comment once the PR is ready, and the above checklist is complete
@changeset-bot
changeset-botBot
commented
Aug 6, 2026•

edited

Loading

Copy link


Copy Markdown

🦋 Changeset detected
Latest commit: e4569c0
The changes in this PR will be included in the next version bump.
This PR includes changesets to release 0 packages
When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types
Not sure what this means? Click here to learn what changesets are.
Click here if you're a maintainer who wants to add another changeset to this PR
@vercel
vercelBot
commented
Aug 6, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
The latest updates on your projects. Learn more about Vercel for GitHub.
ProjectDeploymentActionsUpdated (UTC)
example-nextjs-workflow-turbopackReadyReadyPreviewAug 14, 2026 5:23pm
example-nextjs-workflow-webpackReadyReadyPreviewAug 14, 2026 5:23pm
example-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-astro-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-express-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-fastify-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-hono-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-nestjs-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-nitro-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-nuxt-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-python-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-sveltekit-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-tanstack-start-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workbench-vite-workflowReadyReadyPreviewAug 14, 2026 5:23pm
workflow-swc-playgroundReadyReadyPreviewAug 14, 2026 5:23pm
workflow-tarballsReadyReadyPreviewAug 14, 2026 5:23pm
workflow-webReadyReadyPreviewAug 14, 2026 5:23pm
1 Skipped Deployment
ProjectDeploymentActionsUpdated (UTC)
workflow-docsSkippedSkippedAug 14, 2026 5:23pm
@github-actions
github-actionsBot
commented
Aug 6, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
🧪 E2E Test Results
✅ All tests passed
E2E Test Summary
Summary
PassedFailedSkippedTotal
✅ ▲ Vercel Production347407384212
✅ 💻 Local Development381005584368
✅ 📦 Local Production381005584368
✅ 🐘 Local Postgres381005584368
✅ 🪟 Windows31200312
✅ 🌐 Cross-language Conformance90128137
✅ vercel-multi-region270027
Total152520254017792
Details by Category
✅ ▲ Vercel Production
AppPassedFailedSkipped
✅ astro-node128028
✅ astro-quickjs128028
✅ example-node128028
✅ example-quickjs128028
✅ express-node128028
✅ express-quickjs128028
✅ fastify-node128028
✅ fastify-quickjs128028
✅ hono-node128028
✅ hono-quickjs128028
✅ nest-node128028
✅ nest-quickjs128028
✅ nextjs-turbopack-node15303
✅ nextjs-turbopack-quickjs15303
✅ nextjs-webpack-node15303
✅ nextjs-webpack-quickjs15303
✅ nitro-node128028
✅ nitro-quickjs128028
✅ nuxt-node128028
✅ nuxt-quickjs128028
✅ python-node80148
✅ sveltekit-node14709
✅ sveltekit-quickjs14709
✅ tanstack-start-node128028
✅ tanstack-start-quickjs128028
✅ vite-node128028
✅ vite-quickjs128028
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable-node130026
✅ astro-stable-quickjs130026
✅ express-stable-node130026
✅ express-stable-quickjs130026
✅ fastify-stable-node130026
✅ fastify-stable-quickjs130026
✅ hono-stable-node130026
✅ hono-stable-quickjs130026
✅ nest-stable-node130026
✅ nest-stable-quickjs130026
✅ nextjs-turbopack-canary-node137019
✅ nextjs-turbopack-canary-quickjs137019
✅ nextjs-turbopack-stable-node15600
✅ nextjs-turbopack-stable-quickjs15600
✅ nextjs-webpack-canary-node137019
✅ nextjs-webpack-canary-quickjs137019
✅ nextjs-webpack-stable-node15600
✅ nextjs-webpack-stable-quickjs15600
✅ nitro-stable-node130026
✅ nitro-stable-quickjs130026
✅ nuxt-stable-node130026
✅ nuxt-stable-quickjs130026
✅ sveltekit-stable-node14907
✅ sveltekit-stable-quickjs14907
✅ tanstack-start-node130026
✅ tanstack-start-quickjs130026
✅ vite-stable-node130026
✅ vite-stable-quickjs130026
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack-node15600
✅ nextjs-turbopack-quickjs15600
✅ 🌐 Cross-language Conformance
AppPassedFailedSkipped
✅ python90128
✅ vercel-multi-region
AppPassedFailedSkipped
✅ nextjs-turbopack2700
📋 View full workflow run
@github-actions
github-actionsBot
commented
Aug 6, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
📊 Workflow Benchmarks
commit e4569c0 · Fri, 14 Aug 2026 17:40:20 GMT · run logs
Backend: vercel · app: nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep392 (+62%) 🔻1359 🔴 (+17%) 🔻1368 🔴 (+13%)1554 🔴 (+20%) 🔻30
TTFSstream1298 (+409%) 🔻1372 🔴 (+21%) 🔻1398 🔴 (+18%) 🔻1425 🔴 (+15%) 🔻30
TTFShook + stream501 (+14%)1664 🔴 (+18%) 🔻1751 🔴 (+21%) 🔻1980 🔴 (+28%) 🔻30
Fan-out TTFSPromise.all(100 steps)8855 (-1.1%)9050 (-12%)9256 (-16%) 💚10345 (-91%) 💚10
Fan-out TTLSPromise.all(100 steps)17348 (-2.3%)17542 (-8.1%)17682 (-10%)20103 (-84%) 💚10
STSO1020 steps (inline)117 (-17%) 💚171 (-19%) 💚194 (-23%) 💚281 (-50%) 💚1019
WO1020 steps169733 (-21%) 💚169733 (-21%) 💚169733 (-21%) 💚169733 (-21%) 💚1
SLstream latency89 (-17%) 💚126 🔴 (-16%) 💚140 🔴 (-22%) 💚152 🔴 (-82%) 💚30
SOstream overhead (text)110 (-13%)251 🔴 (-11%)477 (+14%)4679 🔴 (+426%) 🔻30
SOstream overhead (structured)100 (-28%) 💚160 (-56%) 💚168 (-75%) 💚260 (-77%) 💚30
📈 STSO distribution vs main (inline / queue-hop histograms)
1020 steps (inline)
Cumulative STSO time: main 214388ms → this run 169546ms (Δ -44842ms, -21%)
 100-150 ms █░░░┃ main 11 this 171 +160
150-200 ms ████████████████████░░░┃ main 653 this 766 +113
200-250 ms █┃██████ main 248 this 64 -184
250-300 ms ┃█ main 54 this 9 -45
300-350 ms ┃ main 17 this 5 -12
350-400 ms ┃ main 14 this 3 -11
400-450 ms ┃ main 4 this 1 -3
450-500 ms ┃ main 4 this 0 -4
500-550 ms ┃ main 3 this 0 -3
550-600 ms ┃ main 3 this 0 -3
600-650 ms ┃ main 2 this 0 -2
650-700 ms ┃ main 2 this 0 -2
750-800 ms ┃ main 1 this 0 -1
850-900 ms ┃ main 1 this 0 -1
1000-1050 ms ┃ main 1 this 0 -1
5100-5150 ms ┃ main 1 this 0 -1

📜 Previous results (1)
cbf7209
Fri, 14 Aug 2026 17:06:46 GMT · run logs
vercel / nextjs-turbopack
MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep1385 (+96%) 🔻1511 🔴 (+50%) 🔻1534 🔴 (+51%) 🔻1665 🔴 (+10%)30
TTFSstream1357 (+42%) 🔻1437 🔴 (+45%) 🔻1469 🔴 (+45%) 🔻1529 🔴 (+45%) 🔻30
TTFShook + stream1601 (+32%) 🔻1713 🔴 (+33%) 🔻1750 🔴 (+31%) 🔻2367 🔴 (+47%) 🔻30
Fan-out TTFSPromise.all(100 steps)891510634107101469310
Fan-out TTLSPromise.all(100 steps)1735019396194152452310
STSO1020 steps (inline)1351832133951019
WO1020 steps182912 (-53%) 💚182912 (-53%) 💚182912 (-53%) 💚182912 (-53%) 💚1
SLstream latency98 (+21%) 🔻134 🔴 (+1.5%)154 🔴 (+10%)293 🔴 (+64%) 🔻30
SOstream overhead (text)125 (+24%) 🔻186 (+3.3%)210 (+4.0%)275 (+10%)30
SOstream overhead (structured)126 (+27%) 🔻198 (+22%) 🔻273 (+40%) 🔻465 (+114%) 🔻30
ℹ️ Metric definitions & methodology
The collapsed STSO distribution section above buckets every step gap of the sequential-steps run (not a sampled window), split by whether the step ending the gap ran inline — in the same warm process as the step before it, so the gap is pure framework overhead — or after a queue-hop — the first step of a fresh process, which pays queue dispatch, client reinit and event-log replay. Bars overlay the two runs: █ is main, ┃ marks where this run lands, ░ bridges the gap when this run has more samples in a bucket.
Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.
Metrics — TTFS: time to first step body (in-deployment start() → first step body, deployment clocks) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · SL: stream latency (in-deployment write → read propagation, readAt - writtenAt) · SO: stream overhead (end-to-end write+consume time beyond the modelled generation window)
Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · stream latency: parallel reader/writer steps on a dedicated stream; SL is the in-deployment write->read propagation (readAt - writtenAt) · stream overhead (text): writer streams 300 variable-length text token deltas paced at 100/s for 3s (a haiku-size LLM's token throughput) while a parallel reader drains the whole stream; SO is the end-to-end write+consume time beyond the 3s generation window (overhead/backpressure) · stream overhead (structured): same workload as stream overhead (text), but each delta is an AI-SDK-style structured object ({ type: 'text-delta', id, text }) instead of a raw string, so the SO gap vs the text scenario is the added serialization cost
🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · SO 250/500/1000
All metrics are measured from deployment-side timestamps only. Runs are triggered by an in-deployment route that stamps the anchor (clientStart) right before start(), so the CI runner’s request and its path through api.vercel.com sit outside every measured window. TTFS = in-deployment start() → first step body (turbo uses the in-process fast path, non-turbo the dispatch path), and includes the VQS dispatch hop plus any /flow cold start. Fan-out TTFS/TTLS are the first and last step completions of a single Promise.all over trivial steps, from the same anchor, so the gap between the two rows is the spread the runtime adds across the fan-out. STSO/WO are measured between step bodies on the deployment. SL is measured inside the workflow (parallel reader/writer steps), so it no longer includes the api.vercel.com read path.
Cold starts are kept in the numbers on purpose — they are part of real bursty-workload latency. The workbench deployment cold-starts the /flow invocation for a large fraction of runs, inflating P75+; the Best column shows the fastest (warm-start) sample for comparison.
@socket-security
socket-securityBot
commented
Aug 6, 2026•

edited

Loading

Copy link


Copy Markdown

Review the following changes in direct dependencies. Learn more about Socket for GitHub.
DiffPackageSupply Chain
Security		VulnerabilityQualityMaintenanceLicense
Addedpypi/​vercel@​0.10.097100100100100
Addedpypi/​uvicorn@​0.52.398100100100100
View full report
@vercel
vercelBottemporarily deployed
to
Preview – workflow-docs
August 6, 2026 03:00
Inactive

vercel[bot]
vercelBot
reviewed
Aug 11, 2026
View reviewed changes
Comment threadworkbench/python/uv.lock
Outdated

@github-actions
github-actionsBot
commented
Aug 11, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Sim World
Simulated world deterministic testing for races. Traces
🟠 Mint-ordered log — 3 fail of 41 total
log=mint-ordered · fence=per-spec
scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
❌ in-flight-before-decisionfailed91.0mMISMATCH1
❌ in-flight-before-decision-countedfailed91.0mMISMATCH1
❌ in-flight-after-decisionfailed91.0mMISMATCH1
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0
Full trace: world-sim-mint.txt
🟢 Append-only log — 0 fail of 41 total
log=append-only · fence=per-spec
scenariooutcomeeventsvirtreplayviolations
✅ smoke-no-stepscompleted30msok0
✅ smoke-one-stepcompleted60msok0
✅ hook-at-step-startedcompleted120msok0
✅ hook-at-step-completedcompleted120msok0
✅ hook-at-hook-createdcompleted120msok0
✅ deadline-hook-winscompleted71.0hok0
✅ deadline-expirescompleted71.0hok0
✅ long-sleepcompleted1130.0dok0
✅ hook-never-arrivesstalled30msskipped0
✅ step-retries-twicecompleted102.0sok0
✅ parallel-stepscompleted90msok0
✅ hook-on-execution-statecompleted120msok0
✅ peek-hook-before-branchcompleted120msok0
✅ peek-hook-after-branchcompleted120msok0
✅ peek-hook-at-registrationcompleted120msok0
✅ race-hook-before-probecompleted120msok0
✅ race-hook-after-probecompleted120msok0
✅ race-duplicate-deliverycompleted130msok0
✅ attr-hook-before-stepcompleted110msok0
✅ attr-hook-after-stepcompleted110msok0
✅ attr-from-step-bodycompleted130msok0
✅ fork-hook-after-timeoutcompleted141.0mok0
✅ fork-hook-before-timeoutcompleted141.0mok0
✅ count-hook-after-timeoutcompleted171.0mok0
✅ count-hook-before-timeoutcompleted201.0mok0
✅ stale-read-step-count-forkcompleted201.0mok0
✅ stale-read-equal-step-countscompleted141.0mok0
✅ step-vs-step-forkcompleted120msok0
✅ step-vs-step-fork-fencedcompleted120msok0
✅ fence-catches-benign-directioncompleted125msok0
✅ in-flight-before-decisioncompleted171.0mok0
✅ in-flight-before-decision-countedcompleted171.0mok0
✅ in-flight-after-decisioncompleted192.0mok0
✅ stale-read-step-count-fork-fencedcompleted201.0mok0
✅ fork-hook-winscompleted131.0mok0
✅ fork-timeout-winscompleted131.0mok0
✅ unclaimed-payload-under-forkcompleted171.0mok0
✅ claimed-payload-under-forkcompleted171.0mok0
✅ writers-independent-step-bodiescompleted120msok0
✅ writers-scripted-tempocompleted120msok0
✅ cancel-mid-stepcancelled70msskipped0
Full trace: world-sim-append-only.txt
@fantix@claude
[e2e] Run the e2e suite against a Python app
672209f
Turns `packages/core/e2e/e2e.test.ts` into a cross-language conformance
suite and adds `workbench/python` as its first non-JavaScript subject.
Tests whose subject is the JavaScript implementation are marked
`testJsOnly` / `describeJsOnly`; everything else is gated on a per-app
`e2e-conformance.json` declaring which `workflows/99_e2e` fixtures the
app implements, plus any individual tests its runtime cannot pass yet.
Apps without that file — every existing one — are unaffected, so this
is a no-op for the JS matrix.
The Python app is built on vercel-py, pinned by git rev rather than a
release so the suite reports on what is actually being built. It runs
against both `world-local` and a real Vercel deployment, the latter
through the CBOR queue transport rather than HTTP. Baseline is 8 of 137
passing locally and 7 of 156 on Vercel; the rest skip, and the point of
the lane is that they keep skipping for the declared reason.
`workbench/python/README.md` records what is and is not ported, and the
four gaps found on the way: the CBOR queue transport (vercel-py#265,
#266), the Trusted Sources bypass header (vercel-py#278), `encr`
payload decryption (vercel-py#279), and reading the slot-numbered
`specVersion: 6` this repo's world-vercel now stamps (#3389). All four
are fixed in the pinned rev.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@fantix
fantix
commented
Aug 12, 2026

Copy link


Copy Markdown

MemberAuthor
@vercel/workflow this PR is ready.
One test is flaky due to the missing implementation of resilient start in Python, which is being added in vercel/vercel-py#284. This PR should be slightly updated after that fix, but I think the majority of this PR is ready for review.
@fantixfantix
mentioned this pull request
Aug 12, 2026
 Draft

VaguelySerious
VaguelySerious
approved these changes
Aug 12, 2026
View reviewed changes
@VaguelySeriousVaguelySerious
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!
@fantix
Merge remote-tracking branch 'origin/main' into fantix/python-conform…
b5b690e
…ance
@fantix
[e2e] Track the vercel-py runtime's move to its own package
e4569c0
vercel-py#299 split the workflow runtime out of `vercel` into
`vercel-workflow`, so the pin now names one package instead of two —
`vercel-queue` 0.8.0 is released and carries the CBOR queue codec that
forced the second entry. The import namespace is unchanged; the
`_internal` paths are not (`vercel.workflow._internal`). `cryptography`
comes along outright rather than behind an `encryption` extra this app had
to remember to ask for.
What the app *declares* deliberately does not follow the split. Depending
on `vercel-workflow` — the only thing imported here — takes the Vercel lane
down: `@vercel/python` picks queue mode over workers mode by looking for
the literal name `vercel` in `[project].dependencies` and then reading
`importlib.metadata.version("vercel")`, and no released builder has heard
of the new name (checked through 6.56.2). Falling back writes the trigger
for consumer `app_registry` on `__wkf_*` instead of `default` on
`__wkf_workflow_*`, which is quieter than it sounds: deliveries do not
500, they never arrive, and all 8 conformance tests time out having logged
zero requests. So the umbrella package stays declared and pulls the
runtime (`vercel-workflow>=0.9.0,<1.1.0`) until the builder learns the
split. The cost is that the runtime resolves from PyPI rather than the
rev; at this rev that is exact, and the note above the source entry says
how to re-check it on the next bump.
Two API changes come with the bump. `HTTPRequest` now also requires
`method` and `url`, which is what lets `workflow_entrypoint` answer the
`?__health` probe itself (vercel-py#292, matching `withHealthCheck` in
`@workflow/core` down to the CORS headers) — so the hand-rolled
`{"status": "ok"}` here is gone and `app.py` only routes. The three
health-check tests stay JS-only, but the reason has changed: vercel-py
implements both probes and omits `workflowCoreVersion`, which the driver
asserts, on the grounds that it names a JavaScript package's version.
The bump also empties `unsupported`. vercel-py#282 carries `runInput`
through, #284 bootstraps the run from `run_started` instead of reading the
row first, and #283 gives `world-local` the same resilient start, so
*resilient start: addTenWorkflow completes when run_created returns 500*
passes and the local baseline is 9 of 137. That was never confined to the
one test: `start()` writes `run_created` and pushes the queue message in
parallel on purpose, so a consumer that 500s on a missing row loses an
arbitrary run per suite on Vercel — a different fixture each time, which
is why it read as flakiness for two rounds. The README records it as the
last of the Vercel-lane gaps.
@github-actions
github-actionsBot
commented
Aug 14, 2026

Copy link


Copy Markdown

Contributor
No backport to stable for 09b299a (AI decision).
This adds new capability rather than fixing a defect: a cross-language conformance gating mechanism (e2e-conformance.json, testJsOnly/describeJsOnly, new @vitest/runner dependency), a brand-new workbench/python app, and two new CI lanes. It is explicitly a no-op for the existing JavaScript workbench apps, so it fixes nothing that is broken on stable and only expands test surface area on a maintenance branch.
To override, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:
09b299a03e4c4cac3b5ad630625f4f83c1dd908b

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@vercelvercel[bot]vercel[bot] left review comments
@VaguelySeriousVaguelySeriousVaguelySerious approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@fantix@VaguelySerious