Skip to content

[ci] Enable NestJS e2e-vercel-prod and add to docs as "experimental" - #3011

Merged
VaguelySerious merged 4 commits into
mainfrom
peter/nest-e2e-vercel-prod
Jul 20, 2026
Merged

[ci] Enable NestJS e2e-vercel-prod and add to docs as "experimental"#3011
VaguelySerious merged 4 commits into
mainfrom
peter/nest-e2e-vercel-prod

Conversation

@VaguelySerious

@VaguelySeriousVaguelySerious commented Jul 20, 2026

Copy link
Copy Markdown
Member

Note CI is failing due to world.analytics issue unrelated to this PR.

This PR enables the Vercel production e2e suite against the NestJS workbench by adding a nest entry to the e2e-vercel-prod matrix in .github/workflows/tests.yml:

- name: "nest"project-id: "prj_AV3YrzooDhMlvfmZybqeugHRk48Y"project-slug: "workbench-nestjs-workflow"

Follow-up to #2988 (which added NestJS Vercel Build Output support). That PR proved a NestJS app runs on Vercel; this wires the workbench/nest app into the same prod e2e lane every other framework runs in.

Why no other changes are needed

  • Harness already supports nest:packages/core/e2e/utils.ts already carves out nest in hasStepSourceMaps() / hasWorkflowSourceMaps(), and e2e.test.ts / e2e-agent.test.ts (the only files the prod job runs) handle non-Next / non-agent apps through the generic World + .well-known/workflow/v1/* path.
  • Workbench already parity with express:workbench/nest exposes POST /api/test-direct-step-call, the full example workflow set, and the flow/webhook/manifest endpoints via workflow-nest build --vercel. vercel.json sets WORKFLOW_PUBLIC_MANIFEST=1 like the others.
  • Vercel project:workbench-nestjs-workflow (root dir workbench/nest, prod branch main) is connected to this repo; its Git integration was reconnected recently, so this PR's preview deploy is the first fresh build.
  • No package code changes → no changeset required (changeset status clean).

Enables the Vercel production e2e suite against the NestJS workbench, using
the `workbench-nestjs-workflow` Vercel project (root `workbench/nest`).
No test-code changes are required: `packages/core/e2e/utils.ts` already carves
out `nest` for step/workflow source maps, and `e2e.test.ts`/`e2e-agent.test.ts`
handle non-Next/non-agent apps through the generic World + `.well-known/workflow`
path. The nest workbench already exposes `POST /api/test-direct-step-call`, the
full example workflow set, and the flow/webhook/manifest endpoints via
`workflow-nest build --vercel`.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Peter Wielander <peter.wielander@vercel.com>
@vercel

vercelBot commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

@github-actions

github-actionsBot commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

🧪 E2E Test Results

Some tests failed

Summary

PassedFailedSkippedTotal
❌ ▲ Vercel Production1398552301683
✅ 💻 Local Development161702191836
✅ 📦 Local Production161702191836
✅ 🐘 Local Postgres161702191836
✅ 🪟 Windows15300153
❌ 📋 Other1010102041224
✅ vercel-multi-region270027
Total74396510918595

❌ Failed Tests

▲ Vercel Production (55 failed)

astro (5 failed):

  • error handling retry behavior regular Error retries until success
  • error handling retry behavior FatalError fails immediately without retries
  • error handling not registered StepNotRegisteredError fails the step but workflow can catch it
  • stepFunctionPassingWorkflow - step function references can be passed as arguments (without closure vars) | wrun_41KY0Q2AV40GV2XDK1RSCXAWRD | 🔍 observability
  • instanceMethodStepWorkflow - instance methods with "use step" directive | wrun_41KY0Q5Q3W0GGTQYE5A6DS5JYA | 🔍 observability

example (5 failed):

  • error handling retry behavior regular Error retries until success
  • error handling retry behavior FatalError fails immediately without retries
  • error handling not registered StepNotRegisteredError fails the step but workflow can catch it
  • stepFunctionPassingWorkflow - step function references can be passed as arguments (without closure vars) | wrun_41KY0Q2AV40GV2XDK1RSCXAWRD | 🔍 observability
  • instanceMethodStepWorkflow - instance methods with "use step" directive | wrun_41KY0Q5Q3W0GGTQYE5A6DS5JYA | 🔍 observability

express (5 failed):

  • error handling retry behavior regular Error retries until success
  • error handling retry behavior FatalError fails immediately without retries
  • error handling not registered StepNotRegisteredError fails the step but workflow can catch it
  • stepFunctionPassingWorkflow - step function references can be passed as arguments (without closure vars) | wrun_41KY0Q2AV40GV2XDK1RSCXAWRD | 🔍 observability
  • instanceMethodStepWorkflow - instance methods with "use step" directive | wrun_41KY0Q5Q3W0GGTQYE5A6DS5JYA | 🔍 observability

fastify (5 failed):

  • error handling retry behavior regular Error retries until success
  • error handling retry behavior FatalError fails immediately without retries
  • error handling not registered StepNotRegisteredError fails the step but workflow can catch it
  • stepFunctionPassingWorkflow - step function references can be passed as arguments (without closure vars) | wrun_41KY0Q2AV40GV2XDK1RSCXAWRD | 🔍 observability
  • instanceMethodStepWorkflow - instance methods with "use step" directive | wrun_41KY0Q5Q3W0GGTQYE5A6DS5JYA | 🔍 observability

hono (5 failed):

  • error handling retry behavior regular Error retries until success
  • error handling retry behavior FatalError fails immediately without retries
  • error handling not registered StepNotRegisteredError fails the step but workflow can catch it
  • stepFunctionPassingWorkflow - step function references can be passed as arguments (without closure vars) | wrun_41KY0Q2AV40GV2XDK1RSCXAWRD | 🔍 observability
  • instanceMethodStepWorkflow - instance methods with "use step" directive | wrun_41KY0Q5Q3W0GGTQYE5A6DS5JYA | 🔍 observability

nextjs-turbopack (5 failed):

  • error handling retry behavior regular Error retries until success
  • error handling retry behavior FatalError fails immediately without retries
  • error handling not registered StepNotRegisteredError fails the step but workflow can catch it
  • stepFunctionPassingWorkflow - step function references can be passed as arguments (without closure vars) | wrun_41KY0Q2AV40GV2XDK1RSCXAWRD | 🔍 observability
  • instanceMethodStepWorkflow - instance methods with "use step" directive | wrun_41KY0Q5Q3W0GGTQYE5A6DS5JYA | 🔍 observability

nextjs-webpack (5 failed):

  • error handling retry behavior regular Error retries until success
  • error handling retry behavior FatalError fails immediately without retries
  • error handling not registered StepNotRegisteredError fails the step but workflow can catch it
  • stepFunctionPassingWorkflow - step function references can be passed as arguments (without closure vars) | wrun_41KY0Q2AV40GV2XDK1RSCXAWRD | 🔍 observability
  • instanceMethodStepWorkflow - instance methods with "use step" directive | wrun_41KY0Q5Q3W0GGTQYE5A6DS5JYA | 🔍 observability

nitro (5 failed):

  • error handling retry behavior regular Error retries until success
  • error handling retry behavior FatalError fails immediately without retries
  • error handling not registered StepNotRegisteredError fails the step but workflow can catch it
  • stepFunctionPassingWorkflow - step function references can be passed as arguments (without closure vars) | wrun_41KY0Q2AV40GV2XDK1RSCXAWRD | 🔍 observability
  • instanceMethodStepWorkflow - instance methods with "use step" directive | wrun_41KY0Q5Q3W0GGTQYE5A6DS5JYA | 🔍 observability

nuxt (5 failed):

  • error handling retry behavior regular Error retries until success
  • error handling retry behavior FatalError fails immediately without retries
  • error handling not registered StepNotRegisteredError fails the step but workflow can catch it
  • stepFunctionPassingWorkflow - step function references can be passed as arguments (without closure vars) | wrun_41KY0Q2AV40GV2XDK1RSCXAWRD | 🔍 observability
  • instanceMethodStepWorkflow - instance methods with "use step" directive | wrun_41KY0Q5Q3W0GGTQYE5A6DS5JYA | 🔍 observability

sveltekit (5 failed):

  • error handling retry behavior regular Error retries until success
  • error handling retry behavior FatalError fails immediately without retries
  • error handling not registered StepNotRegisteredError fails the step but workflow can catch it
  • stepFunctionPassingWorkflow - step function references can be passed as arguments (without closure vars) | wrun_41KY0Q2AV40GV2XDK1RSCXAWRD | 🔍 observability
  • instanceMethodStepWorkflow - instance methods with "use step" directive | wrun_41KY0Q5Q3W0GGTQYE5A6DS5JYA | 🔍 observability

vite (5 failed):

  • error handling retry behavior regular Error retries until success
  • error handling retry behavior FatalError fails immediately without retries
  • error handling not registered StepNotRegisteredError fails the step but workflow can catch it
  • stepFunctionPassingWorkflow - step function references can be passed as arguments (without closure vars) | wrun_41KY0Q2AV40GV2XDK1RSCXAWRD | 🔍 observability
  • instanceMethodStepWorkflow - instance methods with "use step" directive | wrun_41KY0Q5Q3W0GGTQYE5A6DS5JYA | 🔍 observability
📋 Other (10 failed)

e2e-vercel-prod-nest (5 failed):

  • error handling retry behavior regular Error retries until success
  • error handling retry behavior FatalError fails immediately without retries
  • error handling not registered StepNotRegisteredError fails the step but workflow can catch it
  • stepFunctionPassingWorkflow - step function references can be passed as arguments (without closure vars) | wrun_41KY0Q2AV40GV2XDK1RSCXAWRD
  • instanceMethodStepWorkflow - instance methods with "use step" directive | wrun_41KY0Q5Q3W0GGTQYE5A6DS5JYA

e2e-vercel-prod-tanstack-start (5 failed):

  • error handling retry behavior regular Error retries until success
  • error handling retry behavior FatalError fails immediately without retries
  • error handling not registered StepNotRegisteredError fails the step but workflow can catch it
  • stepFunctionPassingWorkflow - step function references can be passed as arguments (without closure vars) | wrun_41KY0Q2AV40GV2XDK1RSCXAWRD
  • instanceMethodStepWorkflow - instance methods with "use step" directive | wrun_41KY0Q5Q3W0GGTQYE5A6DS5JYA

Details by Category

❌ ▲ Vercel Production
AppPassedFailedSkipped
❌ astro121527
❌ example121527
❌ express121527
❌ fastify121527
❌ hono121527
❌ nextjs-turbopack14553
❌ nextjs-webpack14553
❌ nitro121527
❌ nuxt121527
❌ sveltekit14058
❌ vite121527
✅ 💻 Local Development
AppPassedFailedSkipped
✅ astro-stable128025
✅ express-stable128025
✅ fastify-stable128025
✅ hono-stable128025
✅ nextjs-turbopack-canary134019
✅ nextjs-turbopack-stable15300
✅ nextjs-webpack-canary134019
✅ nextjs-webpack-stable15300
✅ nitro-stable128025
✅ nuxt-stable128025
✅ sveltekit-stable14706
✅ vite-stable128025
✅ 📦 Local Production
AppPassedFailedSkipped
✅ astro-stable128025
✅ express-stable128025
✅ fastify-stable128025
✅ hono-stable128025
✅ nextjs-turbopack-canary134019
✅ nextjs-turbopack-stable15300
✅ nextjs-webpack-canary134019
✅ nextjs-webpack-stable15300
✅ nitro-stable128025
✅ nuxt-stable128025
✅ sveltekit-stable14706
✅ vite-stable128025
✅ 🐘 Local Postgres
AppPassedFailedSkipped
✅ astro-stable128025
✅ express-stable128025
✅ fastify-stable128025
✅ hono-stable128025
✅ nextjs-turbopack-canary134019
✅ nextjs-turbopack-stable15300
✅ nextjs-webpack-canary134019
✅ nextjs-webpack-stable15300
✅ nitro-stable128025
✅ nuxt-stable128025
✅ sveltekit-stable14706
✅ vite-stable128025
✅ 🪟 Windows
AppPassedFailedSkipped
✅ nextjs-turbopack15300
❌ 📋 Other
AppPassedFailedSkipped
✅ e2e-local-dev-nest-stable128025
✅ e2e-local-dev-tanstack-start-128025
✅ e2e-local-postgres-nest-stable128025
✅ e2e-local-postgres-tanstack-start-128025
✅ e2e-local-prod-nest-stable128025
✅ e2e-local-prod-tanstack-start-128025
❌ e2e-vercel-prod-nest121527
❌ e2e-vercel-prod-tanstack-start121527
✅ vercel-multi-region
AppPassedFailedSkipped
✅ nextjs-turbopack2700

📋 View full workflow run


Some E2E test jobs failed:

  • Vercel Prod: failure
  • Local Dev: success
  • Local Prod: success
  • Local Postgres: success
  • Windows: success

Check the workflow run for details.

@github-actions

github-actionsBot commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

📊 Workflow Benchmarks

commit 54c1c32 · Mon, 20 Jul 2026 21:35:32 GMT · run logs

Backend: vercel · app: nextjs-turbopack

MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep224 (-68%) 💚1279 🔴 (+17%) 🔻1305 🔴 (+12%)1377 🔴 (-19%) 💚30
TTFSstream233 (-76%) 💚1280 🔴 (+18%) 🔻1298 🔴 (+18%) 🔻1652 🔴 (+42%) 🔻30
TTFShook + stream536 (-57%) 💚1554 🔴 (+15%)1582 🔴 (+11%)1875 🔴 (+24%) 🔻30
STSO1020 steps (1-20)209 (+14%)399 🔴 (+57%) 🔻405 🔴 (+37%) 🔻562 🔴 (+84%) 🔻19
STSO1020 steps (101-120)204 (+1.5%)276 🔴 (-33%) 💚333 🔴 (-28%) 💚454 🔴 (-16%) 💚19
STSO1020 steps (1001-1020)507 (+7.4%)577 🔴 (+8.7%)620 🔴 (+11%)755 🔴 (+28%) 🔻19
WO1020 steps422879 (+6.0%)422879 (+6.0%)422879 (+6.0%)422879 (+6.0%)1
SLstream latency100 (+27%) 🔻166 🔴 (+29%) 🔻215 🔴 (+49%) 🔻367 🔴 (+135%) 🔻30
📜 Previous results (2)

289071b

Mon, 20 Jul 2026 21:18:47 GMT · run logs

vercel / nextjs-turbopack

MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep202 (-71%) 💚1199 🔴 (+10%)1236 🔴 (+6.3%)1407 🔴 (-17%) 💚30
TTFSstream174 (-82%) 💚1167 🔴 (+8.0%)1232 🔴 (+12%)1309 🔴 (+12%)30
TTFShook + stream495 (-60%) 💚1358 🔴 (±0%)1396 🔴 (-2.2%)1625 🔴 (+7.0%)30
STSO1020 steps (1-20)190 (+3.8%)278 🔴 (+9.4%)318 🔴 (+7.8%)402 🔴 (+31%) 🔻19
STSO1020 steps (101-120)214 (+6.5%)283 🔴 (-32%) 💚319 🔴 (-31%) 💚446 🔴 (-18%) 💚19
STSO1020 steps (1001-1020)499 (+5.7%)566 🔴 (+6.6%)599 🔴 (+7.3%)608 🔴 (+3.4%)19
WO1020 steps419823 (+5.3%)419823 (+5.3%)419823 (+5.3%)419823 (+5.3%)1
SLstream latency87 (+10%)152 🔴 (+18%) 🔻189 🔴 (+31%) 🔻268 🔴 (+72%) 🔻30

9615b13

Mon, 20 Jul 2026 20:29:54 GMT · run logs

vercel / nextjs-turbopack

MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep1275 🔴1286 🔴1332 🔴30
TTFSstream1276 🔴1288 🔴1479 🔴30
TTFShook + stream1542 🔴1569 🔴1700 🔴30
STSO1020 steps (1-20)284 🔴320 🔴1710 🔴19
STSO1020 steps (101-120)269 🔴314 🔴1123 🔴19
STSO1020 steps (1001-1020)517 🔴560 🔴594 🔴19
WO1020 steps3906603906603906601
SLstream latency147 🔴174 🔴3455 🔴30

Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.

Metrics — TTFS: time to first step body (in-deployment start() → first step body, deployment clocks) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · SL: stream latency (in-deployment write → read propagation, readAt - writtenAt)

Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · stream latency: parallel reader/writer steps on a dedicated stream; SL is the in-deployment write->read propagation (readAt - writtenAt)

🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · STSO (1-20) 20/30/60 · STSO (101-120) 30/45/90 · STSO (1001-1020) 40/60/120

All metrics are measured from deployment-side timestamps only. Runs are triggered by an in-deployment route that stamps the anchor (clientStart) right before start(), so the CI runner’s request and its path through api.vercel.com sit outside every measured window. TTFS = in-deployment start() → first step body (turbo uses the in-process fast path, non-turbo the dispatch path), and includes the VQS dispatch hop plus any /flow cold start. STSO/WO are measured between step bodies on the deployment. SL is measured inside the workflow (parallel reader/writer steps), so it no longer includes the api.vercel.com read path.

Cold starts are kept in the numbers on purpose — they are part of real bursty-workload latency. The workbench deployment cold-starts the /flow invocation for a large fraction of runs, inflating P75+; the Best column shows the fastest (warm-start) sample for comparison.

On Vercel the nest project's `vercel-build` script is run directly, so unlike
the framework workbenches (whose `vercel build` goes through Turbo and builds
`packages/*` first), the workspace SDK dist was never built — the build failed
with `Cannot find module '.../packages/nest/dist/cli.js'`. Prepend
`pnpm -w run build` so the SDK packages are compiled (Turbo-cached) before the
NestJS app and the `workflow-nest build --vercel` step run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Peter Wielander <peter.wielander@vercel.com>
<Nest className="size-16 dark:invert" />
<span className="font-medium">NestJS</span>
<Badge variant="secondary">Coming soon</Badge>
<Badge variant="secondary">Beta</Badge>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Align the NestJS support status

This card now labels NestJS as Beta, but the linked NestJS guide still says the integration is experimental. Because this PR explicitly promotes the documented support level, please update the guide callout to Beta as well—or retain the existing status here—so users do not receive conflicting maturity guarantees.

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed to align both to "experimental" since we're unsure if we want to support this officially

Signed-off-by: Peter Wielander <peter.wielander@vercel.com>
@github-actions

Copy link
Copy Markdown
Contributor

No backport to stable for 96719d8 (AI decision).

This commit builds directly on NestJS Vercel Build Output support (#2988) that only exists on main: on stable, workbench/nest/package.json has no vercel-build script and packages/nest's CLI has no --vercel mode, so the modified script line and the new e2e-vercel-prod matrix entry have nothing to run against. The docs change also links to docs/content/docs/v5/getting-started/nestjs, a page that does not exist on stable, so it would introduce a broken card link and meta entry.

To override, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:

96719d8220f197ce7735372433d3d04090db7f9e

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@VaguelySerious@karthikscale3