Uh oh!
There was an error while loading. Please reload this page.
fix(core): resolve forwarded stream keys across deployments - #2191
Merged
pranaygp merged 1 commit intoJun 1, 2026
Conversation
Contributor
🦋 Changeset detectedLatest commit: c12a58f The changes in this PR will be included in the next version bump. This PR includes changesets to release 20 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Contributor
📊 Benchmark Results
workflow with no steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express | Next.js (Turbopack) workflow with 1 step💻 Local Development
▲ Production (Vercel)
🔍 Observability: Next.js (Turbopack) | Express workflow with 10 sequential steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express | Next.js (Turbopack) workflow with 25 sequential steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express | Next.js (Turbopack) workflow with 50 sequential steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express | Next.js (Turbopack) Promise.all with 10 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Next.js (Turbopack) | Express Promise.all with 25 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express | Next.js (Turbopack) Promise.all with 50 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express | Next.js (Turbopack) Promise.race with 10 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Next.js (Turbopack) | Express Promise.race with 25 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express | Next.js (Turbopack) Promise.race with 50 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Next.js (Turbopack) | Express workflow with 10 sequential data payload steps (10KB)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Next.js (Turbopack) | Express workflow with 25 sequential data payload steps (10KB)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express | Next.js (Turbopack) workflow with 50 sequential data payload steps (10KB)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express | Next.js (Turbopack) workflow with 10 concurrent data payload steps (10KB)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express | Next.js (Turbopack) workflow with 25 concurrent data payload steps (10KB)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Next.js (Turbopack) | Express workflow with 50 concurrent data payload steps (10KB)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express | Next.js (Turbopack) Stream Benchmarks(includes TTFB metrics)workflow with stream💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express | Next.js (Turbopack) stream pipeline with 5 transform steps (1MB)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express | Next.js (Turbopack) 10 parallel streams (1MB each)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express | Next.js (Turbopack) fan-out fan-in 10 streams (1MB each)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Next.js (Turbopack) | Express SummaryFastest Framework by WorldWinner determined by most benchmark wins
Fastest World by FrameworkWinner determined by most benchmark wins
Column Definitions
Worlds:
❌ Some benchmark jobs failed:
Check the workflow run for details. |
Contributor
🧪 E2E Test Results✅ All tests passed Summary
Details by Category✅ ▲ Vercel Production
✅ 💻 Local Development
✅ 📦 Local Production
✅ 🐘 Local Postgres
✅ 🪟 Windows
✅ 📋 Other
|
TooTallNate
approved these changes
May 31, 2026
TooTallNate
left a comment
Member
There was a problem hiding this comment.
Approve — solid fix, with one small consistency concern worth thinking about
The cross-deployment encryption bug is real and the fix is well-targeted: add deploymentId to the writable stream descriptor wire format, stamp it via STREAM_SERVER_DEPLOYMENT_ID_SYMBOL on writables that have it, and use it in getEncryptionKeyForRun(runId, { deploymentId }) instead of the assumed-current-deployment fallback. Legacy descriptors without a deploymentId fall back to loading the owning run via world.runs.get(runId) — that's correctness-preserving for in-flight serialized payloads.
Verified
- After rebasing onto current
main: 12 files, +278/-22 (the scary "removals" ofexperimentalSetAttributesfrominterfaces.tsand the encryption error handling inserialization.tsare stale-branch artifacts from being forked before #2134, #2157, and #2145 landed) pnpm install --frozen-lockfile✓pnpm turbo run build --filter @workflow/core✓pnpm --filter @workflow/core test✓ (1083/1083)- The two new serialization tests both pass:
- Happy path — forwarded writable with
deploymentId→ usesgetEncryptionKeyForRun(runId, { deploymentId })and does NOT callruns.get - Legacy fallback — descriptor without
deploymentId→ callsruns.get(runId)first, thengetEncryptionKeyForRun(run)✓
- Happy path — forwarded writable with
CI noise
14 failures on the latest run, but they're all cascading from swc-plugin-workflow WASM build failing with undefined symbol: __emit_diagnostics. That's the bug fixed by #2174 (5dabbeeca fix(swc-plugin): allow wasm host imports during link) which landed on main after this branch was forked. A rebase would resolve them all.
One thing worth thinking about — step-handler.ts uses process.env.VERCEL_DEPLOYMENT_ID
The two workflowDeploymentId sites are:
| Site | Source |
|---|---|
runtime.ts (inline step execution) | workflowRun.deploymentId / bgRun.deploymentId — the workflow's ACTUAL deployment ✓ |
step-handler.ts (background step execution) | process.env.VERCEL_DEPLOYMENT_ID — the CURRENT runtime deployment |
For background steps, these are only the same if the queue routes steps to the workflow's original deployment. If queues route steps to a NEWER deployment, process.env.VERCEL_DEPLOYMENT_ID is the newer one, not the workflow's.
Reading the world-vercel getEncryptionKeyForRun, the existing background-step encryption key resolution ALREADY assumes current-deployment (it calls memoizeEncryptionKey(world, workflowRunId) with no context, which then uses local HKDF when running inside Vercel). So this PR's choice in step-handler.ts is at least consistent with how step encryption already works — if the existing background-step encryption is correct (i.e., queue routes to original deployment, or some other invariant), then the new workflowDeploymentId source is also correct.
If the existing encryption has a latent bug for background steps running on newer deployments, this PR carries that forward — but doesn't make it worse, and the parent/child cross-deployment case (which is what the PR is actually trying to fix) IS handled correctly via the inline runtime.ts paths using workflowRun.deploymentId.
Worth confirming the invariant: do background step queues always route to the workflow's original deployment, or can they route to newer deployments? If the latter, there's a follow-up to also resolve the workflow's actual deployment in step-handler.ts (probably load the run first, similar to the legacy fallback in this PR's getForwardedWritableEncryptionKey).
Not a blocker — the PR's scope is the parent/child writable case, and that's handled correctly.
Bonus: the test design caught a subtle correctness point
The legacy-fallback test asserts the right behavior when descriptors are mid-flight from older SDK versions:
expect(runsGet).toHaveBeenCalledWith('wrun_parent');expect(getEncryptionKeyForRun).toHaveBeenCalledWith(parentRun);That confirms the fallback uses the full WorkflowRun overload (which extracts deploymentId from run.deploymentId), not the broken (runId, undefined-context) overload. Good test instinct.
PR status
Currently draft. Approving in spirit pending:
- Rebase onto current
main(resolves the stale-branch CI cascades + the deletions-in-diff confusion) - Optional: thread an explicit
workflowDeploymentIdthrough the background step path if cross-deployment step routing is actually possible
Approving.
pranaygp
marked this pull request as ready for review
June 1, 2026 17:48
pranaygp
enabled auto-merge (squash)
June 1, 2026 17:49
pranaygpforce-pushed
the
pranaygp/codex/fix-cross-deployment-stream-encryption
branch
from
June 1, 2026 17:51
bf73d05 to
c12a58fComparepranaygp
commented
Jun 1, 2026
ContributorAuthor
Rebased onto current
I left the existing V1 step-handler deployment behavior unchanged; this patch only carries the writable stream owner deployment through serialization so forwarded stream writes resolve their original key. |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR fixes encryption-key resolution for forwarded WritableStream handles when parent/child workflow runs execute on different Vercel deployments, by carrying the owning deployment ID through serialization and using it to resolve the correct per-run encryption key.
Changes:
- Stamp forwarded
WritableStreamdescriptors with an owningdeploymentIdand thread deployment context through step hydration/execution. - Update serialization reducers/revivers to persist and revive the owning deployment ID, and resolve the forwarded-stream encryption key with deployment context (with legacy fallback).
- Add regression tests and a changeset; clarify
World.getEncryptionKeyForRundocumentation to cover forwarded-stream usage.
Reviewed changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| packages/world/src/interfaces.ts | Expands getEncryptionKeyForRun docs to include forwarded-stream key resolution context. |
| packages/core/src/symbols.ts | Adds a new symbol to tag forwarded streams with owning deployment ID. |
| packages/core/src/step/context-storage.ts | Extends StepContext to carry workflowDeploymentId for forwarded streams. |
| packages/core/src/step/writable-stream.ts | Tags created writables with owning deployment ID when available. |
| packages/core/src/step/writable-stream.test.ts | Adds a test asserting the deployment tag is applied to step-level writables. |
| packages/core/src/serialization/types.ts | Extends serializable WritableStream descriptor shape with deploymentId. |
| packages/core/src/serialization.ts | Propagates deploymentId in reducers/revivers and resolves forwarded-stream encryption keys with deployment context + legacy fallback. |
| packages/core/src/serialization.test.ts | Adds regression tests covering deployment-aware key resolution and legacy fallback behavior. |
| packages/core/src/runtime/step-handler.ts | Passes deployment ID into step hydration and step context for stream forwarding. |
| packages/core/src/runtime/step-executor.ts | Threads workflowDeploymentId through step execution to hydration/context. |
| packages/core/src/runtime.ts | Populates workflowDeploymentId from the loaded run when executing steps. |
| .changeset/cross-deployment-stream-keys.md | Publishes patch changes for @workflow/core and @workflow/world. |
Comments suppressed due to low confidence (1)
packages/core/src/serialization.ts:1363
- The legacy (no deploymentId) path loads the full run via
world.runs.get(runId)to resolve the encryption key.runs.get()defaults toresolveData: 'all', which can fetch largeinput/output/errorblobs unnecessarily just to readdeploymentId, increasing latency and memory usage when hydrating forwarded streams.
Consider fetching the owner run with resolveData: 'none' for this fallback lookup.
tagAbortPair(controller, value);
controller.abort(value.reason);
} else if (value.streamName) {
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Uh oh!
There was an error while loading. Please reload this page.
pranaygp
deleted the
pranaygp/codex/fix-cross-deployment-stream-encryption
branch
June 1, 2026 18:38
Contributor
Backport to To resolve manually, push a backport branch and open a PR against git fetch origin stable
git checkout -b backport/pr-2191-to-stable origin/stable
git cherry-pick -S 8f68d3525ce3e420f4d16b9976c97a5598f91afd # -S signs the commit# Fix conflicts, then:
git add -A
git cherry-pick --continue
git push -u origin backport/pr-2191-to-stable
gh pr create --base stable --head backport/pr-2191-to-stable \
--title "Backport #2191: <original PR title>" \
--body "Manual backport of #2191 (cherry-pick 8f68d3525ce3) to \`stable\`." |
pranaygp added a commit
that referenced
this pull request
Jun 1, 2026
(cherry picked from commit 8f68d35)
TooTallNate pushed a commit
that referenced
this pull request
Jun 1, 2026
This was referenced Jun 1, 2026
Merged
pranaygp added a commit
that referenced
this pull request
Jun 2, 2026
* origin/main: [world-vercel] Retry transient response-body parse failures in the HTTP client (#2204) Add virtualization to the trace viewer (#2205) Trace viewer: scroll-load events past an auto-load cap (#2200) fix(core): resolve forwarded stream keys across deployments (#2191) [e2e] Improve error labeling in event-log-race-repro CI job (#2190) [core] Harden event pagination response parsing (#2180) (#2179) Add loading skeleton to the new trace viewer (#2164) Add tooltip components + apply on up/down detail pane (#2163) fix(swc-plugin): allow wasm host imports during link (#2174) [test] Forward-port reused-sleep replay divergence test (#2172) [e2e] Add `event-log-race-repro` label for triggering CI stress-test (#2159) Version Packages (beta) (#2162) fix(world-local): skip Nov 2025 ghost versions on npm (#2168) fix(core,errors): classify SDK encryption failures as RUNTIME_ERROR (#2145) [web-shared][web] Fix events tab search (#2107) Version Packages (beta) (#2147) Allow setting workflow attributes from steps (#2157) Better search handling on the trace viewer (#2144) [docs] Document experimental attributes feature (#2141)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Root Cause
Cross-run writable revivers only carried the parent
runIdand calledgetEncryptionKeyForRun(targetRunId). In@workflow/world-vercel, a string lookup without deployment context resolves against the executing deployment, so a child started on a newer deployment encrypted new chunks using its key while the parent stream is read using the older run key.Context
This upstreams the behavior temporarily patched downstream in https://github.com/vercel/ash/pull/852, while avoiding its extra owner-run lookup for newly serialized descriptors. Legacy or in-flight descriptors retain a correctness fallback.
Validation
pnpm turbo build --filter=@workflow/core...pnpm --filter @workflow/core typecheckpnpm --filter @workflow/core exec vitest run src/serialization.test.ts src/step/writable-stream.test.tspnpm --filter @workflow/core testpnpm biome format <changed files>pnpm changeset status --since=main