Skip to content

feat: decrypt sealed payloads in the dashboard and CLI - #3146

Merged
TooTallNate merged 3 commits into
mainfrom
nate/encp-o11y-decrypt
Jul 28, 2026
Merged

feat: decrypt sealed payloads in the dashboard and CLI#3146
TooTallNate merged 3 commits into
mainfrom
nate/encp-o11y-decrypt

Conversation

@TooTallNate

Copy link
Copy Markdown
Member

Replaces #3097. That PR was force-closed as "merged" by GitHub when I
force-pushed its base branch to a commit that happened to contain its head —
GitHub concluded it had merged into its base. Its code never reached main.
It cannot be reopened, so this PR carries the identical work. Review history
on #3097 still applies.

Ordering change: this now targets main directly and should land before
#3096, per the review there. #3096 begins emitting sealed (encp) hook payloads
while workflow inspect --decrypt can only build a symmetric key, so merging it
first would leave those payloads showing as ciphertext placeholders. Landing the
reader first closes that window.

To make this PR stand alone, deriveRunPayloadKeys moved here from #3096 — it
is the capability the reader needs, and its own doc says to use it "anywhere a
run reads its own event log", so it belongs on this side regardless.


PR 5 of 7. Stacked on #3096; review the stack in order, starting from #3093.

Why this is required, not optional

Once #3096 lands, a hook resumption from another deployment writes a sealed payload into the run's event log. Observability tooling can't open it: hydrateDataWithKey only ever ran AES-GCM. So the dashboard and CLI would render a 🔒 placeholder on data the user is entitled to read and has already supplied the key for — a visible debugging regression on exactly the runs this work speeds up.

What changed

hydrateDataWithKey delegates to the envelope layer, which dispatches on the 4-byte format prefix, instead of unconditionally decrypting with AES-GCM.

All four o11y key-resolution sites now resolve the run's full key capability rather than just its symmetric key:

SitePath
@workflow/web-sharedhydrateResourceIOAsync
@workflow/webuse-stream-reader (hand-rolled frame decode)
@workflow/cli--decrypt field hydration
@workflow/cli--decrypt stream output

Each already had the raw 32 bytes in hand, so this costs one extra key derivation and no additional requests or round trips.

A caller that supplies only a symmetric key still gets the ciphertext placeholder for sealed payloads rather than a decryption error — that key never could have opened them, so surfacing a failure would be misleading.

Browser bundling

The natural import for the new helper is @workflow/core/serialization, but that module graph reaches node:util and node:async_hooks and cannot be bundled for the browser — which is precisely what the @workflow/core/serialization-format entrypoint exists to avoid. I hit this as a build failure and moved the key helpers to a re-export from that browser-safe entrypoint; the two browser consumers import from there, while the CLI keeps the direct import since it runs on Node.

Verified by walking the built import graph: serialization-format.js reaches 6 modules and zero Node built-ins.

Tests

Two new cases assert the o11y path opens a real sealed payload with the derived keypair, and that the same resolved key still opens the run's own symmetric payloads — a run's event log mixes both.

Core suite 1598 passing, web-shared 160, CLI 78. Zero new lint diagnostics (diffed against a stashed baseline).

Without this, any payload another run sealed to this one renders as a lock
icon with no way to open it — a visible regression for anyone debugging a
run that received a cross-deployment hook resumption. The user is entitled
to read the data and has already supplied the key; only the plumbing was
missing.
`hydrateDataWithKey` now delegates to the envelope layer, which dispatches
on the format prefix, instead of unconditionally running AES-GCM. All four
o11y key-resolution sites (web-shared hydration, the web stream reader,
and both CLI `--decrypt` paths) resolve the full capability rather than
just the symmetric key. Each already had the raw 32 bytes in hand, so this
costs one extra derivation and no additional requests.
A caller that supplies only a symmetric key still gets the ciphertext
placeholder for sealed payloads rather than a decryption error, since that
key never could have opened them.
**Browser bundling.** The obvious import for the new helper is
`@workflow/core/serialization`, but that module graph reaches `node:util`
and `node:async_hooks` and cannot be bundled for the browser — which is
what `@workflow/core/serialization-format` exists to avoid. The key
helpers are re-exported from that browser-safe entrypoint instead, and the
two browser consumers import from there; the CLI keeps the direct import
since it runs on Node. Verified by walking the built import graph: the
entrypoint reaches 6 modules and zero Node built-ins.
Unrelated: `pnpm --filter @workflow/web build` currently fails on `main`
too (`reducers/common.js` importing `node:util`). Turbo caching had been
hiding it; touching core caused a cache miss that surfaced it. Not
addressed here.
- `hydrateDataWithKey` accepted `PayloadKey`, which includes `SealTarget`.
A seal target holds only a public key, so it can open neither scheme —
passing one compiled fine and then always failed at runtime. Added a
`DecryptionKey` alias (`CryptoKey | RunPayloadKeys`) and narrowed the
signature, so that misuse is now a compile error. A `@ts-expect-error`
test pins the guarantee.
- `hydrateResourceIOAsync` dynamically imported
`@workflow/core/serialization-format` twice. Destructure both bindings
from the single existing import instead.
@TooTallNate
TooTallNate requested review from a team and ijjk as code ownersJuly 27, 2026 23:49
CopilotAI review requested due to automatic review settings July 27, 2026 23:49
@changeset-bot

changeset-botBot commented Jul 27, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 916bc93

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 16 packages
NameType
@workflow/coreMinor
@workflow/web-sharedPatch
@workflow/webPatch
@workflow/cliPatch
@workflow/buildersPatch
@workflow/nextPatch
@workflow/nitroPatch
@workflow/vitestPatch
workflowMinor
@workflow/world-testingPatch
@workflow/astroPatch
@workflow/nestPatch
@workflow/nuxtPatch
@workflow/rollupPatch
@workflow/sveltekitPatch
@workflow/vitePatch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@vercel

vercelBot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

@github-actions

github-actionsBot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

📊 Workflow Benchmarks

commit 916bc93 · Tue, 28 Jul 2026 00:26:55 GMT · run logs

Backend: vercel · app: nextjs-turbopack

MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep228 (-69%) 💚1407 🔴 (+24%) 🔻1449 🔴 (+20%) 🔻1751 🔴 (+28%) 🔻30
TTFSstream273 (-13%)1428 🔴 (+30%) 🔻1438 🔴 (+26%) 🔻1527 🔴 (+30%) 🔻30
TTFShook + stream572 (-55%) 💚1694 🔴 (+21%) 🔻1740 🔴 (+19%) 🔻1895 🔴 (+16%) 🔻30
STSO1020 steps (1-20)185 (+8.8%)327 🔴 (+8.3%)550 🔴 (+56%) 🔻681 🔴 (+80%) 🔻19
STSO1020 steps (101-120)205 (+4.1%)298 🔴 (-12%)473 🔴 (+16%) 🔻598 🔴 (+20%) 🔻19
STSO1020 steps (1001-1020)473 (-2.1%)563 🔴 (-1.9%)669 🔴 (+3.6%)852 🔴 (+30%) 🔻19
WO1020 steps410726 (-6.4%)410726 (-6.4%)410726 (-6.4%)410726 (-6.4%)1
SLstream latency97 (+4.3%)196 🔴 (+17%) 🔻257 🔴 (+9.8%)592 🔴 (+134%) 🔻30
SOstream overhead (text)126 (-3.8%)303 🔴 (+4.5%)400 (+13%)543 (+29%) 🔻30
SOstream overhead (structured)135 (±0%)391 🔴 (+50%) 🔻729 🔴 (+162%) 🔻5527 🔴 (+1238%) 🔻30
ℹ️ Metric definitions & methodology

Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.

Metrics — TTFS: time to first step body (in-deployment start() → first step body, deployment clocks) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · SL: stream latency (in-deployment write → read propagation, readAt - writtenAt) · SO: stream overhead (end-to-end write+consume time beyond the modelled generation window)

Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · stream latency: parallel reader/writer steps on a dedicated stream; SL is the in-deployment write->read propagation (readAt - writtenAt) · stream overhead (text): writer streams 300 variable-length text token deltas paced at 100/s for 3s (a haiku-size LLM's token throughput) while a parallel reader drains the whole stream; SO is the end-to-end write+consume time beyond the 3s generation window (overhead/backpressure) · stream overhead (structured): same workload as stream overhead (text), but each delta is an AI-SDK-style structured object ({ type: 'text-delta', id, text }) instead of a raw string, so the SO gap vs the text scenario is the added serialization cost

🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · SO 250/500/1000 · STSO (1-20) 20/30/60 · STSO (101-120) 30/45/90 · STSO (1001-1020) 40/60/120

All metrics are measured from deployment-side timestamps only. Runs are triggered by an in-deployment route that stamps the anchor (clientStart) right before start(), so the CI runner’s request and its path through api.vercel.com sit outside every measured window. TTFS = in-deployment start() → first step body (turbo uses the in-process fast path, non-turbo the dispatch path), and includes the VQS dispatch hop plus any /flow cold start. STSO/WO are measured between step bodies on the deployment. SL is measured inside the workflow (parallel reader/writer steps), so it no longer includes the api.vercel.com read path.

Cold starts are kept in the numbers on purpose — they are part of real bursty-workload latency. The workbench deployment cold-starts the /flow invocation for a large fraction of runs, inflating P75+; the Best column shows the fastest (warm-start) sample for comparison.

@github-actions

github-actionsBot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

🧪 E2E Test Results

All tests passed

E2E Test Summary

Summary
PassedFailedSkippedTotal
✅ ▲ Vercel Production145502391694
✅ 💻 Local Development162102271848
✅ 📦 Local Production162102271848
✅ 🐘 Local Postgres162102271848
✅ 🪟 Windows15400154
✅ 📋 Other102002121232
✅ vercel-multi-region270027
Total7519011328651
Details by Category

✅ ▲ Vercel Production

AppPassedFailedSkipped
✅ astro126028
✅ example126028
✅ express126028
✅ fastify126028
✅ hono126028
✅ nextjs-turbopack15103
✅ nextjs-webpack15103
✅ nitro126028
✅ nuxt126028
✅ sveltekit14509
✅ vite126028

✅ 💻 Local Development

AppPassedFailedSkipped
✅ astro-stable128026
✅ express-stable128026
✅ fastify-stable128026
✅ hono-stable128026
✅ nextjs-turbopack-canary135019
✅ nextjs-turbopack-stable15400
✅ nextjs-webpack-canary135019
✅ nextjs-webpack-stable15400
✅ nitro-stable128026
✅ nuxt-stable128026
✅ sveltekit-stable14707
✅ vite-stable128026

✅ 📦 Local Production

AppPassedFailedSkipped
✅ astro-stable128026
✅ express-stable128026
✅ fastify-stable128026
✅ hono-stable128026
✅ nextjs-turbopack-canary135019
✅ nextjs-turbopack-stable15400
✅ nextjs-webpack-canary135019
✅ nextjs-webpack-stable15400
✅ nitro-stable128026
✅ nuxt-stable128026
✅ sveltekit-stable14707
✅ vite-stable128026

✅ 🐘 Local Postgres

AppPassedFailedSkipped
✅ astro-stable128026
✅ express-stable128026
✅ fastify-stable128026
✅ hono-stable128026
✅ nextjs-turbopack-canary135019
✅ nextjs-turbopack-stable15400
✅ nextjs-webpack-canary135019
✅ nextjs-webpack-stable15400
✅ nitro-stable128026
✅ nuxt-stable128026
✅ sveltekit-stable14707
✅ vite-stable128026

✅ 🪟 Windows

AppPassedFailedSkipped
✅ nextjs-turbopack15400

✅ 📋 Other

AppPassedFailedSkipped
✅ e2e-local-dev-nest-stable128026
✅ e2e-local-dev-tanstack-start-128026
✅ e2e-local-postgres-nest-stable128026
✅ e2e-local-postgres-tanstack-start-128026
✅ e2e-local-prod-nest-stable128026
✅ e2e-local-prod-tanstack-start-128026
✅ e2e-vercel-prod-nest126028
✅ e2e-vercel-prod-tanstack-start126028

✅ vercel-multi-region

AppPassedFailedSkipped
✅ nextjs-turbopack2700

📋 View full workflow run

@karthikscale3karthikscale3 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ai review: The decryption changes look correct, but the published @workflow/web package is missing from the changeset.

Comment thread.changeset/encp-o11y-decrypt.md

@karthikscale3karthikscale3 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at 85e262c. Implementation looks good; the existing inline changeset note is non-blocking.

This PR changes the dashboard's stream reader
(`packages/web/app/lib/hooks/use-stream-reader.ts`) so it dispatches on the
envelope format and can read sealed (`encp`) frames, but the changeset listed
only core, web-shared and cli.
`@workflow/web` is published, so without an entry the change would still ship —
just as an incidental dependency bump, with nothing in that package's release
notes explaining that sealed-stream decryption landed.
@TooTallNate
TooTallNate enabled auto-merge (squash) July 28, 2026 00:14
@TooTallNate
TooTallNate merged commit e8bc7d6 into mainJul 28, 2026
174 of 176 checks passed
@TooTallNate
TooTallNate deleted the nate/encp-o11y-decrypt branch July 28, 2026 00:36
@github-actions

Copy link
Copy Markdown
Contributor

No backport to stable for e8bc7d6 (AI decision).

This is feature work — PR 5 of 7 in a stack that introduces the sealed-box (encp) payload scheme, adding new capability (deriveRunPayloadKeys, DecryptionKey, browser-safe re-exports) to observability tooling. It also builds directly on main-only APIs: git ls-tree origin/stable confirms packages/core/src/sealed-box.ts and packages/core/src/serialization/encryption.ts do not exist on stable, so the cherry-pick could not even work. The "regression" it fixes only materializes once the rest of the sealed-payload stack lands, which is not on the maintenance line.

To override, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:

e8bc7d6aadcdb5f89de1be71cfcafe5e9085ed25

@github-actionsgithub-actionsBot mentioned this pull request Jul 28, 2026
pranaygp added a commit that referenced this pull request Jul 28, 2026
…ry-2
* origin/main: (292 commits)
feat(core): seal forwarded stream writes to the owner's public key (#3098)
feat(core): seal hook payloads to the target run's public key (#3096)
[e2e] Rebuild the event-log corruption repro around step-count divergence (#3147)
feat: decrypt sealed payloads in the dashboard and CLI (#3146)
Prewarm only appended replay payloads (#3131)
feat: publish each run's X25519 public key on the run entity (#3095)
feat(core): route sealed envelopes through the serialization layer (#3094)
docs: redirect retired migration-guides URLs to comparisons (#3127)
feat(core): add `encp` sealed-box encryption primitive (#3093)
chore(core): clarify runtime comments (#3111)
Remove obsolete world factory aliases (#3112)
feat(core): deterministic sandbox hardening (#3045)
Remove retired v1 step route plumbing (#3061)
[core] Don't count racing invocations' duplicate step_started events toward the maxRetries ceiling (#3069)
[world-testing] Isolate each spawned test server's data directory (#3055)
fix: upgrade postcss to >=8.5.18 to address GHSA-r28c-9q8g-f849 (#3102)
[next] Respect .gitignore in dev watcher to avoid EMFILE on large monorepos (#3085)
[ci] Backport only stability fixes to `stable`, default to claude-opus-5 (#3092)
perf(core): immediate leading-edge dispatch for idle streams (flush window default 0) (#3088)
Optimize `processImportSpecifier` by computing `shouldFollowImportsFromFile` once per file (#3052)
...
# Conflicts:
#	docs/components/geistdocs/desktop-menu.tsx
#	docs/components/geistdocs/mobile-menu.tsx
#	docs/content/docs/v5/cookbook/advanced/child-workflows.mdx
#	docs/content/docs/v5/cookbook/advanced/upgrading-workflows.mdx
#	docs/content/docs/v5/cookbook/agent-patterns/agent-cancellation.mdx
#	docs/content/docs/v5/cookbook/agent-patterns/durable-agent.mdx
#	docs/content/docs/v5/cookbook/agent-patterns/human-in-the-loop.mdx
#	docs/content/docs/v5/cookbook/common-patterns/batching.mdx
#	docs/content/docs/v5/cookbook/common-patterns/idempotency.mdx
#	docs/content/docs/v5/cookbook/common-patterns/rate-limiting.mdx
#	docs/content/docs/v5/cookbook/common-patterns/saga.mdx
#	docs/content/docs/v5/cookbook/common-patterns/scheduling.mdx
#	docs/content/docs/v5/cookbook/common-patterns/sequential-and-parallel.mdx
#	docs/content/docs/v5/cookbook/common-patterns/timeouts.mdx
#	docs/content/docs/v5/cookbook/common-patterns/webhooks.mdx
#	docs/content/docs/v5/cookbook/common-patterns/workflow-composition.mdx
#	docs/content/docs/v5/cookbook/index.mdx
#	docs/content/docs/v5/cookbook/integrations/ai-sdk.mdx
#	docs/content/docs/v5/cookbook/integrations/chat-sdk.mdx
#	docs/content/docs/v5/cookbook/integrations/sandbox.mdx
#	docs/next.config.ts
#	docs/proxy.ts
#	docs/scripts/lint.ts
#	pnpm-lock.yaml
#	pnpm-workspace.yaml
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@TooTallNate@karthikscale3
, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
feat: decrypt sealed payloads in the dashboard and CLI by TooTallNate · Pull Request #3146 · vercel/workflow · GitHub
Skip to content

feat: decrypt sealed payloads in the dashboard and CLI - #3146

Merged
TooTallNate merged 3 commits into
mainfrom
nate/encp-o11y-decrypt
Jul 28, 2026
Merged

feat: decrypt sealed payloads in the dashboard and CLI#3146
TooTallNate merged 3 commits into
mainfrom
nate/encp-o11y-decrypt

Conversation

@TooTallNate

Copy link
Copy Markdown
Member

Replaces #3097. That PR was force-closed as "merged" by GitHub when I
force-pushed its base branch to a commit that happened to contain its head —
GitHub concluded it had merged into its base. Its code never reached main.
It cannot be reopened, so this PR carries the identical work. Review history
on #3097 still applies.

Ordering change: this now targets main directly and should land before
#3096, per the review there. #3096 begins emitting sealed (encp) hook payloads
while workflow inspect --decrypt can only build a symmetric key, so merging it
first would leave those payloads showing as ciphertext placeholders. Landing the
reader first closes that window.

To make this PR stand alone, deriveRunPayloadKeys moved here from #3096 — it
is the capability the reader needs, and its own doc says to use it "anywhere a
run reads its own event log", so it belongs on this side regardless.


PR 5 of 7. Stacked on #3096; review the stack in order, starting from #3093.

Why this is required, not optional

Once #3096 lands, a hook resumption from another deployment writes a sealed payload into the run's event log. Observability tooling can't open it: hydrateDataWithKey only ever ran AES-GCM. So the dashboard and CLI would render a 🔒 placeholder on data the user is entitled to read and has already supplied the key for — a visible debugging regression on exactly the runs this work speeds up.

What changed

hydrateDataWithKey delegates to the envelope layer, which dispatches on the 4-byte format prefix, instead of unconditionally decrypting with AES-GCM.

All four o11y key-resolution sites now resolve the run's full key capability rather than just its symmetric key:

SitePath
@workflow/web-sharedhydrateResourceIOAsync
@workflow/webuse-stream-reader (hand-rolled frame decode)
@workflow/cli--decrypt field hydration
@workflow/cli--decrypt stream output

Each already had the raw 32 bytes in hand, so this costs one extra key derivation and no additional requests or round trips.

A caller that supplies only a symmetric key still gets the ciphertext placeholder for sealed payloads rather than a decryption error — that key never could have opened them, so surfacing a failure would be misleading.

Browser bundling

The natural import for the new helper is @workflow/core/serialization, but that module graph reaches node:util and node:async_hooks and cannot be bundled for the browser — which is precisely what the @workflow/core/serialization-format entrypoint exists to avoid. I hit this as a build failure and moved the key helpers to a re-export from that browser-safe entrypoint; the two browser consumers import from there, while the CLI keeps the direct import since it runs on Node.

Verified by walking the built import graph: serialization-format.js reaches 6 modules and zero Node built-ins.

Tests

Two new cases assert the o11y path opens a real sealed payload with the derived keypair, and that the same resolved key still opens the run's own symmetric payloads — a run's event log mixes both.

Core suite 1598 passing, web-shared 160, CLI 78. Zero new lint diagnostics (diffed against a stashed baseline).

Without this, any payload another run sealed to this one renders as a lock
icon with no way to open it — a visible regression for anyone debugging a
run that received a cross-deployment hook resumption. The user is entitled
to read the data and has already supplied the key; only the plumbing was
missing.
`hydrateDataWithKey` now delegates to the envelope layer, which dispatches
on the format prefix, instead of unconditionally running AES-GCM. All four
o11y key-resolution sites (web-shared hydration, the web stream reader,
and both CLI `--decrypt` paths) resolve the full capability rather than
just the symmetric key. Each already had the raw 32 bytes in hand, so this
costs one extra derivation and no additional requests.
A caller that supplies only a symmetric key still gets the ciphertext
placeholder for sealed payloads rather than a decryption error, since that
key never could have opened them.
**Browser bundling.** The obvious import for the new helper is
`@workflow/core/serialization`, but that module graph reaches `node:util`
and `node:async_hooks` and cannot be bundled for the browser — which is
what `@workflow/core/serialization-format` exists to avoid. The key
helpers are re-exported from that browser-safe entrypoint instead, and the
two browser consumers import from there; the CLI keeps the direct import
since it runs on Node. Verified by walking the built import graph: the
entrypoint reaches 6 modules and zero Node built-ins.
Unrelated: `pnpm --filter @workflow/web build` currently fails on `main`
too (`reducers/common.js` importing `node:util`). Turbo caching had been
hiding it; touching core caused a cache miss that surfaced it. Not
addressed here.
- `hydrateDataWithKey` accepted `PayloadKey`, which includes `SealTarget`.
A seal target holds only a public key, so it can open neither scheme —
passing one compiled fine and then always failed at runtime. Added a
`DecryptionKey` alias (`CryptoKey | RunPayloadKeys`) and narrowed the
signature, so that misuse is now a compile error. A `@ts-expect-error`
test pins the guarantee.
- `hydrateResourceIOAsync` dynamically imported
`@workflow/core/serialization-format` twice. Destructure both bindings
from the single existing import instead.
@TooTallNate
TooTallNate requested review from a team and ijjk as code ownersJuly 27, 2026 23:49
CopilotAI review requested due to automatic review settings July 27, 2026 23:49
@changeset-bot

changeset-botBot commented Jul 27, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 916bc93

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 16 packages
NameType
@workflow/coreMinor
@workflow/web-sharedPatch
@workflow/webPatch
@workflow/cliPatch
@workflow/buildersPatch
@workflow/nextPatch
@workflow/nitroPatch
@workflow/vitestPatch
workflowMinor
@workflow/world-testingPatch
@workflow/astroPatch
@workflow/nestPatch
@workflow/nuxtPatch
@workflow/rollupPatch
@workflow/sveltekitPatch
@workflow/vitePatch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@vercel

vercelBot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

@github-actions

github-actionsBot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

📊 Workflow Benchmarks

commit 916bc93 · Tue, 28 Jul 2026 00:26:55 GMT · run logs

Backend: vercel · app: nextjs-turbopack

MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep228 (-69%) 💚1407 🔴 (+24%) 🔻1449 🔴 (+20%) 🔻1751 🔴 (+28%) 🔻30
TTFSstream273 (-13%)1428 🔴 (+30%) 🔻1438 🔴 (+26%) 🔻1527 🔴 (+30%) 🔻30
TTFShook + stream572 (-55%) 💚1694 🔴 (+21%) 🔻1740 🔴 (+19%) 🔻1895 🔴 (+16%) 🔻30
STSO1020 steps (1-20)185 (+8.8%)327 🔴 (+8.3%)550 🔴 (+56%) 🔻681 🔴 (+80%) 🔻19
STSO1020 steps (101-120)205 (+4.1%)298 🔴 (-12%)473 🔴 (+16%) 🔻598 🔴 (+20%) 🔻19
STSO1020 steps (1001-1020)473 (-2.1%)563 🔴 (-1.9%)669 🔴 (+3.6%)852 🔴 (+30%) 🔻19
WO1020 steps410726 (-6.4%)410726 (-6.4%)410726 (-6.4%)410726 (-6.4%)1
SLstream latency97 (+4.3%)196 🔴 (+17%) 🔻257 🔴 (+9.8%)592 🔴 (+134%) 🔻30
SOstream overhead (text)126 (-3.8%)303 🔴 (+4.5%)400 (+13%)543 (+29%) 🔻30
SOstream overhead (structured)135 (±0%)391 🔴 (+50%) 🔻729 🔴 (+162%) 🔻5527 🔴 (+1238%) 🔻30
ℹ️ Metric definitions & methodology

Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.

Metrics — TTFS: time to first step body (in-deployment start() → first step body, deployment clocks) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · SL: stream latency (in-deployment write → read propagation, readAt - writtenAt) · SO: stream overhead (end-to-end write+consume time beyond the modelled generation window)

Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · stream latency: parallel reader/writer steps on a dedicated stream; SL is the in-deployment write->read propagation (readAt - writtenAt) · stream overhead (text): writer streams 300 variable-length text token deltas paced at 100/s for 3s (a haiku-size LLM's token throughput) while a parallel reader drains the whole stream; SO is the end-to-end write+consume time beyond the 3s generation window (overhead/backpressure) · stream overhead (structured): same workload as stream overhead (text), but each delta is an AI-SDK-style structured object ({ type: 'text-delta', id, text }) instead of a raw string, so the SO gap vs the text scenario is the added serialization cost

🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · SO 250/500/1000 · STSO (1-20) 20/30/60 · STSO (101-120) 30/45/90 · STSO (1001-1020) 40/60/120

All metrics are measured from deployment-side timestamps only. Runs are triggered by an in-deployment route that stamps the anchor (clientStart) right before start(), so the CI runner’s request and its path through api.vercel.com sit outside every measured window. TTFS = in-deployment start() → first step body (turbo uses the in-process fast path, non-turbo the dispatch path), and includes the VQS dispatch hop plus any /flow cold start. STSO/WO are measured between step bodies on the deployment. SL is measured inside the workflow (parallel reader/writer steps), so it no longer includes the api.vercel.com read path.

Cold starts are kept in the numbers on purpose — they are part of real bursty-workload latency. The workbench deployment cold-starts the /flow invocation for a large fraction of runs, inflating P75+; the Best column shows the fastest (warm-start) sample for comparison.

@github-actions

github-actionsBot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

🧪 E2E Test Results

All tests passed

E2E Test Summary

Summary
PassedFailedSkippedTotal
✅ ▲ Vercel Production145502391694
✅ 💻 Local Development162102271848
✅ 📦 Local Production162102271848
✅ 🐘 Local Postgres162102271848
✅ 🪟 Windows15400154
✅ 📋 Other102002121232
✅ vercel-multi-region270027
Total7519011328651
Details by Category

✅ ▲ Vercel Production

AppPassedFailedSkipped
✅ astro126028
✅ example126028
✅ express126028
✅ fastify126028
✅ hono126028
✅ nextjs-turbopack15103
✅ nextjs-webpack15103
✅ nitro126028
✅ nuxt126028
✅ sveltekit14509
✅ vite126028

✅ 💻 Local Development

AppPassedFailedSkipped
✅ astro-stable128026
✅ express-stable128026
✅ fastify-stable128026
✅ hono-stable128026
✅ nextjs-turbopack-canary135019
✅ nextjs-turbopack-stable15400
✅ nextjs-webpack-canary135019
✅ nextjs-webpack-stable15400
✅ nitro-stable128026
✅ nuxt-stable128026
✅ sveltekit-stable14707
✅ vite-stable128026

✅ 📦 Local Production

AppPassedFailedSkipped
✅ astro-stable128026
✅ express-stable128026
✅ fastify-stable128026
✅ hono-stable128026
✅ nextjs-turbopack-canary135019
✅ nextjs-turbopack-stable15400
✅ nextjs-webpack-canary135019
✅ nextjs-webpack-stable15400
✅ nitro-stable128026
✅ nuxt-stable128026
✅ sveltekit-stable14707
✅ vite-stable128026

✅ 🐘 Local Postgres

AppPassedFailedSkipped
✅ astro-stable128026
✅ express-stable128026
✅ fastify-stable128026
✅ hono-stable128026
✅ nextjs-turbopack-canary135019
✅ nextjs-turbopack-stable15400
✅ nextjs-webpack-canary135019
✅ nextjs-webpack-stable15400
✅ nitro-stable128026
✅ nuxt-stable128026
✅ sveltekit-stable14707
✅ vite-stable128026

✅ 🪟 Windows

AppPassedFailedSkipped
✅ nextjs-turbopack15400

✅ 📋 Other

AppPassedFailedSkipped
✅ e2e-local-dev-nest-stable128026
✅ e2e-local-dev-tanstack-start-128026
✅ e2e-local-postgres-nest-stable128026
✅ e2e-local-postgres-tanstack-start-128026
✅ e2e-local-prod-nest-stable128026
✅ e2e-local-prod-tanstack-start-128026
✅ e2e-vercel-prod-nest126028
✅ e2e-vercel-prod-tanstack-start126028

✅ vercel-multi-region

AppPassedFailedSkipped
✅ nextjs-turbopack2700

📋 View full workflow run

@karthikscale3karthikscale3 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ai review: The decryption changes look correct, but the published @workflow/web package is missing from the changeset.

Comment thread.changeset/encp-o11y-decrypt.md

@karthikscale3karthikscale3 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at 85e262c. Implementation looks good; the existing inline changeset note is non-blocking.

This PR changes the dashboard's stream reader
(`packages/web/app/lib/hooks/use-stream-reader.ts`) so it dispatches on the
envelope format and can read sealed (`encp`) frames, but the changeset listed
only core, web-shared and cli.
`@workflow/web` is published, so without an entry the change would still ship —
just as an incidental dependency bump, with nothing in that package's release
notes explaining that sealed-stream decryption landed.
@TooTallNate
TooTallNate enabled auto-merge (squash) July 28, 2026 00:14
@TooTallNate
TooTallNate merged commit e8bc7d6 into mainJul 28, 2026
174 of 176 checks passed
@TooTallNate
TooTallNate deleted the nate/encp-o11y-decrypt branch July 28, 2026 00:36
@github-actions

Copy link
Copy Markdown
Contributor

No backport to stable for e8bc7d6 (AI decision).

This is feature work — PR 5 of 7 in a stack that introduces the sealed-box (encp) payload scheme, adding new capability (deriveRunPayloadKeys, DecryptionKey, browser-safe re-exports) to observability tooling. It also builds directly on main-only APIs: git ls-tree origin/stable confirms packages/core/src/sealed-box.ts and packages/core/src/serialization/encryption.ts do not exist on stable, so the cherry-pick could not even work. The "regression" it fixes only materializes once the rest of the sealed-payload stack lands, which is not on the maintenance line.

To override, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:

e8bc7d6aadcdb5f89de1be71cfcafe5e9085ed25

@github-actionsgithub-actionsBot mentioned this pull request Jul 28, 2026
pranaygp added a commit that referenced this pull request Jul 28, 2026
…ry-2
* origin/main: (292 commits)
feat(core): seal forwarded stream writes to the owner's public key (#3098)
feat(core): seal hook payloads to the target run's public key (#3096)
[e2e] Rebuild the event-log corruption repro around step-count divergence (#3147)
feat: decrypt sealed payloads in the dashboard and CLI (#3146)
Prewarm only appended replay payloads (#3131)
feat: publish each run's X25519 public key on the run entity (#3095)
feat(core): route sealed envelopes through the serialization layer (#3094)
docs: redirect retired migration-guides URLs to comparisons (#3127)
feat(core): add `encp` sealed-box encryption primitive (#3093)
chore(core): clarify runtime comments (#3111)
Remove obsolete world factory aliases (#3112)
feat(core): deterministic sandbox hardening (#3045)
Remove retired v1 step route plumbing (#3061)
[core] Don't count racing invocations' duplicate step_started events toward the maxRetries ceiling (#3069)
[world-testing] Isolate each spawned test server's data directory (#3055)
fix: upgrade postcss to >=8.5.18 to address GHSA-r28c-9q8g-f849 (#3102)
[next] Respect .gitignore in dev watcher to avoid EMFILE on large monorepos (#3085)
[ci] Backport only stability fixes to `stable`, default to claude-opus-5 (#3092)
perf(core): immediate leading-edge dispatch for idle streams (flush window default 0) (#3088)
Optimize `processImportSpecifier` by computing `shouldFollowImportsFromFile` once per file (#3052)
...
# Conflicts:
#	docs/components/geistdocs/desktop-menu.tsx
#	docs/components/geistdocs/mobile-menu.tsx
#	docs/content/docs/v5/cookbook/advanced/child-workflows.mdx
#	docs/content/docs/v5/cookbook/advanced/upgrading-workflows.mdx
#	docs/content/docs/v5/cookbook/agent-patterns/agent-cancellation.mdx
#	docs/content/docs/v5/cookbook/agent-patterns/durable-agent.mdx
#	docs/content/docs/v5/cookbook/agent-patterns/human-in-the-loop.mdx
#	docs/content/docs/v5/cookbook/common-patterns/batching.mdx
#	docs/content/docs/v5/cookbook/common-patterns/idempotency.mdx
#	docs/content/docs/v5/cookbook/common-patterns/rate-limiting.mdx
#	docs/content/docs/v5/cookbook/common-patterns/saga.mdx
#	docs/content/docs/v5/cookbook/common-patterns/scheduling.mdx
#	docs/content/docs/v5/cookbook/common-patterns/sequential-and-parallel.mdx
#	docs/content/docs/v5/cookbook/common-patterns/timeouts.mdx
#	docs/content/docs/v5/cookbook/common-patterns/webhooks.mdx
#	docs/content/docs/v5/cookbook/common-patterns/workflow-composition.mdx
#	docs/content/docs/v5/cookbook/index.mdx
#	docs/content/docs/v5/cookbook/integrations/ai-sdk.mdx
#	docs/content/docs/v5/cookbook/integrations/chat-sdk.mdx
#	docs/content/docs/v5/cookbook/integrations/sandbox.mdx
#	docs/next.config.ts
#	docs/proxy.ts
#	docs/scripts/lint.ts
#	pnpm-lock.yaml
#	pnpm-workspace.yaml
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@TooTallNate@karthikscale3
, 'i'); if (__m === '*' || __re.test(location.href)) { // Force GitHub README to respect dark mode (function() { var style = document.createElement('style'); style.textContent = ' .markdown-body { color-scheme: dark light; } .markdown-body pre { background: #161b22 !important; } .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; } .markdown-body table th, .markdown-body table td { border-color: #30363d !important; } .markdown-body img { background: #0d1117; } .markdown-body blockquote { border-left-color: #8b949e; } .markdown-body hr { border-color: #30363d; } '; document.head.appendChild(style); })(); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' feat: decrypt sealed payloads in the dashboard and CLI by TooTallNate · Pull Request #3146 · vercel/workflow · GitHub
Skip to content

feat: decrypt sealed payloads in the dashboard and CLI - #3146

Merged
TooTallNate merged 3 commits into
mainfrom
nate/encp-o11y-decrypt
Jul 28, 2026
Merged

feat: decrypt sealed payloads in the dashboard and CLI#3146
TooTallNate merged 3 commits into
mainfrom
nate/encp-o11y-decrypt

Conversation

@TooTallNate

Copy link
Copy Markdown
Member

Replaces #3097. That PR was force-closed as "merged" by GitHub when I
force-pushed its base branch to a commit that happened to contain its head —
GitHub concluded it had merged into its base. Its code never reached main.
It cannot be reopened, so this PR carries the identical work. Review history
on #3097 still applies.

Ordering change: this now targets main directly and should land before
#3096, per the review there. #3096 begins emitting sealed (encp) hook payloads
while workflow inspect --decrypt can only build a symmetric key, so merging it
first would leave those payloads showing as ciphertext placeholders. Landing the
reader first closes that window.

To make this PR stand alone, deriveRunPayloadKeys moved here from #3096 — it
is the capability the reader needs, and its own doc says to use it "anywhere a
run reads its own event log", so it belongs on this side regardless.


PR 5 of 7. Stacked on #3096; review the stack in order, starting from #3093.

Why this is required, not optional

Once #3096 lands, a hook resumption from another deployment writes a sealed payload into the run's event log. Observability tooling can't open it: hydrateDataWithKey only ever ran AES-GCM. So the dashboard and CLI would render a 🔒 placeholder on data the user is entitled to read and has already supplied the key for — a visible debugging regression on exactly the runs this work speeds up.

What changed

hydrateDataWithKey delegates to the envelope layer, which dispatches on the 4-byte format prefix, instead of unconditionally decrypting with AES-GCM.

All four o11y key-resolution sites now resolve the run's full key capability rather than just its symmetric key:

SitePath
@workflow/web-sharedhydrateResourceIOAsync
@workflow/webuse-stream-reader (hand-rolled frame decode)
@workflow/cli--decrypt field hydration
@workflow/cli--decrypt stream output

Each already had the raw 32 bytes in hand, so this costs one extra key derivation and no additional requests or round trips.

A caller that supplies only a symmetric key still gets the ciphertext placeholder for sealed payloads rather than a decryption error — that key never could have opened them, so surfacing a failure would be misleading.

Browser bundling

The natural import for the new helper is @workflow/core/serialization, but that module graph reaches node:util and node:async_hooks and cannot be bundled for the browser — which is precisely what the @workflow/core/serialization-format entrypoint exists to avoid. I hit this as a build failure and moved the key helpers to a re-export from that browser-safe entrypoint; the two browser consumers import from there, while the CLI keeps the direct import since it runs on Node.

Verified by walking the built import graph: serialization-format.js reaches 6 modules and zero Node built-ins.

Tests

Two new cases assert the o11y path opens a real sealed payload with the derived keypair, and that the same resolved key still opens the run's own symmetric payloads — a run's event log mixes both.

Core suite 1598 passing, web-shared 160, CLI 78. Zero new lint diagnostics (diffed against a stashed baseline).

Without this, any payload another run sealed to this one renders as a lock
icon with no way to open it — a visible regression for anyone debugging a
run that received a cross-deployment hook resumption. The user is entitled
to read the data and has already supplied the key; only the plumbing was
missing.
`hydrateDataWithKey` now delegates to the envelope layer, which dispatches
on the format prefix, instead of unconditionally running AES-GCM. All four
o11y key-resolution sites (web-shared hydration, the web stream reader,
and both CLI `--decrypt` paths) resolve the full capability rather than
just the symmetric key. Each already had the raw 32 bytes in hand, so this
costs one extra derivation and no additional requests.
A caller that supplies only a symmetric key still gets the ciphertext
placeholder for sealed payloads rather than a decryption error, since that
key never could have opened them.
**Browser bundling.** The obvious import for the new helper is
`@workflow/core/serialization`, but that module graph reaches `node:util`
and `node:async_hooks` and cannot be bundled for the browser — which is
what `@workflow/core/serialization-format` exists to avoid. The key
helpers are re-exported from that browser-safe entrypoint instead, and the
two browser consumers import from there; the CLI keeps the direct import
since it runs on Node. Verified by walking the built import graph: the
entrypoint reaches 6 modules and zero Node built-ins.
Unrelated: `pnpm --filter @workflow/web build` currently fails on `main`
too (`reducers/common.js` importing `node:util`). Turbo caching had been
hiding it; touching core caused a cache miss that surfaced it. Not
addressed here.
- `hydrateDataWithKey` accepted `PayloadKey`, which includes `SealTarget`.
A seal target holds only a public key, so it can open neither scheme —
passing one compiled fine and then always failed at runtime. Added a
`DecryptionKey` alias (`CryptoKey | RunPayloadKeys`) and narrowed the
signature, so that misuse is now a compile error. A `@ts-expect-error`
test pins the guarantee.
- `hydrateResourceIOAsync` dynamically imported
`@workflow/core/serialization-format` twice. Destructure both bindings
from the single existing import instead.
@TooTallNate
TooTallNate requested review from a team and ijjk as code ownersJuly 27, 2026 23:49
CopilotAI review requested due to automatic review settings July 27, 2026 23:49
@changeset-bot

changeset-botBot commented Jul 27, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 916bc93

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 16 packages
NameType
@workflow/coreMinor
@workflow/web-sharedPatch
@workflow/webPatch
@workflow/cliPatch
@workflow/buildersPatch
@workflow/nextPatch
@workflow/nitroPatch
@workflow/vitestPatch
workflowMinor
@workflow/world-testingPatch
@workflow/astroPatch
@workflow/nestPatch
@workflow/nuxtPatch
@workflow/rollupPatch
@workflow/sveltekitPatch
@workflow/vitePatch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@vercel

vercelBot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

@github-actions

github-actionsBot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

📊 Workflow Benchmarks

commit 916bc93 · Tue, 28 Jul 2026 00:26:55 GMT · run logs

Backend: vercel · app: nextjs-turbopack

MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep228 (-69%) 💚1407 🔴 (+24%) 🔻1449 🔴 (+20%) 🔻1751 🔴 (+28%) 🔻30
TTFSstream273 (-13%)1428 🔴 (+30%) 🔻1438 🔴 (+26%) 🔻1527 🔴 (+30%) 🔻30
TTFShook + stream572 (-55%) 💚1694 🔴 (+21%) 🔻1740 🔴 (+19%) 🔻1895 🔴 (+16%) 🔻30
STSO1020 steps (1-20)185 (+8.8%)327 🔴 (+8.3%)550 🔴 (+56%) 🔻681 🔴 (+80%) 🔻19
STSO1020 steps (101-120)205 (+4.1%)298 🔴 (-12%)473 🔴 (+16%) 🔻598 🔴 (+20%) 🔻19
STSO1020 steps (1001-1020)473 (-2.1%)563 🔴 (-1.9%)669 🔴 (+3.6%)852 🔴 (+30%) 🔻19
WO1020 steps410726 (-6.4%)410726 (-6.4%)410726 (-6.4%)410726 (-6.4%)1
SLstream latency97 (+4.3%)196 🔴 (+17%) 🔻257 🔴 (+9.8%)592 🔴 (+134%) 🔻30
SOstream overhead (text)126 (-3.8%)303 🔴 (+4.5%)400 (+13%)543 (+29%) 🔻30
SOstream overhead (structured)135 (±0%)391 🔴 (+50%) 🔻729 🔴 (+162%) 🔻5527 🔴 (+1238%) 🔻30
ℹ️ Metric definitions & methodology

Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.

Metrics — TTFS: time to first step body (in-deployment start() → first step body, deployment clocks) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · SL: stream latency (in-deployment write → read propagation, readAt - writtenAt) · SO: stream overhead (end-to-end write+consume time beyond the modelled generation window)

Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · stream latency: parallel reader/writer steps on a dedicated stream; SL is the in-deployment write->read propagation (readAt - writtenAt) · stream overhead (text): writer streams 300 variable-length text token deltas paced at 100/s for 3s (a haiku-size LLM's token throughput) while a parallel reader drains the whole stream; SO is the end-to-end write+consume time beyond the 3s generation window (overhead/backpressure) · stream overhead (structured): same workload as stream overhead (text), but each delta is an AI-SDK-style structured object ({ type: 'text-delta', id, text }) instead of a raw string, so the SO gap vs the text scenario is the added serialization cost

🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · SO 250/500/1000 · STSO (1-20) 20/30/60 · STSO (101-120) 30/45/90 · STSO (1001-1020) 40/60/120

All metrics are measured from deployment-side timestamps only. Runs are triggered by an in-deployment route that stamps the anchor (clientStart) right before start(), so the CI runner’s request and its path through api.vercel.com sit outside every measured window. TTFS = in-deployment start() → first step body (turbo uses the in-process fast path, non-turbo the dispatch path), and includes the VQS dispatch hop plus any /flow cold start. STSO/WO are measured between step bodies on the deployment. SL is measured inside the workflow (parallel reader/writer steps), so it no longer includes the api.vercel.com read path.

Cold starts are kept in the numbers on purpose — they are part of real bursty-workload latency. The workbench deployment cold-starts the /flow invocation for a large fraction of runs, inflating P75+; the Best column shows the fastest (warm-start) sample for comparison.

@github-actions

github-actionsBot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

🧪 E2E Test Results

All tests passed

E2E Test Summary

Summary
PassedFailedSkippedTotal
✅ ▲ Vercel Production145502391694
✅ 💻 Local Development162102271848
✅ 📦 Local Production162102271848
✅ 🐘 Local Postgres162102271848
✅ 🪟 Windows15400154
✅ 📋 Other102002121232
✅ vercel-multi-region270027
Total7519011328651
Details by Category

✅ ▲ Vercel Production

AppPassedFailedSkipped
✅ astro126028
✅ example126028
✅ express126028
✅ fastify126028
✅ hono126028
✅ nextjs-turbopack15103
✅ nextjs-webpack15103
✅ nitro126028
✅ nuxt126028
✅ sveltekit14509
✅ vite126028

✅ 💻 Local Development

AppPassedFailedSkipped
✅ astro-stable128026
✅ express-stable128026
✅ fastify-stable128026
✅ hono-stable128026
✅ nextjs-turbopack-canary135019
✅ nextjs-turbopack-stable15400
✅ nextjs-webpack-canary135019
✅ nextjs-webpack-stable15400
✅ nitro-stable128026
✅ nuxt-stable128026
✅ sveltekit-stable14707
✅ vite-stable128026

✅ 📦 Local Production

AppPassedFailedSkipped
✅ astro-stable128026
✅ express-stable128026
✅ fastify-stable128026
✅ hono-stable128026
✅ nextjs-turbopack-canary135019
✅ nextjs-turbopack-stable15400
✅ nextjs-webpack-canary135019
✅ nextjs-webpack-stable15400
✅ nitro-stable128026
✅ nuxt-stable128026
✅ sveltekit-stable14707
✅ vite-stable128026

✅ 🐘 Local Postgres

AppPassedFailedSkipped
✅ astro-stable128026
✅ express-stable128026
✅ fastify-stable128026
✅ hono-stable128026
✅ nextjs-turbopack-canary135019
✅ nextjs-turbopack-stable15400
✅ nextjs-webpack-canary135019
✅ nextjs-webpack-stable15400
✅ nitro-stable128026
✅ nuxt-stable128026
✅ sveltekit-stable14707
✅ vite-stable128026

✅ 🪟 Windows

AppPassedFailedSkipped
✅ nextjs-turbopack15400

✅ 📋 Other

AppPassedFailedSkipped
✅ e2e-local-dev-nest-stable128026
✅ e2e-local-dev-tanstack-start-128026
✅ e2e-local-postgres-nest-stable128026
✅ e2e-local-postgres-tanstack-start-128026
✅ e2e-local-prod-nest-stable128026
✅ e2e-local-prod-tanstack-start-128026
✅ e2e-vercel-prod-nest126028
✅ e2e-vercel-prod-tanstack-start126028

✅ vercel-multi-region

AppPassedFailedSkipped
✅ nextjs-turbopack2700

📋 View full workflow run

@karthikscale3karthikscale3 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ai review: The decryption changes look correct, but the published @workflow/web package is missing from the changeset.

Comment thread.changeset/encp-o11y-decrypt.md

@karthikscale3karthikscale3 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at 85e262c. Implementation looks good; the existing inline changeset note is non-blocking.

This PR changes the dashboard's stream reader
(`packages/web/app/lib/hooks/use-stream-reader.ts`) so it dispatches on the
envelope format and can read sealed (`encp`) frames, but the changeset listed
only core, web-shared and cli.
`@workflow/web` is published, so without an entry the change would still ship —
just as an incidental dependency bump, with nothing in that package's release
notes explaining that sealed-stream decryption landed.
@TooTallNate
TooTallNate enabled auto-merge (squash) July 28, 2026 00:14
@TooTallNate
TooTallNate merged commit e8bc7d6 into mainJul 28, 2026
174 of 176 checks passed
@TooTallNate
TooTallNate deleted the nate/encp-o11y-decrypt branch July 28, 2026 00:36
@github-actions

Copy link
Copy Markdown
Contributor

No backport to stable for e8bc7d6 (AI decision).

This is feature work — PR 5 of 7 in a stack that introduces the sealed-box (encp) payload scheme, adding new capability (deriveRunPayloadKeys, DecryptionKey, browser-safe re-exports) to observability tooling. It also builds directly on main-only APIs: git ls-tree origin/stable confirms packages/core/src/sealed-box.ts and packages/core/src/serialization/encryption.ts do not exist on stable, so the cherry-pick could not even work. The "regression" it fixes only materializes once the rest of the sealed-payload stack lands, which is not on the maintenance line.

To override, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:

e8bc7d6aadcdb5f89de1be71cfcafe5e9085ed25

@github-actionsgithub-actionsBot mentioned this pull request Jul 28, 2026
pranaygp added a commit that referenced this pull request Jul 28, 2026
…ry-2
* origin/main: (292 commits)
feat(core): seal forwarded stream writes to the owner's public key (#3098)
feat(core): seal hook payloads to the target run's public key (#3096)
[e2e] Rebuild the event-log corruption repro around step-count divergence (#3147)
feat: decrypt sealed payloads in the dashboard and CLI (#3146)
Prewarm only appended replay payloads (#3131)
feat: publish each run's X25519 public key on the run entity (#3095)
feat(core): route sealed envelopes through the serialization layer (#3094)
docs: redirect retired migration-guides URLs to comparisons (#3127)
feat(core): add `encp` sealed-box encryption primitive (#3093)
chore(core): clarify runtime comments (#3111)
Remove obsolete world factory aliases (#3112)
feat(core): deterministic sandbox hardening (#3045)
Remove retired v1 step route plumbing (#3061)
[core] Don't count racing invocations' duplicate step_started events toward the maxRetries ceiling (#3069)
[world-testing] Isolate each spawned test server's data directory (#3055)
fix: upgrade postcss to >=8.5.18 to address GHSA-r28c-9q8g-f849 (#3102)
[next] Respect .gitignore in dev watcher to avoid EMFILE on large monorepos (#3085)
[ci] Backport only stability fixes to `stable`, default to claude-opus-5 (#3092)
perf(core): immediate leading-edge dispatch for idle streams (flush window default 0) (#3088)
Optimize `processImportSpecifier` by computing `shouldFollowImportsFromFile` once per file (#3052)
...
# Conflicts:
#	docs/components/geistdocs/desktop-menu.tsx
#	docs/components/geistdocs/mobile-menu.tsx
#	docs/content/docs/v5/cookbook/advanced/child-workflows.mdx
#	docs/content/docs/v5/cookbook/advanced/upgrading-workflows.mdx
#	docs/content/docs/v5/cookbook/agent-patterns/agent-cancellation.mdx
#	docs/content/docs/v5/cookbook/agent-patterns/durable-agent.mdx
#	docs/content/docs/v5/cookbook/agent-patterns/human-in-the-loop.mdx
#	docs/content/docs/v5/cookbook/common-patterns/batching.mdx
#	docs/content/docs/v5/cookbook/common-patterns/idempotency.mdx
#	docs/content/docs/v5/cookbook/common-patterns/rate-limiting.mdx
#	docs/content/docs/v5/cookbook/common-patterns/saga.mdx
#	docs/content/docs/v5/cookbook/common-patterns/scheduling.mdx
#	docs/content/docs/v5/cookbook/common-patterns/sequential-and-parallel.mdx
#	docs/content/docs/v5/cookbook/common-patterns/timeouts.mdx
#	docs/content/docs/v5/cookbook/common-patterns/webhooks.mdx
#	docs/content/docs/v5/cookbook/common-patterns/workflow-composition.mdx
#	docs/content/docs/v5/cookbook/index.mdx
#	docs/content/docs/v5/cookbook/integrations/ai-sdk.mdx
#	docs/content/docs/v5/cookbook/integrations/chat-sdk.mdx
#	docs/content/docs/v5/cookbook/integrations/sandbox.mdx
#	docs/next.config.ts
#	docs/proxy.ts
#	docs/scripts/lint.ts
#	pnpm-lock.yaml
#	pnpm-workspace.yaml
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@TooTallNate@karthikscale3
, 'i'); if (__m === '*' || __re.test(location.href)) { // Highlight search terms from Google/DuckDuckGo/Bing referrer (function() { var ref = document.referrer; var terms = []; if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) { var url = new URL(ref); var q = url.searchParams.get('q') || url.searchParams.get('p'); if (q) { terms = q.split(/\s+/).filter(function(t) { return t.length > 2; }); } } if (terms.length === 0) return; var style = document.createElement('style'); style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }'; document.head.appendChild(style); function highlight(node) { if (node.nodeType === 3) { // text node var text = node.textContent; var found = false; terms.forEach(function(term) { var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi'); if (regex.test(text)) { found = true; var frag = document.createDocumentFragment(); var parts = text.split(regex); parts.forEach(function(part, i) { if (i % 2 === 0) { frag.appendChild(document.createTextNode(part)); } else { var span = document.createElement('span'); span.className = 'userscript-highlight'; span.textContent = part; frag.appendChild(span); } }); node.parentNode.replaceChild(frag, node); } }); } else if (node.nodeType === 1 && node.childNodes) { // element var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT']; if (!skipTags.includes(node.tagName)) { Array.from(node.childNodes).forEach(highlight); } } } highlight(document.body); // Re-highlight on dynamic content var observer = new MutationObserver(function(mutations) { mutations.forEach(function(m) { m.addedNodes.forEach(function(node) { if (node.nodeType === 1 || node.nodeType === 3) highlight(node); }); }); }); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' feat: decrypt sealed payloads in the dashboard and CLI by TooTallNate · Pull Request #3146 · vercel/workflow · GitHub
Skip to content

feat: decrypt sealed payloads in the dashboard and CLI - #3146

Merged
TooTallNate merged 3 commits into
mainfrom
nate/encp-o11y-decrypt
Jul 28, 2026
Merged

feat: decrypt sealed payloads in the dashboard and CLI#3146
TooTallNate merged 3 commits into
mainfrom
nate/encp-o11y-decrypt

Conversation

@TooTallNate

Copy link
Copy Markdown
Member

Replaces #3097. That PR was force-closed as "merged" by GitHub when I
force-pushed its base branch to a commit that happened to contain its head —
GitHub concluded it had merged into its base. Its code never reached main.
It cannot be reopened, so this PR carries the identical work. Review history
on #3097 still applies.

Ordering change: this now targets main directly and should land before
#3096, per the review there. #3096 begins emitting sealed (encp) hook payloads
while workflow inspect --decrypt can only build a symmetric key, so merging it
first would leave those payloads showing as ciphertext placeholders. Landing the
reader first closes that window.

To make this PR stand alone, deriveRunPayloadKeys moved here from #3096 — it
is the capability the reader needs, and its own doc says to use it "anywhere a
run reads its own event log", so it belongs on this side regardless.


PR 5 of 7. Stacked on #3096; review the stack in order, starting from #3093.

Why this is required, not optional

Once #3096 lands, a hook resumption from another deployment writes a sealed payload into the run's event log. Observability tooling can't open it: hydrateDataWithKey only ever ran AES-GCM. So the dashboard and CLI would render a 🔒 placeholder on data the user is entitled to read and has already supplied the key for — a visible debugging regression on exactly the runs this work speeds up.

What changed

hydrateDataWithKey delegates to the envelope layer, which dispatches on the 4-byte format prefix, instead of unconditionally decrypting with AES-GCM.

All four o11y key-resolution sites now resolve the run's full key capability rather than just its symmetric key:

SitePath
@workflow/web-sharedhydrateResourceIOAsync
@workflow/webuse-stream-reader (hand-rolled frame decode)
@workflow/cli--decrypt field hydration
@workflow/cli--decrypt stream output

Each already had the raw 32 bytes in hand, so this costs one extra key derivation and no additional requests or round trips.

A caller that supplies only a symmetric key still gets the ciphertext placeholder for sealed payloads rather than a decryption error — that key never could have opened them, so surfacing a failure would be misleading.

Browser bundling

The natural import for the new helper is @workflow/core/serialization, but that module graph reaches node:util and node:async_hooks and cannot be bundled for the browser — which is precisely what the @workflow/core/serialization-format entrypoint exists to avoid. I hit this as a build failure and moved the key helpers to a re-export from that browser-safe entrypoint; the two browser consumers import from there, while the CLI keeps the direct import since it runs on Node.

Verified by walking the built import graph: serialization-format.js reaches 6 modules and zero Node built-ins.

Tests

Two new cases assert the o11y path opens a real sealed payload with the derived keypair, and that the same resolved key still opens the run's own symmetric payloads — a run's event log mixes both.

Core suite 1598 passing, web-shared 160, CLI 78. Zero new lint diagnostics (diffed against a stashed baseline).

Without this, any payload another run sealed to this one renders as a lock
icon with no way to open it — a visible regression for anyone debugging a
run that received a cross-deployment hook resumption. The user is entitled
to read the data and has already supplied the key; only the plumbing was
missing.
`hydrateDataWithKey` now delegates to the envelope layer, which dispatches
on the format prefix, instead of unconditionally running AES-GCM. All four
o11y key-resolution sites (web-shared hydration, the web stream reader,
and both CLI `--decrypt` paths) resolve the full capability rather than
just the symmetric key. Each already had the raw 32 bytes in hand, so this
costs one extra derivation and no additional requests.
A caller that supplies only a symmetric key still gets the ciphertext
placeholder for sealed payloads rather than a decryption error, since that
key never could have opened them.
**Browser bundling.** The obvious import for the new helper is
`@workflow/core/serialization`, but that module graph reaches `node:util`
and `node:async_hooks` and cannot be bundled for the browser — which is
what `@workflow/core/serialization-format` exists to avoid. The key
helpers are re-exported from that browser-safe entrypoint instead, and the
two browser consumers import from there; the CLI keeps the direct import
since it runs on Node. Verified by walking the built import graph: the
entrypoint reaches 6 modules and zero Node built-ins.
Unrelated: `pnpm --filter @workflow/web build` currently fails on `main`
too (`reducers/common.js` importing `node:util`). Turbo caching had been
hiding it; touching core caused a cache miss that surfaced it. Not
addressed here.
- `hydrateDataWithKey` accepted `PayloadKey`, which includes `SealTarget`.
A seal target holds only a public key, so it can open neither scheme —
passing one compiled fine and then always failed at runtime. Added a
`DecryptionKey` alias (`CryptoKey | RunPayloadKeys`) and narrowed the
signature, so that misuse is now a compile error. A `@ts-expect-error`
test pins the guarantee.
- `hydrateResourceIOAsync` dynamically imported
`@workflow/core/serialization-format` twice. Destructure both bindings
from the single existing import instead.
@TooTallNate
TooTallNate requested review from a team and ijjk as code ownersJuly 27, 2026 23:49
CopilotAI review requested due to automatic review settings July 27, 2026 23:49
@changeset-bot

changeset-botBot commented Jul 27, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 916bc93

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 16 packages
NameType
@workflow/coreMinor
@workflow/web-sharedPatch
@workflow/webPatch
@workflow/cliPatch
@workflow/buildersPatch
@workflow/nextPatch
@workflow/nitroPatch
@workflow/vitestPatch
workflowMinor
@workflow/world-testingPatch
@workflow/astroPatch
@workflow/nestPatch
@workflow/nuxtPatch
@workflow/rollupPatch
@workflow/sveltekitPatch
@workflow/vitePatch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@vercel

vercelBot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

@github-actions

github-actionsBot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

📊 Workflow Benchmarks

commit 916bc93 · Tue, 28 Jul 2026 00:26:55 GMT · run logs

Backend: vercel · app: nextjs-turbopack

MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep228 (-69%) 💚1407 🔴 (+24%) 🔻1449 🔴 (+20%) 🔻1751 🔴 (+28%) 🔻30
TTFSstream273 (-13%)1428 🔴 (+30%) 🔻1438 🔴 (+26%) 🔻1527 🔴 (+30%) 🔻30
TTFShook + stream572 (-55%) 💚1694 🔴 (+21%) 🔻1740 🔴 (+19%) 🔻1895 🔴 (+16%) 🔻30
STSO1020 steps (1-20)185 (+8.8%)327 🔴 (+8.3%)550 🔴 (+56%) 🔻681 🔴 (+80%) 🔻19
STSO1020 steps (101-120)205 (+4.1%)298 🔴 (-12%)473 🔴 (+16%) 🔻598 🔴 (+20%) 🔻19
STSO1020 steps (1001-1020)473 (-2.1%)563 🔴 (-1.9%)669 🔴 (+3.6%)852 🔴 (+30%) 🔻19
WO1020 steps410726 (-6.4%)410726 (-6.4%)410726 (-6.4%)410726 (-6.4%)1
SLstream latency97 (+4.3%)196 🔴 (+17%) 🔻257 🔴 (+9.8%)592 🔴 (+134%) 🔻30
SOstream overhead (text)126 (-3.8%)303 🔴 (+4.5%)400 (+13%)543 (+29%) 🔻30
SOstream overhead (structured)135 (±0%)391 🔴 (+50%) 🔻729 🔴 (+162%) 🔻5527 🔴 (+1238%) 🔻30
ℹ️ Metric definitions & methodology

Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.

Metrics — TTFS: time to first step body (in-deployment start() → first step body, deployment clocks) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · SL: stream latency (in-deployment write → read propagation, readAt - writtenAt) · SO: stream overhead (end-to-end write+consume time beyond the modelled generation window)

Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · stream latency: parallel reader/writer steps on a dedicated stream; SL is the in-deployment write->read propagation (readAt - writtenAt) · stream overhead (text): writer streams 300 variable-length text token deltas paced at 100/s for 3s (a haiku-size LLM's token throughput) while a parallel reader drains the whole stream; SO is the end-to-end write+consume time beyond the 3s generation window (overhead/backpressure) · stream overhead (structured): same workload as stream overhead (text), but each delta is an AI-SDK-style structured object ({ type: 'text-delta', id, text }) instead of a raw string, so the SO gap vs the text scenario is the added serialization cost

🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · SO 250/500/1000 · STSO (1-20) 20/30/60 · STSO (101-120) 30/45/90 · STSO (1001-1020) 40/60/120

All metrics are measured from deployment-side timestamps only. Runs are triggered by an in-deployment route that stamps the anchor (clientStart) right before start(), so the CI runner’s request and its path through api.vercel.com sit outside every measured window. TTFS = in-deployment start() → first step body (turbo uses the in-process fast path, non-turbo the dispatch path), and includes the VQS dispatch hop plus any /flow cold start. STSO/WO are measured between step bodies on the deployment. SL is measured inside the workflow (parallel reader/writer steps), so it no longer includes the api.vercel.com read path.

Cold starts are kept in the numbers on purpose — they are part of real bursty-workload latency. The workbench deployment cold-starts the /flow invocation for a large fraction of runs, inflating P75+; the Best column shows the fastest (warm-start) sample for comparison.

@github-actions

github-actionsBot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

🧪 E2E Test Results

All tests passed

E2E Test Summary

Summary
PassedFailedSkippedTotal
✅ ▲ Vercel Production145502391694
✅ 💻 Local Development162102271848
✅ 📦 Local Production162102271848
✅ 🐘 Local Postgres162102271848
✅ 🪟 Windows15400154
✅ 📋 Other102002121232
✅ vercel-multi-region270027
Total7519011328651
Details by Category

✅ ▲ Vercel Production

AppPassedFailedSkipped
✅ astro126028
✅ example126028
✅ express126028
✅ fastify126028
✅ hono126028
✅ nextjs-turbopack15103
✅ nextjs-webpack15103
✅ nitro126028
✅ nuxt126028
✅ sveltekit14509
✅ vite126028

✅ 💻 Local Development

AppPassedFailedSkipped
✅ astro-stable128026
✅ express-stable128026
✅ fastify-stable128026
✅ hono-stable128026
✅ nextjs-turbopack-canary135019
✅ nextjs-turbopack-stable15400
✅ nextjs-webpack-canary135019
✅ nextjs-webpack-stable15400
✅ nitro-stable128026
✅ nuxt-stable128026
✅ sveltekit-stable14707
✅ vite-stable128026

✅ 📦 Local Production

AppPassedFailedSkipped
✅ astro-stable128026
✅ express-stable128026
✅ fastify-stable128026
✅ hono-stable128026
✅ nextjs-turbopack-canary135019
✅ nextjs-turbopack-stable15400
✅ nextjs-webpack-canary135019
✅ nextjs-webpack-stable15400
✅ nitro-stable128026
✅ nuxt-stable128026
✅ sveltekit-stable14707
✅ vite-stable128026

✅ 🐘 Local Postgres

AppPassedFailedSkipped
✅ astro-stable128026
✅ express-stable128026
✅ fastify-stable128026
✅ hono-stable128026
✅ nextjs-turbopack-canary135019
✅ nextjs-turbopack-stable15400
✅ nextjs-webpack-canary135019
✅ nextjs-webpack-stable15400
✅ nitro-stable128026
✅ nuxt-stable128026
✅ sveltekit-stable14707
✅ vite-stable128026

✅ 🪟 Windows

AppPassedFailedSkipped
✅ nextjs-turbopack15400

✅ 📋 Other

AppPassedFailedSkipped
✅ e2e-local-dev-nest-stable128026
✅ e2e-local-dev-tanstack-start-128026
✅ e2e-local-postgres-nest-stable128026
✅ e2e-local-postgres-tanstack-start-128026
✅ e2e-local-prod-nest-stable128026
✅ e2e-local-prod-tanstack-start-128026
✅ e2e-vercel-prod-nest126028
✅ e2e-vercel-prod-tanstack-start126028

✅ vercel-multi-region

AppPassedFailedSkipped
✅ nextjs-turbopack2700

📋 View full workflow run

@karthikscale3karthikscale3 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ai review: The decryption changes look correct, but the published @workflow/web package is missing from the changeset.

Comment thread.changeset/encp-o11y-decrypt.md

@karthikscale3karthikscale3 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at 85e262c. Implementation looks good; the existing inline changeset note is non-blocking.

This PR changes the dashboard's stream reader
(`packages/web/app/lib/hooks/use-stream-reader.ts`) so it dispatches on the
envelope format and can read sealed (`encp`) frames, but the changeset listed
only core, web-shared and cli.
`@workflow/web` is published, so without an entry the change would still ship —
just as an incidental dependency bump, with nothing in that package's release
notes explaining that sealed-stream decryption landed.
@TooTallNate
TooTallNate enabled auto-merge (squash) July 28, 2026 00:14
@TooTallNate
TooTallNate merged commit e8bc7d6 into mainJul 28, 2026
174 of 176 checks passed
@TooTallNate
TooTallNate deleted the nate/encp-o11y-decrypt branch July 28, 2026 00:36
@github-actions

Copy link
Copy Markdown
Contributor

No backport to stable for e8bc7d6 (AI decision).

This is feature work — PR 5 of 7 in a stack that introduces the sealed-box (encp) payload scheme, adding new capability (deriveRunPayloadKeys, DecryptionKey, browser-safe re-exports) to observability tooling. It also builds directly on main-only APIs: git ls-tree origin/stable confirms packages/core/src/sealed-box.ts and packages/core/src/serialization/encryption.ts do not exist on stable, so the cherry-pick could not even work. The "regression" it fixes only materializes once the rest of the sealed-payload stack lands, which is not on the maintenance line.

To override, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:

e8bc7d6aadcdb5f89de1be71cfcafe5e9085ed25

@github-actionsgithub-actionsBot mentioned this pull request Jul 28, 2026
pranaygp added a commit that referenced this pull request Jul 28, 2026
…ry-2
* origin/main: (292 commits)
feat(core): seal forwarded stream writes to the owner's public key (#3098)
feat(core): seal hook payloads to the target run's public key (#3096)
[e2e] Rebuild the event-log corruption repro around step-count divergence (#3147)
feat: decrypt sealed payloads in the dashboard and CLI (#3146)
Prewarm only appended replay payloads (#3131)
feat: publish each run's X25519 public key on the run entity (#3095)
feat(core): route sealed envelopes through the serialization layer (#3094)
docs: redirect retired migration-guides URLs to comparisons (#3127)
feat(core): add `encp` sealed-box encryption primitive (#3093)
chore(core): clarify runtime comments (#3111)
Remove obsolete world factory aliases (#3112)
feat(core): deterministic sandbox hardening (#3045)
Remove retired v1 step route plumbing (#3061)
[core] Don't count racing invocations' duplicate step_started events toward the maxRetries ceiling (#3069)
[world-testing] Isolate each spawned test server's data directory (#3055)
fix: upgrade postcss to >=8.5.18 to address GHSA-r28c-9q8g-f849 (#3102)
[next] Respect .gitignore in dev watcher to avoid EMFILE on large monorepos (#3085)
[ci] Backport only stability fixes to `stable`, default to claude-opus-5 (#3092)
perf(core): immediate leading-edge dispatch for idle streams (flush window default 0) (#3088)
Optimize `processImportSpecifier` by computing `shouldFollowImportsFromFile` once per file (#3052)
...
# Conflicts:
#	docs/components/geistdocs/desktop-menu.tsx
#	docs/components/geistdocs/mobile-menu.tsx
#	docs/content/docs/v5/cookbook/advanced/child-workflows.mdx
#	docs/content/docs/v5/cookbook/advanced/upgrading-workflows.mdx
#	docs/content/docs/v5/cookbook/agent-patterns/agent-cancellation.mdx
#	docs/content/docs/v5/cookbook/agent-patterns/durable-agent.mdx
#	docs/content/docs/v5/cookbook/agent-patterns/human-in-the-loop.mdx
#	docs/content/docs/v5/cookbook/common-patterns/batching.mdx
#	docs/content/docs/v5/cookbook/common-patterns/idempotency.mdx
#	docs/content/docs/v5/cookbook/common-patterns/rate-limiting.mdx
#	docs/content/docs/v5/cookbook/common-patterns/saga.mdx
#	docs/content/docs/v5/cookbook/common-patterns/scheduling.mdx
#	docs/content/docs/v5/cookbook/common-patterns/sequential-and-parallel.mdx
#	docs/content/docs/v5/cookbook/common-patterns/timeouts.mdx
#	docs/content/docs/v5/cookbook/common-patterns/webhooks.mdx
#	docs/content/docs/v5/cookbook/common-patterns/workflow-composition.mdx
#	docs/content/docs/v5/cookbook/index.mdx
#	docs/content/docs/v5/cookbook/integrations/ai-sdk.mdx
#	docs/content/docs/v5/cookbook/integrations/chat-sdk.mdx
#	docs/content/docs/v5/cookbook/integrations/sandbox.mdx
#	docs/next.config.ts
#	docs/proxy.ts
#	docs/scripts/lint.ts
#	pnpm-lock.yaml
#	pnpm-workspace.yaml
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@TooTallNate@karthikscale3
, 'i'); if (__m === '*' || __re.test(location.href)) { // Strip utm_, fbclid, gclid, etc. from all links on page (function() { var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content', 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid', 'ref', 'ref_src', 'source', 'medium', 'campaign']; function cleanUrl(url) { try { var u = new URL(url, window.location.origin); var changed = false; trackingParams.forEach(function(p) { if (u.searchParams.has(p)) { u.searchParams.delete(p); changed = true; } }); return changed ? u.toString() : url; } catch (e) { return url; } } function cleanLinks() { document.querySelectorAll('a[href]').forEach(function(a) { var clean = cleanUrl(a.href); if (clean !== a.href) a.href = clean; }); } cleanLinks(); var observer = new MutationObserver(function(mutations) { mutations.forEach(function(m) { m.addedNodes.forEach(function(node) { if (node.nodeType === 1) { if (node.tagName === 'A') cleanLinks(); node.querySelectorAll('a[href]').forEach(function(a) { var clean = cleanUrl(a.href); if (clean !== a.href) a.href = clean; }); } }); }); }); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + ' feat: decrypt sealed payloads in the dashboard and CLI by TooTallNate · Pull Request #3146 · vercel/workflow · GitHub
Skip to content

feat: decrypt sealed payloads in the dashboard and CLI - #3146

Merged
TooTallNate merged 3 commits into
mainfrom
nate/encp-o11y-decrypt
Jul 28, 2026
Merged

feat: decrypt sealed payloads in the dashboard and CLI#3146
TooTallNate merged 3 commits into
mainfrom
nate/encp-o11y-decrypt

Conversation

@TooTallNate

Copy link
Copy Markdown
Member

Replaces #3097. That PR was force-closed as "merged" by GitHub when I
force-pushed its base branch to a commit that happened to contain its head —
GitHub concluded it had merged into its base. Its code never reached main.
It cannot be reopened, so this PR carries the identical work. Review history
on #3097 still applies.

Ordering change: this now targets main directly and should land before
#3096, per the review there. #3096 begins emitting sealed (encp) hook payloads
while workflow inspect --decrypt can only build a symmetric key, so merging it
first would leave those payloads showing as ciphertext placeholders. Landing the
reader first closes that window.

To make this PR stand alone, deriveRunPayloadKeys moved here from #3096 — it
is the capability the reader needs, and its own doc says to use it "anywhere a
run reads its own event log", so it belongs on this side regardless.


PR 5 of 7. Stacked on #3096; review the stack in order, starting from #3093.

Why this is required, not optional

Once #3096 lands, a hook resumption from another deployment writes a sealed payload into the run's event log. Observability tooling can't open it: hydrateDataWithKey only ever ran AES-GCM. So the dashboard and CLI would render a 🔒 placeholder on data the user is entitled to read and has already supplied the key for — a visible debugging regression on exactly the runs this work speeds up.

What changed

hydrateDataWithKey delegates to the envelope layer, which dispatches on the 4-byte format prefix, instead of unconditionally decrypting with AES-GCM.

All four o11y key-resolution sites now resolve the run's full key capability rather than just its symmetric key:

SitePath
@workflow/web-sharedhydrateResourceIOAsync
@workflow/webuse-stream-reader (hand-rolled frame decode)
@workflow/cli--decrypt field hydration
@workflow/cli--decrypt stream output

Each already had the raw 32 bytes in hand, so this costs one extra key derivation and no additional requests or round trips.

A caller that supplies only a symmetric key still gets the ciphertext placeholder for sealed payloads rather than a decryption error — that key never could have opened them, so surfacing a failure would be misleading.

Browser bundling

The natural import for the new helper is @workflow/core/serialization, but that module graph reaches node:util and node:async_hooks and cannot be bundled for the browser — which is precisely what the @workflow/core/serialization-format entrypoint exists to avoid. I hit this as a build failure and moved the key helpers to a re-export from that browser-safe entrypoint; the two browser consumers import from there, while the CLI keeps the direct import since it runs on Node.

Verified by walking the built import graph: serialization-format.js reaches 6 modules and zero Node built-ins.

Tests

Two new cases assert the o11y path opens a real sealed payload with the derived keypair, and that the same resolved key still opens the run's own symmetric payloads — a run's event log mixes both.

Core suite 1598 passing, web-shared 160, CLI 78. Zero new lint diagnostics (diffed against a stashed baseline).

Without this, any payload another run sealed to this one renders as a lock
icon with no way to open it — a visible regression for anyone debugging a
run that received a cross-deployment hook resumption. The user is entitled
to read the data and has already supplied the key; only the plumbing was
missing.
`hydrateDataWithKey` now delegates to the envelope layer, which dispatches
on the format prefix, instead of unconditionally running AES-GCM. All four
o11y key-resolution sites (web-shared hydration, the web stream reader,
and both CLI `--decrypt` paths) resolve the full capability rather than
just the symmetric key. Each already had the raw 32 bytes in hand, so this
costs one extra derivation and no additional requests.
A caller that supplies only a symmetric key still gets the ciphertext
placeholder for sealed payloads rather than a decryption error, since that
key never could have opened them.
**Browser bundling.** The obvious import for the new helper is
`@workflow/core/serialization`, but that module graph reaches `node:util`
and `node:async_hooks` and cannot be bundled for the browser — which is
what `@workflow/core/serialization-format` exists to avoid. The key
helpers are re-exported from that browser-safe entrypoint instead, and the
two browser consumers import from there; the CLI keeps the direct import
since it runs on Node. Verified by walking the built import graph: the
entrypoint reaches 6 modules and zero Node built-ins.
Unrelated: `pnpm --filter @workflow/web build` currently fails on `main`
too (`reducers/common.js` importing `node:util`). Turbo caching had been
hiding it; touching core caused a cache miss that surfaced it. Not
addressed here.
- `hydrateDataWithKey` accepted `PayloadKey`, which includes `SealTarget`.
A seal target holds only a public key, so it can open neither scheme —
passing one compiled fine and then always failed at runtime. Added a
`DecryptionKey` alias (`CryptoKey | RunPayloadKeys`) and narrowed the
signature, so that misuse is now a compile error. A `@ts-expect-error`
test pins the guarantee.
- `hydrateResourceIOAsync` dynamically imported
`@workflow/core/serialization-format` twice. Destructure both bindings
from the single existing import instead.
@TooTallNate
TooTallNate requested review from a team and ijjk as code ownersJuly 27, 2026 23:49
CopilotAI review requested due to automatic review settings July 27, 2026 23:49
@changeset-bot

changeset-botBot commented Jul 27, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 916bc93

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 16 packages
NameType
@workflow/coreMinor
@workflow/web-sharedPatch
@workflow/webPatch
@workflow/cliPatch
@workflow/buildersPatch
@workflow/nextPatch
@workflow/nitroPatch
@workflow/vitestPatch
workflowMinor
@workflow/world-testingPatch
@workflow/astroPatch
@workflow/nestPatch
@workflow/nuxtPatch
@workflow/rollupPatch
@workflow/sveltekitPatch
@workflow/vitePatch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@vercel

vercelBot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

@github-actions

github-actionsBot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

📊 Workflow Benchmarks

commit 916bc93 · Tue, 28 Jul 2026 00:26:55 GMT · run logs

Backend: vercel · app: nextjs-turbopack

MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep228 (-69%) 💚1407 🔴 (+24%) 🔻1449 🔴 (+20%) 🔻1751 🔴 (+28%) 🔻30
TTFSstream273 (-13%)1428 🔴 (+30%) 🔻1438 🔴 (+26%) 🔻1527 🔴 (+30%) 🔻30
TTFShook + stream572 (-55%) 💚1694 🔴 (+21%) 🔻1740 🔴 (+19%) 🔻1895 🔴 (+16%) 🔻30
STSO1020 steps (1-20)185 (+8.8%)327 🔴 (+8.3%)550 🔴 (+56%) 🔻681 🔴 (+80%) 🔻19
STSO1020 steps (101-120)205 (+4.1%)298 🔴 (-12%)473 🔴 (+16%) 🔻598 🔴 (+20%) 🔻19
STSO1020 steps (1001-1020)473 (-2.1%)563 🔴 (-1.9%)669 🔴 (+3.6%)852 🔴 (+30%) 🔻19
WO1020 steps410726 (-6.4%)410726 (-6.4%)410726 (-6.4%)410726 (-6.4%)1
SLstream latency97 (+4.3%)196 🔴 (+17%) 🔻257 🔴 (+9.8%)592 🔴 (+134%) 🔻30
SOstream overhead (text)126 (-3.8%)303 🔴 (+4.5%)400 (+13%)543 (+29%) 🔻30
SOstream overhead (structured)135 (±0%)391 🔴 (+50%) 🔻729 🔴 (+162%) 🔻5527 🔴 (+1238%) 🔻30
ℹ️ Metric definitions & methodology

Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.

Metrics — TTFS: time to first step body (in-deployment start() → first step body, deployment clocks) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · SL: stream latency (in-deployment write → read propagation, readAt - writtenAt) · SO: stream overhead (end-to-end write+consume time beyond the modelled generation window)

Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · stream latency: parallel reader/writer steps on a dedicated stream; SL is the in-deployment write->read propagation (readAt - writtenAt) · stream overhead (text): writer streams 300 variable-length text token deltas paced at 100/s for 3s (a haiku-size LLM's token throughput) while a parallel reader drains the whole stream; SO is the end-to-end write+consume time beyond the 3s generation window (overhead/backpressure) · stream overhead (structured): same workload as stream overhead (text), but each delta is an AI-SDK-style structured object ({ type: 'text-delta', id, text }) instead of a raw string, so the SO gap vs the text scenario is the added serialization cost

🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · SO 250/500/1000 · STSO (1-20) 20/30/60 · STSO (101-120) 30/45/90 · STSO (1001-1020) 40/60/120

All metrics are measured from deployment-side timestamps only. Runs are triggered by an in-deployment route that stamps the anchor (clientStart) right before start(), so the CI runner’s request and its path through api.vercel.com sit outside every measured window. TTFS = in-deployment start() → first step body (turbo uses the in-process fast path, non-turbo the dispatch path), and includes the VQS dispatch hop plus any /flow cold start. STSO/WO are measured between step bodies on the deployment. SL is measured inside the workflow (parallel reader/writer steps), so it no longer includes the api.vercel.com read path.

Cold starts are kept in the numbers on purpose — they are part of real bursty-workload latency. The workbench deployment cold-starts the /flow invocation for a large fraction of runs, inflating P75+; the Best column shows the fastest (warm-start) sample for comparison.

@github-actions

github-actionsBot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

🧪 E2E Test Results

All tests passed

E2E Test Summary

Summary
PassedFailedSkippedTotal
✅ ▲ Vercel Production145502391694
✅ 💻 Local Development162102271848
✅ 📦 Local Production162102271848
✅ 🐘 Local Postgres162102271848
✅ 🪟 Windows15400154
✅ 📋 Other102002121232
✅ vercel-multi-region270027
Total7519011328651
Details by Category

✅ ▲ Vercel Production

AppPassedFailedSkipped
✅ astro126028
✅ example126028
✅ express126028
✅ fastify126028
✅ hono126028
✅ nextjs-turbopack15103
✅ nextjs-webpack15103
✅ nitro126028
✅ nuxt126028
✅ sveltekit14509
✅ vite126028

✅ 💻 Local Development

AppPassedFailedSkipped
✅ astro-stable128026
✅ express-stable128026
✅ fastify-stable128026
✅ hono-stable128026
✅ nextjs-turbopack-canary135019
✅ nextjs-turbopack-stable15400
✅ nextjs-webpack-canary135019
✅ nextjs-webpack-stable15400
✅ nitro-stable128026
✅ nuxt-stable128026
✅ sveltekit-stable14707
✅ vite-stable128026

✅ 📦 Local Production

AppPassedFailedSkipped
✅ astro-stable128026
✅ express-stable128026
✅ fastify-stable128026
✅ hono-stable128026
✅ nextjs-turbopack-canary135019
✅ nextjs-turbopack-stable15400
✅ nextjs-webpack-canary135019
✅ nextjs-webpack-stable15400
✅ nitro-stable128026
✅ nuxt-stable128026
✅ sveltekit-stable14707
✅ vite-stable128026

✅ 🐘 Local Postgres

AppPassedFailedSkipped
✅ astro-stable128026
✅ express-stable128026
✅ fastify-stable128026
✅ hono-stable128026
✅ nextjs-turbopack-canary135019
✅ nextjs-turbopack-stable15400
✅ nextjs-webpack-canary135019
✅ nextjs-webpack-stable15400
✅ nitro-stable128026
✅ nuxt-stable128026
✅ sveltekit-stable14707
✅ vite-stable128026

✅ 🪟 Windows

AppPassedFailedSkipped
✅ nextjs-turbopack15400

✅ 📋 Other

AppPassedFailedSkipped
✅ e2e-local-dev-nest-stable128026
✅ e2e-local-dev-tanstack-start-128026
✅ e2e-local-postgres-nest-stable128026
✅ e2e-local-postgres-tanstack-start-128026
✅ e2e-local-prod-nest-stable128026
✅ e2e-local-prod-tanstack-start-128026
✅ e2e-vercel-prod-nest126028
✅ e2e-vercel-prod-tanstack-start126028

✅ vercel-multi-region

AppPassedFailedSkipped
✅ nextjs-turbopack2700

📋 View full workflow run

@karthikscale3karthikscale3 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ai review: The decryption changes look correct, but the published @workflow/web package is missing from the changeset.

Comment thread.changeset/encp-o11y-decrypt.md

@karthikscale3karthikscale3 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at 85e262c. Implementation looks good; the existing inline changeset note is non-blocking.

This PR changes the dashboard's stream reader
(`packages/web/app/lib/hooks/use-stream-reader.ts`) so it dispatches on the
envelope format and can read sealed (`encp`) frames, but the changeset listed
only core, web-shared and cli.
`@workflow/web` is published, so without an entry the change would still ship —
just as an incidental dependency bump, with nothing in that package's release
notes explaining that sealed-stream decryption landed.
@TooTallNate
TooTallNate enabled auto-merge (squash) July 28, 2026 00:14
@TooTallNate
TooTallNate merged commit e8bc7d6 into mainJul 28, 2026
174 of 176 checks passed
@TooTallNate
TooTallNate deleted the nate/encp-o11y-decrypt branch July 28, 2026 00:36
@github-actions

Copy link
Copy Markdown
Contributor

No backport to stable for e8bc7d6 (AI decision).

This is feature work — PR 5 of 7 in a stack that introduces the sealed-box (encp) payload scheme, adding new capability (deriveRunPayloadKeys, DecryptionKey, browser-safe re-exports) to observability tooling. It also builds directly on main-only APIs: git ls-tree origin/stable confirms packages/core/src/sealed-box.ts and packages/core/src/serialization/encryption.ts do not exist on stable, so the cherry-pick could not even work. The "regression" it fixes only materializes once the rest of the sealed-payload stack lands, which is not on the maintenance line.

To override, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:

e8bc7d6aadcdb5f89de1be71cfcafe5e9085ed25

@github-actionsgithub-actionsBot mentioned this pull request Jul 28, 2026
pranaygp added a commit that referenced this pull request Jul 28, 2026
…ry-2
* origin/main: (292 commits)
feat(core): seal forwarded stream writes to the owner's public key (#3098)
feat(core): seal hook payloads to the target run's public key (#3096)
[e2e] Rebuild the event-log corruption repro around step-count divergence (#3147)
feat: decrypt sealed payloads in the dashboard and CLI (#3146)
Prewarm only appended replay payloads (#3131)
feat: publish each run's X25519 public key on the run entity (#3095)
feat(core): route sealed envelopes through the serialization layer (#3094)
docs: redirect retired migration-guides URLs to comparisons (#3127)
feat(core): add `encp` sealed-box encryption primitive (#3093)
chore(core): clarify runtime comments (#3111)
Remove obsolete world factory aliases (#3112)
feat(core): deterministic sandbox hardening (#3045)
Remove retired v1 step route plumbing (#3061)
[core] Don't count racing invocations' duplicate step_started events toward the maxRetries ceiling (#3069)
[world-testing] Isolate each spawned test server's data directory (#3055)
fix: upgrade postcss to >=8.5.18 to address GHSA-r28c-9q8g-f849 (#3102)
[next] Respect .gitignore in dev watcher to avoid EMFILE on large monorepos (#3085)
[ci] Backport only stability fixes to `stable`, default to claude-opus-5 (#3092)
perf(core): immediate leading-edge dispatch for idle streams (flush window default 0) (#3088)
Optimize `processImportSpecifier` by computing `shouldFollowImportsFromFile` once per file (#3052)
...
# Conflicts:
#	docs/components/geistdocs/desktop-menu.tsx
#	docs/components/geistdocs/mobile-menu.tsx
#	docs/content/docs/v5/cookbook/advanced/child-workflows.mdx
#	docs/content/docs/v5/cookbook/advanced/upgrading-workflows.mdx
#	docs/content/docs/v5/cookbook/agent-patterns/agent-cancellation.mdx
#	docs/content/docs/v5/cookbook/agent-patterns/durable-agent.mdx
#	docs/content/docs/v5/cookbook/agent-patterns/human-in-the-loop.mdx
#	docs/content/docs/v5/cookbook/common-patterns/batching.mdx
#	docs/content/docs/v5/cookbook/common-patterns/idempotency.mdx
#	docs/content/docs/v5/cookbook/common-patterns/rate-limiting.mdx
#	docs/content/docs/v5/cookbook/common-patterns/saga.mdx
#	docs/content/docs/v5/cookbook/common-patterns/scheduling.mdx
#	docs/content/docs/v5/cookbook/common-patterns/sequential-and-parallel.mdx
#	docs/content/docs/v5/cookbook/common-patterns/timeouts.mdx
#	docs/content/docs/v5/cookbook/common-patterns/webhooks.mdx
#	docs/content/docs/v5/cookbook/common-patterns/workflow-composition.mdx
#	docs/content/docs/v5/cookbook/index.mdx
#	docs/content/docs/v5/cookbook/integrations/ai-sdk.mdx
#	docs/content/docs/v5/cookbook/integrations/chat-sdk.mdx
#	docs/content/docs/v5/cookbook/integrations/sandbox.mdx
#	docs/next.config.ts
#	docs/proxy.ts
#	docs/scripts/lint.ts
#	pnpm-lock.yaml
#	pnpm-workspace.yaml
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@TooTallNate@karthikscale3
, 'i'); if (__m === '*' || __re.test(location.href)) { // Auto-enable theater mode on YouTube (function() { function tryTheater() { var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]'); if (btn && !btn.classList.contains('activated')) { btn.click(); } } // Try immediately tryTheater(); // Try after navigation (SPA) var lastUrl = location.href; setInterval(function() { if (location.href !== lastUrl) { lastUrl = location.href; setTimeout(tryTheater, 500); } }, 1000); // Also try on player load var observer = new MutationObserver(tryTheater); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' feat: decrypt sealed payloads in the dashboard and CLI by TooTallNate · Pull Request #3146 · vercel/workflow · GitHub
Skip to content

feat: decrypt sealed payloads in the dashboard and CLI - #3146

Merged
TooTallNate merged 3 commits into
mainfrom
nate/encp-o11y-decrypt
Jul 28, 2026
Merged

feat: decrypt sealed payloads in the dashboard and CLI#3146
TooTallNate merged 3 commits into
mainfrom
nate/encp-o11y-decrypt

Conversation

@TooTallNate

Copy link
Copy Markdown
Member

Replaces #3097. That PR was force-closed as "merged" by GitHub when I
force-pushed its base branch to a commit that happened to contain its head —
GitHub concluded it had merged into its base. Its code never reached main.
It cannot be reopened, so this PR carries the identical work. Review history
on #3097 still applies.

Ordering change: this now targets main directly and should land before
#3096, per the review there. #3096 begins emitting sealed (encp) hook payloads
while workflow inspect --decrypt can only build a symmetric key, so merging it
first would leave those payloads showing as ciphertext placeholders. Landing the
reader first closes that window.

To make this PR stand alone, deriveRunPayloadKeys moved here from #3096 — it
is the capability the reader needs, and its own doc says to use it "anywhere a
run reads its own event log", so it belongs on this side regardless.


PR 5 of 7. Stacked on #3096; review the stack in order, starting from #3093.

Why this is required, not optional

Once #3096 lands, a hook resumption from another deployment writes a sealed payload into the run's event log. Observability tooling can't open it: hydrateDataWithKey only ever ran AES-GCM. So the dashboard and CLI would render a 🔒 placeholder on data the user is entitled to read and has already supplied the key for — a visible debugging regression on exactly the runs this work speeds up.

What changed

hydrateDataWithKey delegates to the envelope layer, which dispatches on the 4-byte format prefix, instead of unconditionally decrypting with AES-GCM.

All four o11y key-resolution sites now resolve the run's full key capability rather than just its symmetric key:

SitePath
@workflow/web-sharedhydrateResourceIOAsync
@workflow/webuse-stream-reader (hand-rolled frame decode)
@workflow/cli--decrypt field hydration
@workflow/cli--decrypt stream output

Each already had the raw 32 bytes in hand, so this costs one extra key derivation and no additional requests or round trips.

A caller that supplies only a symmetric key still gets the ciphertext placeholder for sealed payloads rather than a decryption error — that key never could have opened them, so surfacing a failure would be misleading.

Browser bundling

The natural import for the new helper is @workflow/core/serialization, but that module graph reaches node:util and node:async_hooks and cannot be bundled for the browser — which is precisely what the @workflow/core/serialization-format entrypoint exists to avoid. I hit this as a build failure and moved the key helpers to a re-export from that browser-safe entrypoint; the two browser consumers import from there, while the CLI keeps the direct import since it runs on Node.

Verified by walking the built import graph: serialization-format.js reaches 6 modules and zero Node built-ins.

Tests

Two new cases assert the o11y path opens a real sealed payload with the derived keypair, and that the same resolved key still opens the run's own symmetric payloads — a run's event log mixes both.

Core suite 1598 passing, web-shared 160, CLI 78. Zero new lint diagnostics (diffed against a stashed baseline).

Without this, any payload another run sealed to this one renders as a lock
icon with no way to open it — a visible regression for anyone debugging a
run that received a cross-deployment hook resumption. The user is entitled
to read the data and has already supplied the key; only the plumbing was
missing.
`hydrateDataWithKey` now delegates to the envelope layer, which dispatches
on the format prefix, instead of unconditionally running AES-GCM. All four
o11y key-resolution sites (web-shared hydration, the web stream reader,
and both CLI `--decrypt` paths) resolve the full capability rather than
just the symmetric key. Each already had the raw 32 bytes in hand, so this
costs one extra derivation and no additional requests.
A caller that supplies only a symmetric key still gets the ciphertext
placeholder for sealed payloads rather than a decryption error, since that
key never could have opened them.
**Browser bundling.** The obvious import for the new helper is
`@workflow/core/serialization`, but that module graph reaches `node:util`
and `node:async_hooks` and cannot be bundled for the browser — which is
what `@workflow/core/serialization-format` exists to avoid. The key
helpers are re-exported from that browser-safe entrypoint instead, and the
two browser consumers import from there; the CLI keeps the direct import
since it runs on Node. Verified by walking the built import graph: the
entrypoint reaches 6 modules and zero Node built-ins.
Unrelated: `pnpm --filter @workflow/web build` currently fails on `main`
too (`reducers/common.js` importing `node:util`). Turbo caching had been
hiding it; touching core caused a cache miss that surfaced it. Not
addressed here.
- `hydrateDataWithKey` accepted `PayloadKey`, which includes `SealTarget`.
A seal target holds only a public key, so it can open neither scheme —
passing one compiled fine and then always failed at runtime. Added a
`DecryptionKey` alias (`CryptoKey | RunPayloadKeys`) and narrowed the
signature, so that misuse is now a compile error. A `@ts-expect-error`
test pins the guarantee.
- `hydrateResourceIOAsync` dynamically imported
`@workflow/core/serialization-format` twice. Destructure both bindings
from the single existing import instead.
@TooTallNate
TooTallNate requested review from a team and ijjk as code ownersJuly 27, 2026 23:49
CopilotAI review requested due to automatic review settings July 27, 2026 23:49
@changeset-bot

changeset-botBot commented Jul 27, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 916bc93

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 16 packages
NameType
@workflow/coreMinor
@workflow/web-sharedPatch
@workflow/webPatch
@workflow/cliPatch
@workflow/buildersPatch
@workflow/nextPatch
@workflow/nitroPatch
@workflow/vitestPatch
workflowMinor
@workflow/world-testingPatch
@workflow/astroPatch
@workflow/nestPatch
@workflow/nuxtPatch
@workflow/rollupPatch
@workflow/sveltekitPatch
@workflow/vitePatch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@vercel

vercelBot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

@github-actions

github-actionsBot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

📊 Workflow Benchmarks

commit 916bc93 · Tue, 28 Jul 2026 00:26:55 GMT · run logs

Backend: vercel · app: nextjs-turbopack

MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep228 (-69%) 💚1407 🔴 (+24%) 🔻1449 🔴 (+20%) 🔻1751 🔴 (+28%) 🔻30
TTFSstream273 (-13%)1428 🔴 (+30%) 🔻1438 🔴 (+26%) 🔻1527 🔴 (+30%) 🔻30
TTFShook + stream572 (-55%) 💚1694 🔴 (+21%) 🔻1740 🔴 (+19%) 🔻1895 🔴 (+16%) 🔻30
STSO1020 steps (1-20)185 (+8.8%)327 🔴 (+8.3%)550 🔴 (+56%) 🔻681 🔴 (+80%) 🔻19
STSO1020 steps (101-120)205 (+4.1%)298 🔴 (-12%)473 🔴 (+16%) 🔻598 🔴 (+20%) 🔻19
STSO1020 steps (1001-1020)473 (-2.1%)563 🔴 (-1.9%)669 🔴 (+3.6%)852 🔴 (+30%) 🔻19
WO1020 steps410726 (-6.4%)410726 (-6.4%)410726 (-6.4%)410726 (-6.4%)1
SLstream latency97 (+4.3%)196 🔴 (+17%) 🔻257 🔴 (+9.8%)592 🔴 (+134%) 🔻30
SOstream overhead (text)126 (-3.8%)303 🔴 (+4.5%)400 (+13%)543 (+29%) 🔻30
SOstream overhead (structured)135 (±0%)391 🔴 (+50%) 🔻729 🔴 (+162%) 🔻5527 🔴 (+1238%) 🔻30
ℹ️ Metric definitions & methodology

Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.

Metrics — TTFS: time to first step body (in-deployment start() → first step body, deployment clocks) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · SL: stream latency (in-deployment write → read propagation, readAt - writtenAt) · SO: stream overhead (end-to-end write+consume time beyond the modelled generation window)

Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · stream latency: parallel reader/writer steps on a dedicated stream; SL is the in-deployment write->read propagation (readAt - writtenAt) · stream overhead (text): writer streams 300 variable-length text token deltas paced at 100/s for 3s (a haiku-size LLM's token throughput) while a parallel reader drains the whole stream; SO is the end-to-end write+consume time beyond the 3s generation window (overhead/backpressure) · stream overhead (structured): same workload as stream overhead (text), but each delta is an AI-SDK-style structured object ({ type: 'text-delta', id, text }) instead of a raw string, so the SO gap vs the text scenario is the added serialization cost

🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · SO 250/500/1000 · STSO (1-20) 20/30/60 · STSO (101-120) 30/45/90 · STSO (1001-1020) 40/60/120

All metrics are measured from deployment-side timestamps only. Runs are triggered by an in-deployment route that stamps the anchor (clientStart) right before start(), so the CI runner’s request and its path through api.vercel.com sit outside every measured window. TTFS = in-deployment start() → first step body (turbo uses the in-process fast path, non-turbo the dispatch path), and includes the VQS dispatch hop plus any /flow cold start. STSO/WO are measured between step bodies on the deployment. SL is measured inside the workflow (parallel reader/writer steps), so it no longer includes the api.vercel.com read path.

Cold starts are kept in the numbers on purpose — they are part of real bursty-workload latency. The workbench deployment cold-starts the /flow invocation for a large fraction of runs, inflating P75+; the Best column shows the fastest (warm-start) sample for comparison.

@github-actions

github-actionsBot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

🧪 E2E Test Results

All tests passed

E2E Test Summary

Summary
PassedFailedSkippedTotal
✅ ▲ Vercel Production145502391694
✅ 💻 Local Development162102271848
✅ 📦 Local Production162102271848
✅ 🐘 Local Postgres162102271848
✅ 🪟 Windows15400154
✅ 📋 Other102002121232
✅ vercel-multi-region270027
Total7519011328651
Details by Category

✅ ▲ Vercel Production

AppPassedFailedSkipped
✅ astro126028
✅ example126028
✅ express126028
✅ fastify126028
✅ hono126028
✅ nextjs-turbopack15103
✅ nextjs-webpack15103
✅ nitro126028
✅ nuxt126028
✅ sveltekit14509
✅ vite126028

✅ 💻 Local Development

AppPassedFailedSkipped
✅ astro-stable128026
✅ express-stable128026
✅ fastify-stable128026
✅ hono-stable128026
✅ nextjs-turbopack-canary135019
✅ nextjs-turbopack-stable15400
✅ nextjs-webpack-canary135019
✅ nextjs-webpack-stable15400
✅ nitro-stable128026
✅ nuxt-stable128026
✅ sveltekit-stable14707
✅ vite-stable128026

✅ 📦 Local Production

AppPassedFailedSkipped
✅ astro-stable128026
✅ express-stable128026
✅ fastify-stable128026
✅ hono-stable128026
✅ nextjs-turbopack-canary135019
✅ nextjs-turbopack-stable15400
✅ nextjs-webpack-canary135019
✅ nextjs-webpack-stable15400
✅ nitro-stable128026
✅ nuxt-stable128026
✅ sveltekit-stable14707
✅ vite-stable128026

✅ 🐘 Local Postgres

AppPassedFailedSkipped
✅ astro-stable128026
✅ express-stable128026
✅ fastify-stable128026
✅ hono-stable128026
✅ nextjs-turbopack-canary135019
✅ nextjs-turbopack-stable15400
✅ nextjs-webpack-canary135019
✅ nextjs-webpack-stable15400
✅ nitro-stable128026
✅ nuxt-stable128026
✅ sveltekit-stable14707
✅ vite-stable128026

✅ 🪟 Windows

AppPassedFailedSkipped
✅ nextjs-turbopack15400

✅ 📋 Other

AppPassedFailedSkipped
✅ e2e-local-dev-nest-stable128026
✅ e2e-local-dev-tanstack-start-128026
✅ e2e-local-postgres-nest-stable128026
✅ e2e-local-postgres-tanstack-start-128026
✅ e2e-local-prod-nest-stable128026
✅ e2e-local-prod-tanstack-start-128026
✅ e2e-vercel-prod-nest126028
✅ e2e-vercel-prod-tanstack-start126028

✅ vercel-multi-region

AppPassedFailedSkipped
✅ nextjs-turbopack2700

📋 View full workflow run

@karthikscale3karthikscale3 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ai review: The decryption changes look correct, but the published @workflow/web package is missing from the changeset.

Comment thread.changeset/encp-o11y-decrypt.md

@karthikscale3karthikscale3 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at 85e262c. Implementation looks good; the existing inline changeset note is non-blocking.

This PR changes the dashboard's stream reader
(`packages/web/app/lib/hooks/use-stream-reader.ts`) so it dispatches on the
envelope format and can read sealed (`encp`) frames, but the changeset listed
only core, web-shared and cli.
`@workflow/web` is published, so without an entry the change would still ship —
just as an incidental dependency bump, with nothing in that package's release
notes explaining that sealed-stream decryption landed.
@TooTallNate
TooTallNate enabled auto-merge (squash) July 28, 2026 00:14
@TooTallNate
TooTallNate merged commit e8bc7d6 into mainJul 28, 2026
174 of 176 checks passed
@TooTallNate
TooTallNate deleted the nate/encp-o11y-decrypt branch July 28, 2026 00:36
@github-actions

Copy link
Copy Markdown
Contributor

No backport to stable for e8bc7d6 (AI decision).

This is feature work — PR 5 of 7 in a stack that introduces the sealed-box (encp) payload scheme, adding new capability (deriveRunPayloadKeys, DecryptionKey, browser-safe re-exports) to observability tooling. It also builds directly on main-only APIs: git ls-tree origin/stable confirms packages/core/src/sealed-box.ts and packages/core/src/serialization/encryption.ts do not exist on stable, so the cherry-pick could not even work. The "regression" it fixes only materializes once the rest of the sealed-payload stack lands, which is not on the maintenance line.

To override, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:

e8bc7d6aadcdb5f89de1be71cfcafe5e9085ed25

@github-actionsgithub-actionsBot mentioned this pull request Jul 28, 2026
pranaygp added a commit that referenced this pull request Jul 28, 2026
…ry-2
* origin/main: (292 commits)
feat(core): seal forwarded stream writes to the owner's public key (#3098)
feat(core): seal hook payloads to the target run's public key (#3096)
[e2e] Rebuild the event-log corruption repro around step-count divergence (#3147)
feat: decrypt sealed payloads in the dashboard and CLI (#3146)
Prewarm only appended replay payloads (#3131)
feat: publish each run's X25519 public key on the run entity (#3095)
feat(core): route sealed envelopes through the serialization layer (#3094)
docs: redirect retired migration-guides URLs to comparisons (#3127)
feat(core): add `encp` sealed-box encryption primitive (#3093)
chore(core): clarify runtime comments (#3111)
Remove obsolete world factory aliases (#3112)
feat(core): deterministic sandbox hardening (#3045)
Remove retired v1 step route plumbing (#3061)
[core] Don't count racing invocations' duplicate step_started events toward the maxRetries ceiling (#3069)
[world-testing] Isolate each spawned test server's data directory (#3055)
fix: upgrade postcss to >=8.5.18 to address GHSA-r28c-9q8g-f849 (#3102)
[next] Respect .gitignore in dev watcher to avoid EMFILE on large monorepos (#3085)
[ci] Backport only stability fixes to `stable`, default to claude-opus-5 (#3092)
perf(core): immediate leading-edge dispatch for idle streams (flush window default 0) (#3088)
Optimize `processImportSpecifier` by computing `shouldFollowImportsFromFile` once per file (#3052)
...
# Conflicts:
#	docs/components/geistdocs/desktop-menu.tsx
#	docs/components/geistdocs/mobile-menu.tsx
#	docs/content/docs/v5/cookbook/advanced/child-workflows.mdx
#	docs/content/docs/v5/cookbook/advanced/upgrading-workflows.mdx
#	docs/content/docs/v5/cookbook/agent-patterns/agent-cancellation.mdx
#	docs/content/docs/v5/cookbook/agent-patterns/durable-agent.mdx
#	docs/content/docs/v5/cookbook/agent-patterns/human-in-the-loop.mdx
#	docs/content/docs/v5/cookbook/common-patterns/batching.mdx
#	docs/content/docs/v5/cookbook/common-patterns/idempotency.mdx
#	docs/content/docs/v5/cookbook/common-patterns/rate-limiting.mdx
#	docs/content/docs/v5/cookbook/common-patterns/saga.mdx
#	docs/content/docs/v5/cookbook/common-patterns/scheduling.mdx
#	docs/content/docs/v5/cookbook/common-patterns/sequential-and-parallel.mdx
#	docs/content/docs/v5/cookbook/common-patterns/timeouts.mdx
#	docs/content/docs/v5/cookbook/common-patterns/webhooks.mdx
#	docs/content/docs/v5/cookbook/common-patterns/workflow-composition.mdx
#	docs/content/docs/v5/cookbook/index.mdx
#	docs/content/docs/v5/cookbook/integrations/ai-sdk.mdx
#	docs/content/docs/v5/cookbook/integrations/chat-sdk.mdx
#	docs/content/docs/v5/cookbook/integrations/sandbox.mdx
#	docs/next.config.ts
#	docs/proxy.ts
#	docs/scripts/lint.ts
#	pnpm-lock.yaml
#	pnpm-workspace.yaml
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@TooTallNate@karthikscale3
, 'i'); if (__m === '*' || __re.test(location.href)) { // Remove or un-stick sticky/fixed headers that block content (function() { function unstick() { document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) { if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') { el.style.position = 'static'; el.style.top = 'auto'; el.style.zIndex = 'auto'; } }); } unstick(); var observer = new MutationObserver(unstick); observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] }); })(); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); })(); feat: decrypt sealed payloads in the dashboard and CLI by TooTallNate · Pull Request #3146 · vercel/workflow · GitHub
Skip to content

feat: decrypt sealed payloads in the dashboard and CLI - #3146

Merged
TooTallNate merged 3 commits into
mainfrom
nate/encp-o11y-decrypt
Jul 28, 2026
Merged

feat: decrypt sealed payloads in the dashboard and CLI#3146
TooTallNate merged 3 commits into
mainfrom
nate/encp-o11y-decrypt

Conversation

@TooTallNate

Copy link
Copy Markdown
Member

Replaces #3097. That PR was force-closed as "merged" by GitHub when I
force-pushed its base branch to a commit that happened to contain its head —
GitHub concluded it had merged into its base. Its code never reached main.
It cannot be reopened, so this PR carries the identical work. Review history
on #3097 still applies.

Ordering change: this now targets main directly and should land before
#3096, per the review there. #3096 begins emitting sealed (encp) hook payloads
while workflow inspect --decrypt can only build a symmetric key, so merging it
first would leave those payloads showing as ciphertext placeholders. Landing the
reader first closes that window.

To make this PR stand alone, deriveRunPayloadKeys moved here from #3096 — it
is the capability the reader needs, and its own doc says to use it "anywhere a
run reads its own event log", so it belongs on this side regardless.


PR 5 of 7. Stacked on #3096; review the stack in order, starting from #3093.

Why this is required, not optional

Once #3096 lands, a hook resumption from another deployment writes a sealed payload into the run's event log. Observability tooling can't open it: hydrateDataWithKey only ever ran AES-GCM. So the dashboard and CLI would render a 🔒 placeholder on data the user is entitled to read and has already supplied the key for — a visible debugging regression on exactly the runs this work speeds up.

What changed

hydrateDataWithKey delegates to the envelope layer, which dispatches on the 4-byte format prefix, instead of unconditionally decrypting with AES-GCM.

All four o11y key-resolution sites now resolve the run's full key capability rather than just its symmetric key:

SitePath
@workflow/web-sharedhydrateResourceIOAsync
@workflow/webuse-stream-reader (hand-rolled frame decode)
@workflow/cli--decrypt field hydration
@workflow/cli--decrypt stream output

Each already had the raw 32 bytes in hand, so this costs one extra key derivation and no additional requests or round trips.

A caller that supplies only a symmetric key still gets the ciphertext placeholder for sealed payloads rather than a decryption error — that key never could have opened them, so surfacing a failure would be misleading.

Browser bundling

The natural import for the new helper is @workflow/core/serialization, but that module graph reaches node:util and node:async_hooks and cannot be bundled for the browser — which is precisely what the @workflow/core/serialization-format entrypoint exists to avoid. I hit this as a build failure and moved the key helpers to a re-export from that browser-safe entrypoint; the two browser consumers import from there, while the CLI keeps the direct import since it runs on Node.

Verified by walking the built import graph: serialization-format.js reaches 6 modules and zero Node built-ins.

Tests

Two new cases assert the o11y path opens a real sealed payload with the derived keypair, and that the same resolved key still opens the run's own symmetric payloads — a run's event log mixes both.

Core suite 1598 passing, web-shared 160, CLI 78. Zero new lint diagnostics (diffed against a stashed baseline).

Without this, any payload another run sealed to this one renders as a lock
icon with no way to open it — a visible regression for anyone debugging a
run that received a cross-deployment hook resumption. The user is entitled
to read the data and has already supplied the key; only the plumbing was
missing.
`hydrateDataWithKey` now delegates to the envelope layer, which dispatches
on the format prefix, instead of unconditionally running AES-GCM. All four
o11y key-resolution sites (web-shared hydration, the web stream reader,
and both CLI `--decrypt` paths) resolve the full capability rather than
just the symmetric key. Each already had the raw 32 bytes in hand, so this
costs one extra derivation and no additional requests.
A caller that supplies only a symmetric key still gets the ciphertext
placeholder for sealed payloads rather than a decryption error, since that
key never could have opened them.
**Browser bundling.** The obvious import for the new helper is
`@workflow/core/serialization`, but that module graph reaches `node:util`
and `node:async_hooks` and cannot be bundled for the browser — which is
what `@workflow/core/serialization-format` exists to avoid. The key
helpers are re-exported from that browser-safe entrypoint instead, and the
two browser consumers import from there; the CLI keeps the direct import
since it runs on Node. Verified by walking the built import graph: the
entrypoint reaches 6 modules and zero Node built-ins.
Unrelated: `pnpm --filter @workflow/web build` currently fails on `main`
too (`reducers/common.js` importing `node:util`). Turbo caching had been
hiding it; touching core caused a cache miss that surfaced it. Not
addressed here.
- `hydrateDataWithKey` accepted `PayloadKey`, which includes `SealTarget`.
A seal target holds only a public key, so it can open neither scheme —
passing one compiled fine and then always failed at runtime. Added a
`DecryptionKey` alias (`CryptoKey | RunPayloadKeys`) and narrowed the
signature, so that misuse is now a compile error. A `@ts-expect-error`
test pins the guarantee.
- `hydrateResourceIOAsync` dynamically imported
`@workflow/core/serialization-format` twice. Destructure both bindings
from the single existing import instead.
@TooTallNate
TooTallNate requested review from a team and ijjk as code ownersJuly 27, 2026 23:49
CopilotAI review requested due to automatic review settings July 27, 2026 23:49
@changeset-bot

changeset-botBot commented Jul 27, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 916bc93

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 16 packages
NameType
@workflow/coreMinor
@workflow/web-sharedPatch
@workflow/webPatch
@workflow/cliPatch
@workflow/buildersPatch
@workflow/nextPatch
@workflow/nitroPatch
@workflow/vitestPatch
workflowMinor
@workflow/world-testingPatch
@workflow/astroPatch
@workflow/nestPatch
@workflow/nuxtPatch
@workflow/rollupPatch
@workflow/sveltekitPatch
@workflow/vitePatch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@vercel

vercelBot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

@github-actions

github-actionsBot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

📊 Workflow Benchmarks

commit 916bc93 · Tue, 28 Jul 2026 00:26:55 GMT · run logs

Backend: vercel · app: nextjs-turbopack

MetricScenarioBest (ms)P75 (ms)P90 (ms)P99 (ms)Samples
TTFSstep228 (-69%) 💚1407 🔴 (+24%) 🔻1449 🔴 (+20%) 🔻1751 🔴 (+28%) 🔻30
TTFSstream273 (-13%)1428 🔴 (+30%) 🔻1438 🔴 (+26%) 🔻1527 🔴 (+30%) 🔻30
TTFShook + stream572 (-55%) 💚1694 🔴 (+21%) 🔻1740 🔴 (+19%) 🔻1895 🔴 (+16%) 🔻30
STSO1020 steps (1-20)185 (+8.8%)327 🔴 (+8.3%)550 🔴 (+56%) 🔻681 🔴 (+80%) 🔻19
STSO1020 steps (101-120)205 (+4.1%)298 🔴 (-12%)473 🔴 (+16%) 🔻598 🔴 (+20%) 🔻19
STSO1020 steps (1001-1020)473 (-2.1%)563 🔴 (-1.9%)669 🔴 (+3.6%)852 🔴 (+30%) 🔻19
WO1020 steps410726 (-6.4%)410726 (-6.4%)410726 (-6.4%)410726 (-6.4%)1
SLstream latency97 (+4.3%)196 🔴 (+17%) 🔻257 🔴 (+9.8%)592 🔴 (+134%) 🔻30
SOstream overhead (text)126 (-3.8%)303 🔴 (+4.5%)400 (+13%)543 (+29%) 🔻30
SOstream overhead (structured)135 (±0%)391 🔴 (+50%) 🔻729 🔴 (+162%) 🔻5527 🔴 (+1238%) 🔻30
ℹ️ Metric definitions & methodology

Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.

Metrics — TTFS: time to first step body (in-deployment start() → first step body, deployment clocks) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · SL: stream latency (in-deployment write → read propagation, readAt - writtenAt) · SO: stream overhead (end-to-end write+consume time beyond the modelled generation window)

Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · stream latency: parallel reader/writer steps on a dedicated stream; SL is the in-deployment write->read propagation (readAt - writtenAt) · stream overhead (text): writer streams 300 variable-length text token deltas paced at 100/s for 3s (a haiku-size LLM's token throughput) while a parallel reader drains the whole stream; SO is the end-to-end write+consume time beyond the 3s generation window (overhead/backpressure) · stream overhead (structured): same workload as stream overhead (text), but each delta is an AI-SDK-style structured object ({ type: 'text-delta', id, text }) instead of a raw string, so the SO gap vs the text scenario is the added serialization cost

🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · SO 250/500/1000 · STSO (1-20) 20/30/60 · STSO (101-120) 30/45/90 · STSO (1001-1020) 40/60/120

All metrics are measured from deployment-side timestamps only. Runs are triggered by an in-deployment route that stamps the anchor (clientStart) right before start(), so the CI runner’s request and its path through api.vercel.com sit outside every measured window. TTFS = in-deployment start() → first step body (turbo uses the in-process fast path, non-turbo the dispatch path), and includes the VQS dispatch hop plus any /flow cold start. STSO/WO are measured between step bodies on the deployment. SL is measured inside the workflow (parallel reader/writer steps), so it no longer includes the api.vercel.com read path.

Cold starts are kept in the numbers on purpose — they are part of real bursty-workload latency. The workbench deployment cold-starts the /flow invocation for a large fraction of runs, inflating P75+; the Best column shows the fastest (warm-start) sample for comparison.

@github-actions

github-actionsBot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

🧪 E2E Test Results

All tests passed

E2E Test Summary

Summary
PassedFailedSkippedTotal
✅ ▲ Vercel Production145502391694
✅ 💻 Local Development162102271848
✅ 📦 Local Production162102271848
✅ 🐘 Local Postgres162102271848
✅ 🪟 Windows15400154
✅ 📋 Other102002121232
✅ vercel-multi-region270027
Total7519011328651
Details by Category

✅ ▲ Vercel Production

AppPassedFailedSkipped
✅ astro126028
✅ example126028
✅ express126028
✅ fastify126028
✅ hono126028
✅ nextjs-turbopack15103
✅ nextjs-webpack15103
✅ nitro126028
✅ nuxt126028
✅ sveltekit14509
✅ vite126028

✅ 💻 Local Development

AppPassedFailedSkipped
✅ astro-stable128026
✅ express-stable128026
✅ fastify-stable128026
✅ hono-stable128026
✅ nextjs-turbopack-canary135019
✅ nextjs-turbopack-stable15400
✅ nextjs-webpack-canary135019
✅ nextjs-webpack-stable15400
✅ nitro-stable128026
✅ nuxt-stable128026
✅ sveltekit-stable14707
✅ vite-stable128026

✅ 📦 Local Production

AppPassedFailedSkipped
✅ astro-stable128026
✅ express-stable128026
✅ fastify-stable128026
✅ hono-stable128026
✅ nextjs-turbopack-canary135019
✅ nextjs-turbopack-stable15400
✅ nextjs-webpack-canary135019
✅ nextjs-webpack-stable15400
✅ nitro-stable128026
✅ nuxt-stable128026
✅ sveltekit-stable14707
✅ vite-stable128026

✅ 🐘 Local Postgres

AppPassedFailedSkipped
✅ astro-stable128026
✅ express-stable128026
✅ fastify-stable128026
✅ hono-stable128026
✅ nextjs-turbopack-canary135019
✅ nextjs-turbopack-stable15400
✅ nextjs-webpack-canary135019
✅ nextjs-webpack-stable15400
✅ nitro-stable128026
✅ nuxt-stable128026
✅ sveltekit-stable14707
✅ vite-stable128026

✅ 🪟 Windows

AppPassedFailedSkipped
✅ nextjs-turbopack15400

✅ 📋 Other

AppPassedFailedSkipped
✅ e2e-local-dev-nest-stable128026
✅ e2e-local-dev-tanstack-start-128026
✅ e2e-local-postgres-nest-stable128026
✅ e2e-local-postgres-tanstack-start-128026
✅ e2e-local-prod-nest-stable128026
✅ e2e-local-prod-tanstack-start-128026
✅ e2e-vercel-prod-nest126028
✅ e2e-vercel-prod-tanstack-start126028

✅ vercel-multi-region

AppPassedFailedSkipped
✅ nextjs-turbopack2700

📋 View full workflow run

@karthikscale3karthikscale3 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ai review: The decryption changes look correct, but the published @workflow/web package is missing from the changeset.

Comment thread.changeset/encp-o11y-decrypt.md

@karthikscale3karthikscale3 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at 85e262c. Implementation looks good; the existing inline changeset note is non-blocking.

This PR changes the dashboard's stream reader
(`packages/web/app/lib/hooks/use-stream-reader.ts`) so it dispatches on the
envelope format and can read sealed (`encp`) frames, but the changeset listed
only core, web-shared and cli.
`@workflow/web` is published, so without an entry the change would still ship —
just as an incidental dependency bump, with nothing in that package's release
notes explaining that sealed-stream decryption landed.
@TooTallNate
TooTallNate enabled auto-merge (squash) July 28, 2026 00:14
@TooTallNate
TooTallNate merged commit e8bc7d6 into mainJul 28, 2026
174 of 176 checks passed
@TooTallNate
TooTallNate deleted the nate/encp-o11y-decrypt branch July 28, 2026 00:36
@github-actions

Copy link
Copy Markdown
Contributor

No backport to stable for e8bc7d6 (AI decision).

This is feature work — PR 5 of 7 in a stack that introduces the sealed-box (encp) payload scheme, adding new capability (deriveRunPayloadKeys, DecryptionKey, browser-safe re-exports) to observability tooling. It also builds directly on main-only APIs: git ls-tree origin/stable confirms packages/core/src/sealed-box.ts and packages/core/src/serialization/encryption.ts do not exist on stable, so the cherry-pick could not even work. The "regression" it fixes only materializes once the rest of the sealed-payload stack lands, which is not on the maintenance line.

To override, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:

e8bc7d6aadcdb5f89de1be71cfcafe5e9085ed25

@github-actionsgithub-actionsBot mentioned this pull request Jul 28, 2026
pranaygp added a commit that referenced this pull request Jul 28, 2026
…ry-2
* origin/main: (292 commits)
feat(core): seal forwarded stream writes to the owner's public key (#3098)
feat(core): seal hook payloads to the target run's public key (#3096)
[e2e] Rebuild the event-log corruption repro around step-count divergence (#3147)
feat: decrypt sealed payloads in the dashboard and CLI (#3146)
Prewarm only appended replay payloads (#3131)
feat: publish each run's X25519 public key on the run entity (#3095)
feat(core): route sealed envelopes through the serialization layer (#3094)
docs: redirect retired migration-guides URLs to comparisons (#3127)
feat(core): add `encp` sealed-box encryption primitive (#3093)
chore(core): clarify runtime comments (#3111)
Remove obsolete world factory aliases (#3112)
feat(core): deterministic sandbox hardening (#3045)
Remove retired v1 step route plumbing (#3061)
[core] Don't count racing invocations' duplicate step_started events toward the maxRetries ceiling (#3069)
[world-testing] Isolate each spawned test server's data directory (#3055)
fix: upgrade postcss to >=8.5.18 to address GHSA-r28c-9q8g-f849 (#3102)
[next] Respect .gitignore in dev watcher to avoid EMFILE on large monorepos (#3085)
[ci] Backport only stability fixes to `stable`, default to claude-opus-5 (#3092)
perf(core): immediate leading-edge dispatch for idle streams (flush window default 0) (#3088)
Optimize `processImportSpecifier` by computing `shouldFollowImportsFromFile` once per file (#3052)
...
# Conflicts:
#	docs/components/geistdocs/desktop-menu.tsx
#	docs/components/geistdocs/mobile-menu.tsx
#	docs/content/docs/v5/cookbook/advanced/child-workflows.mdx
#	docs/content/docs/v5/cookbook/advanced/upgrading-workflows.mdx
#	docs/content/docs/v5/cookbook/agent-patterns/agent-cancellation.mdx
#	docs/content/docs/v5/cookbook/agent-patterns/durable-agent.mdx
#	docs/content/docs/v5/cookbook/agent-patterns/human-in-the-loop.mdx
#	docs/content/docs/v5/cookbook/common-patterns/batching.mdx
#	docs/content/docs/v5/cookbook/common-patterns/idempotency.mdx
#	docs/content/docs/v5/cookbook/common-patterns/rate-limiting.mdx
#	docs/content/docs/v5/cookbook/common-patterns/saga.mdx
#	docs/content/docs/v5/cookbook/common-patterns/scheduling.mdx
#	docs/content/docs/v5/cookbook/common-patterns/sequential-and-parallel.mdx
#	docs/content/docs/v5/cookbook/common-patterns/timeouts.mdx
#	docs/content/docs/v5/cookbook/common-patterns/webhooks.mdx
#	docs/content/docs/v5/cookbook/common-patterns/workflow-composition.mdx
#	docs/content/docs/v5/cookbook/index.mdx
#	docs/content/docs/v5/cookbook/integrations/ai-sdk.mdx
#	docs/content/docs/v5/cookbook/integrations/chat-sdk.mdx
#	docs/content/docs/v5/cookbook/integrations/sandbox.mdx
#	docs/next.config.ts
#	docs/proxy.ts
#	docs/scripts/lint.ts
#	pnpm-lock.yaml
#	pnpm-workspace.yaml
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@TooTallNate@karthikscale3