Uh oh!
There was an error while loading. Please reload this page.
feat: publish each run's X25519 public key on the run entity - #3095
Conversation
🦋 Changeset detectedLatest commit: e7c2128 The changes in this PR will be included in the next version bump. This PR includes changesets to release 20 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
🧪 E2E Test Results✅ All tests passed E2E Test SummarySummary
Details by Category✅ ▲ Vercel Production
✅ 💻 Local Development
✅ 📦 Local Production
✅ 🐘 Local Postgres
✅ 🪟 Windows
✅ 📋 Other
✅ vercel-multi-region
|
There was a problem hiding this comment.
Pull request overview
This PR publishes each workflow run’s X25519 public key (encryptionPublicKey) onto the run entity at creation time, so cross-run writers can seal (encp) payloads to a run without an additional run-key fetch, while keeping private key material derivable-only (not stored).
Changes:
- Derive the run’s X25519 public key from existing per-run key material during
start()and include it onrun_createdand queuedrunInput(resilient start). - Plumb the optional
encryptionPublicKeyfield through world schemas, Vercel v4 meta splitting, and local/postgres run materialization (plus a postgres migration). - Add base64 encode/decode helpers and tests; hide the field in the web attribute panel.
Reviewed changes
Copilot reviewed 17 out of 17 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| packages/world/src/runs.ts | Adds optional encryptionPublicKey to the run schema with rationale/semantics. |
| packages/world/src/events.ts | Adds encryptionPublicKey to run_created and resilient-start run_started event schemas. |
| packages/world-vercel/src/events.ts | Routes encryptionPublicKey into the v4 meta block so it isn’t dropped on the wire. |
| packages/world-postgres/src/storage.ts | Persists encryptionPublicKey from run_created into the run row. |
| packages/world-postgres/src/drizzle/schema.ts | Adds encryptionPublicKey column mapping to the runs table schema. |
| packages/world-postgres/src/drizzle/migrations/meta/_journal.json | Registers migration 0016_add_encryption_public_key. |
| packages/world-postgres/src/drizzle/migrations/0016_add_encryption_public_key.sql | Adds encryption_public_key column to workflow_runs. |
| packages/world-local/src/storage/events-storage.ts | Persists encryptionPublicKey onto the local run entity for run_created. |
| packages/web-shared/src/components/sidebar/attribute-panel.tsx | Hides encryptionPublicKey in the attribute panel UI. |
| packages/core/src/sealed-box.ts | Adds VM/browser-safe base64 helpers used for publishing the run public key. |
| packages/core/src/sealed-box.test.ts | Tests base64 helpers against Buffer plus malformed input behavior. |
| packages/core/src/runtime/start.ts | Derives/stamps encryptionPublicKey on run_created and queued runInput. |
| packages/core/src/runtime/start.test.ts | Adds tests asserting stamping, usability (seal/open), isolation, and disabled-encryption behavior. |
| .changeset/encp-run-public-key-worlds.md | Changeset for world packages to persist/transport encryptionPublicKey. |
| .changeset/encp-run-public-key-web-shared.md | Changeset for hiding encryptionPublicKey in the web attribute panel. |
| .changeset/encp-run-public-key-postgres.md | Changeset for the postgres column/migration. |
| .changeset/encp-run-public-key-core.md | Changeset for core stamping at start(). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Additional Suggestions:
- world-local lifecycle handlers (run_started/completed/failed/cancelled) rebuild the run row without
encryptionPublicKey, so the key stamped atrun_createdis dropped as soon as the run starts.
- The world-postgres resilient-start path (run_started recreating a run) drops
encryptionPublicKey, so runs recovered this way permanently lack their public key and cannot receive sealed writes.
- The resilient-start path (run_started recreating a run when run_created was missed) drops
encryptionPublicKey, so a run recovered this way permanently loses its ability to receive sealed writes on the world-local (and world-postgres) backends.
Heads up — this PR's public key never actually reached the server. Found it during manual verification; the fix is now the third commit on this branch ( What was happeningThe key was computed, put in the
The reason nothing caught it: const{ payload, meta }=splitEventDataForV4(data);awaitcreateWorkflowRunEventV4({runId: id,/* … */ payload, ...meta},config);TypeScript doesn't apply excess-property checks to spreads, so an unforwarded field is not a type error. The exhaustiveness guard added in this PR covers Why this was hard to seeEvery symptom pointed at the server side rather than at us:
I spent a while looking server-side. The problem was here: the SDK was dropping the field before it ever went out. FixAdd Verified in productionThen a genuinely cross-deployment The round trip is the important part: an Where the fix landedOriginally I committed this on the stack tip, which would have meant merging #3095–#3098 with encryption silently downgraded to |
881ce18 to
34056d5Compare
karthikscale3
left a comment
There was a problem hiding this comment.
ai review: Reviewed the run public-key publication path end to end. The key is derived only when encryption material exists and is propagated through run_created, queued runInput, resilient-start reconstruction, lifecycle rewrites, Postgres persistence/migration, and the Vercel v4 split/meta wire path. The three existing inline findings and the lifecycle/resilient-start bot findings are fixed on this head. The stale generated-file warning is not present and module-boundary checks pass. The matrix-wide Vercel production failures remain unrelated to this change.
📊 Workflow Benchmarkscommit Backend:
📜 Previous results (1)57d0c1bMon, 27 Jul 2026 20:08:48 GMT · run logs
ℹ️ Metric definitions & methodologyBest/P75/P90/P99 deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body (in-deployment start() → first step body, deployment clocks) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · SL: stream latency (in-deployment write → read propagation, readAt - writtenAt) · SO: stream overhead (end-to-end write+consume time beyond the modelled generation window) Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · stream latency: parallel reader/writer steps on a dedicated stream; SL is the in-deployment write->read propagation (readAt - writtenAt) · stream overhead (text): writer streams 300 variable-length text token deltas paced at 100/s for 3s (a haiku-size LLM's token throughput) while a parallel reader drains the whole stream; SO is the end-to-end write+consume time beyond the 3s generation window (overhead/backpressure) · stream overhead (structured): same workload as stream overhead (text), but each delta is an AI-SDK-style structured object ({ type: 'text-delta', id, text }) instead of a raw string, so the SO gap vs the text scenario is the added serialization cost 🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · SO 250/500/1000 · STSO (1-20) 20/30/60 · STSO (101-120) 30/45/90 · STSO (1001-1020) 40/60/120 All metrics are measured from deployment-side timestamps only. Runs are triggered by an in-deployment route that stamps the anchor ( Cold starts are kept in the numbers on purpose — they are part of real bursty-workload latency. The workbench deployment cold-starts the |
A cross-run writer needs the recipient run's public key to seal a payload to it. Derive that key at `start()` and stamp it on the run, so a hook resumption or a forwarded-stream writer can find it on a run fetch it was already making instead of spending ~350ms on `run-key`. The key is derived from the per-run key material `getEncryptionKeyForRun()` already returns, so nothing about key acquisition changes. It is not secret: the matching private scalar is never stored anywhere, only re-derived on demand from the deployment's own env seed. Storing it beside run metadata therefore does not weaken the run's confidentiality. **Presence is the writer-side gate for sealed envelopes.** A run only carries a public key if the runtime that created it could also open one — which holds by construction, since derivation and `encp` dispatch both live in `@workflow/core`, so any core that can stamp can also open. Runs are pinned to their creating deployment, so the capability this attests to is still accurate at resume time. Writers seal iff the field is set and otherwise fall back to the symmetric path, which makes version skew degrade gracefully instead of wedging a run. The field rides on `run_created`, and is mirrored onto the queued `runInput` so the resilient-start path (server recreates the run from the queue message when the `run_created` write failed) doesn't silently produce a run that can't receive sealed writes. world-vercel's compile-time wire-contract guard caught the new field before it could be silently dropped on the v4 path, exactly as designed — routed into the frame meta block as plaintext metadata. Also adds browser- and VM-safe base64 helpers to `sealed-box.ts`, since neither `Buffer` nor `btoa` can be assumed in every context that module runs in. `base64ToBytes` returns undefined on malformed input rather than throwing, so a corrupt stored key degrades to "no usable public key" and falls back to the symmetric path instead of crashing a resumption. Both are cross-validated against `Buffer` in tests.
Two real bugs found in review, both in the local worlds. Neither surfaces as an error — a run just silently stops accepting sealed cross-run writes and falls back to the slow symmetric path forever. **Resilient start dropped the key.** When a `run_started` arrives for a run that was never created, world-local and world-postgres rebuild the run from the queued message. Neither copied `encryptionPublicKey` onto the run row or the synthetic `run_created` event they write. That is precisely the scenario this field exists to survive. (The equivalent server-side path was already handled.) **world-local also wiped the key on every lifecycle transition.** Its run_started / run_completed / run_failed / run_cancelled handlers rewrite the whole run document field-by-field, so any field not explicitly listed is dropped — meaning the key was lost on the *first* `run_started`, not just on the resilient path. All four rebuild sites now carry it. world-postgres is safe here by construction because it issues column-scoped SQL UPDATEs rather than rewriting the row. **base64 decoding is now strict.** The decoder accepted shapes that cannot describe a whole number of bytes (`length % 4 === 1`) and ignored anything after a mid-string `=`, returning a short array instead of `undefined`. That is worse than throwing: a corrupt stored key looked *present*, so callers sealed to garbage rather than taking the symmetric fallback. Now rejects out-of-alphabet characters, bad lengths, misplaced padding, and non-zero trailing bits — with a round-trip test over every length 0–48 to make sure the strictness does not overshoot.
`splitEventDataForV4` lifted the run's public key into the frame meta and `events.ts` spread that meta into `CreateEventV4Input`, but `buildPostFrameMeta` — which copies meta onto the wire field by field — never forwarded `encryptionPublicKey`, and the field was missing from `CreateEventV4Input` entirely. Because the meta is applied with a spread, TypeScript's excess-property check doesn't fire, so the key was computed, put in the meta, and then silently dropped before the request was sent. The server therefore never received the key, never stored it on the run entity, and every cross-run writer fell back to the symmetric envelope. Every symptom pointed away from the SDK: a deliberately oversized key was accepted rather than rejected (the field never arrived), the key was absent from the run row, and `resumeHook()` always chose `encr`. Add the field to `CreateEventV4Input`, forward it in `buildPostFrameMeta`, and cover it for both `run_created` and resilient-start `run_started`. Also add a generic guard asserting that every field the splitter puts in the meta reaches the wire, so the next omission in this hand-maintained mapping fails a test instead of silently degrading encryption.
No backport to This is feature work: it adds a new To override, re-run the Backport to stable workflow manually via |
PR 3 of 7. Stacked on #3094; review the stack in order, starting from #3093.
encpsealed-box encryption primitive #3093 — theencpsealed-box primitiveresumeHook()seals instead of fetching the key ← the payoffstart()gets the key from the probe it already awaitsWhat
A cross-run writer needs the recipient run's public key to seal to it. Derive that key at
start()and stamp it on the run, so a hook resumption or forwarded-stream writer finds it on a run fetch it was already making instead of spending ~350ms onrun-key.Derived from the per-run key material
getEncryptionKeyForRun()already returns, so nothing about key acquisition changes.Why storing it is safe
It is not secret. The matching private scalar is never stored anywhere — only re-derived on demand from the deployment's own env seed. So this does not violate the requirement that no key material live beside run metadata (that requirement is about secrets), and the server remains zero-knowledge: it stores and serves 32 public bytes it cannot use to decrypt anything.
Anyone with tenant-scoped read access to a run gains encryption capability for it, and nothing more.
Presence is the writer-side gate
A run only carries a public key if the runtime that created it could also open a sealed payload. That holds by construction: derivation and
encpdispatch both live in@workflow/core, so any core that can stamp can also open. Runs are pinned to their creating deployment, so the capability this attests to is still accurate at resume time.This matters because
@workflow/coreand@workflow/world-vercelare versioned independently and do drift in practice. A gate that consulted only one of them would wedge runs under one drift direction or the other:encp, a writer would seal a payload the target cannot decode — run wedged. Locating derivation, stamping and dispatch all in@workflow/coremakes this combination impossible to construct.encpbut its world package never persisted the field, the run simply has no public key, and writers fall back to the symmetric path — degraded, not broken.Presence therefore attests to both halves at once, which a version comparison cannot.
Resilient start
The field rides on
run_createdand is mirrored onto the queuedrunInput, because the resilient-start path recreates the run from the queue message when therun_createdwrite failed. Without that mirror, a run recovered that way would silently lose the ability to receive sealed writes.Surface
@workflow/corestart(); browser/VM-safe base64 helpers@workflow/worldencryptionPublicKeyon the run schema +run_created/run_started@workflow/world-local@workflow/world-postgresencryption_public_keycolumn + migration0016@workflow/world-vercel@workflow/web-sharedNotes for reviewers
encryptionPublicKey. Nice.sealed-box.tsruns in the browser (o11y) and inside the workflow VM, where neitherBuffernorbtoacan be assumed. They're cross-validated againstBufferin tests across a range of lengths.base64ToBytesreturnsundefinedon malformed input rather than throwing, so a corrupt stored key degrades to "no usable public key" → symmetric fallback, instead of crashing a resumption.Postgres migration is additive and nullable. Full core suite (1585) passes; all world suites pass; workspace typecheck clean.