Uh oh!
There was an error while loading. Please reload this page.
[core][world][world-vercel] Add World.createRunId() and region-aware queue routing - #1981
Merged
Conversation
Encodes a tag bit, 5-bit version, and 6-bit Vercel region ID into a ULID-shaped string used for workflow run IDs. Tagged values remain valid 26-char Crockford-Base32 ULIDs so they still sort and round-trip through any system that accepts ULIDs.
Add exact-string expectations for encoded outputs at known inputs, covering the default region/version pair, numeric region IDs, version overrides, boundary values (all-zero, all-max), the dirty-input overwrite case, and the lexicographic-order checks. Also adds an explicit byte-array expectation for the canonical ULID-spec example string and an additional first-char-range coverage test for isTagged.
- isTaggedString now fully validates the input as a 26-char Crockford Base32 ULID (delegating to ulidToBytes) instead of only inspecting the first character. This fixes false positives on inputs like '4UUUU...' that have a valid tag-bit position but invalid chars later in the string. - isTagged() now accepts `unknown` to match its documented behavior of safely rejecting non-string inputs without requiring callers to cast. - Introduce `RegionKey` for the full set of keys including 'unknown', and narrow `RegionCode` to `Exclude<RegionKey, 'unknown'>` so the return type of `lookupRegion` and the `DecodedRunId.region` field accurately reflect that 'unknown' is never produced. Updates `encode` to reject 'unknown' as a region code string at runtime (callers wanting the unknown sentinel should pass numeric 0).
…e queue routing - @workflow/world: add optional createRunId(input?) to the World interface so worlds can mint run IDs with embedded metadata, and add an optional 'region' field to QueueOptions for per-message routing hints. - @workflow/core: start() now delegates run ID generation to world.createRunId() when defined (falling back to a monotonic ULID otherwise), and accepts a new 'runIdInput' option that is forwarded verbatim to createRunId. When runIdInput.region is a string, it is also threaded onto the queue options so the initial workflow message is dispatched to the matching region. - @workflow/world-vercel: implement createRunId() to mint region-tagged ULIDs, preferring an explicit runIdInput.region and falling back to the VERCEL_REGION env var. The queue now resolves its destination region from (in order): an explicit opts.region, the region embedded in the payload's tagged run ID, the VERCEL_REGION env var, and finally a hardcoded 'iad1' default. This replaces the previous unconditional 'iad1' region passed to the @vercel/queue client. Monotonicity within a process is preserved by tracking the last emitted run ID and bumping the bit immediately above the 11-bit metadata window when a same-ms collision would otherwise occur, then re-stamping the requested region/version on top so metadata remains stable.
Contributor
🦋 Changeset detectedLatest commit: 4f77bc1 The changes in this PR will be included in the next version bump. This PR includes changesets to release 20 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Contributor
🧪 E2E Test Results✅ All tests passed Summary
Details by Category✅ ▲ Vercel Production
✅ 💻 Local Development
✅ 📦 Local Production
✅ 🐘 Local Postgres
✅ 🪟 Windows
✅ 📋 Other
✅ vercel-multi-region
|
Contributor
There was a problem hiding this comment.
Pull request overview
This PR extends the workflow runtime/world contract to support (1) world-minted run IDs and (2) region-aware queue routing, and wires those capabilities through @workflow/core and the Vercel world implementation.
Changes:
- Add optional
World.createRunId(input?)andQueueOptions.regionin@workflow/world. - Update
@workflow/corestart()to delegate run ID generation toworld.createRunId(opts.runIdInput)when available, and to forwardrunIdInput.regionto queue options. - Implement region-tagged run IDs + region-resolving queue routing in
@workflow/world-vercel, including new tests and a newuliddependency.
Reviewed changes
Copilot reviewed 13 out of 14 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| pnpm-lock.yaml | Adds lockfile entry for ulid used by world-vercel. |
| packages/world/src/queue.ts | Extends QueueOptions with a region routing hint + docs. |
| packages/world/src/interfaces.ts | Extends World interface with optional createRunId. |
| packages/world-vercel/src/queue.ts | Adds region resolution order (opts → tagged runId → env → fallback) and updates CBOR encode/decode imports. |
| packages/world-vercel/src/queue.test.ts | Adds region-routing test coverage. |
| packages/world-vercel/src/index.ts | Wires createRunId into createVercelWorld and re-exports it. |
| packages/world-vercel/src/create-run-id.ts | Implements region-tagged ULID minting with per-process monotonicity. |
| packages/world-vercel/src/create-run-id.test.ts | Tests for env/input fallback and monotonicity behavior. |
| packages/world-vercel/package.json | Adds ulid dependency. |
| packages/core/src/runtime/start.ts | Uses world.createRunId when available; forwards runIdInput.region into queue options. |
| packages/core/src/runtime/start.test.ts | Adds tests validating createRunId usage and region threading. |
| .changeset/world-vercel-create-run-id.md | Release notes for new run ID minting + routing behavior in world-vercel. |
| .changeset/world-create-run-id.md | Release notes for new @workflow/world interface fields. |
| .changeset/core-run-id-input.md | Release notes for core start() runIdInput behavior. |
Files not reviewed (1)
- pnpm-lock.yaml: Language not supported
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
…unId Drop the dedicated 'runIdInput' field on StartOptions and forward the entire options bag to world.createRunId() instead. This keeps the public API surface smaller and lets each World pick the fields it recognises (e.g. world-vercel reads 'region'). The top-level 'region' option remains on StartOptionsBase and is also threaded onto the queue's per-call region opt when set.
…ride - queue: validate opts.region and VERCEL_REGION against the known region table before routing, ignoring unrecognised codes so a bad override can't clobber the payload-derived region (Copilot) - add isKnownRegionCode() runtime guard to run-id/regions - reset WORKFLOW_SERVER_URL_OVERRIDE to '' (must be empty on main) - fold the within-PR iad1-default changeset into the main world-vercel changeset and delete it (review) - start.test: declare specVersion on createRunId mock worlds now that the merged world-compatibility check requires it - cover the new region-validation fall-through paths in queue.test Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
VaguelySerious
commented
Jul 1, 2026
Member
(AI) Deleted |
…i-region preview BRANCH-ONLY — revert the override to '' before merge (lint enforces). Points this PR's e2e/benchmark runs at the wave-1 multi-region workflow-server preview (vercel/workflow-server#590: iad1+sfo1+fra1 serving, staging data backends) so region-tagged runs are validated against real multi-region serving end-to-end. Also makes the unit-test mock origins in events-v4.test.ts and trace-propagation.test.ts override-aware (same pattern the rest of the file and utils.test.ts already use), so the suite passes whether or not the override is set — these two files were the only spots hardcoding https://vercel-workflow.com.
Adds a dedicated e2e suite validating @workflow/world-vercel region
routing end to end, run as its own CI job (e2e-vercel-multi-region)
against the nextjs-turbopack workbench only — deliberately separate
from e2e.test.ts, which runs as a matrix across all worlds/frameworks
where Vercel-specific multi-region behavior doesn't apply.
- workbench/nextjs-turbopack/vercel.json: deploy to iad1+sfo1+fra1 so
region-routed flow messages have a function to land on in each region.
- workflows/99_e2e.ts: regionProbeWorkflow returns the VERCEL_REGION
observed by both the workflow and a step, so tests can assert the run
EXECUTED in the intended region (not just that it was tagged).
- packages/core/e2e/e2e-region.test.ts: per-region cases assert
1) start(..., { region }) mints a region-tagged run ID (decoded via
@workflow/world-vercel/run-id),
2) the workflow + step both observed VERCEL_REGION === region,
3) the server reports the run completed;
plus a concurrent all-regions case guarding against cross-region
misrouting under simultaneous multi-region traffic. Skips on local
deployments.
- .github/workflows/tests.yml: new e2e-vercel-multi-region job
mirroring e2e-vercel-prod's env/deployment-wait, running only the
new suite.The first multi-region CI run surfaced two issues: 1. sfo1/fra1-tagged runs executed in iad1. The suite started runs from the external test process, which uses the api.vercel.com token proxy — and the proxy's queues path forwards every send to the region-less VQS host (the world's proxy-mode resolveBaseUrl ignores the region argument, and the proxy's x-vercel-vqs-api-url escape hatch only allowlists vqs-server-*.vercel.sh preview hosts). Production traffic publishes IN-FUNCTION (direct regional queue routing), so the suite now triggers start() through a new workbench route (/api/e2e-region-start) and rehydrates the run with getRun() — testing the path production actually takes. Proxy-mode regional queue routing is a known gap to address separately in api-workflow. 2. TypeError on world.runs.get: getWorld() is async and was called without await.
Textually clean but semantically conflicting with #2789: main added the proxy-mode x-vercel-queue-region header in createQueue's shared clientOptions, referencing the static region const this branch removed in favor of per-send resolution. Resolution: the header now rides the per-send QueueClient construction, carrying resolveTargetRegion's result — proxied sends route to the region resolved per message (opts.region -> run-ID tag -> VERCEL_REGION -> iad1). world-vercel: 253/253 (includes #2789's proxy-header tests, which hold under per-send resolution).
…ion suite With regional VQS routing now working through the api.vercel.com proxy (vercel/api#79056 + #2789 + this branch's per-send region resolution), the suite covers both start configurations, asserting the same three properties for each (region-tagged run ID, execution in the intended region via VERCEL_REGION echoed in the return value, server-side completion): 1. EXPLICIT: start(..., { region }) called directly in the vitest runner — publishes through the token proxy, per-send region carried by x-vercel-queue-region. Restores the direct-start shape the suite had originally, plus the concurrent all-regions case. 2. IMPLICIT: dedicated per-region workbench routes (/api/e2e-region-implicit/{iad1,sfo1,fra1}), each pinned to a single region via a per-function 'regions' entry in the workbench vercel.json, calling start() with NO region option — createRunId derives the tag from the minting function's VERCEL_REGION. The test also asserts the route reported executing in its pinned region, so the implicit-tagging assertion can't pass vacuously. Replaces the interim /api/e2e-region-start route (explicit region via request body), which existed to work around the pre-#79056 proxy gap.
…ng is in production workflow-server#590 (iad1+sfo1+fra1 serving) merged and deployed to production and the e2e backend, so this branch's e2e/benchmark runs no longer need to target the wave-1 preview. Restores the empty override the No Test Overrides lint job enforces for merge. The override-aware unit-test origins (events-v4/trace-propagation) stay — they are correct under any override value.
saasjesus
commented
Jul 7, 2026
damn its happening |
Regression coverage for a backend bug that made cross-region stream reads report zero chunks on IN-PROGRESS streams (completed streams were unaffected), which forced the multi-region serving rollback. The new case exercises exactly that geometry: - crossRegionStreamWorkflow (99_e2e.ts) writes N chunks to the default output stream, then holds the stream OPEN for 45s before closing — the in-progress window is the point, since completed streams are the easy case. - The e2e starts it with region iad1, waits (same-region, via the api.vercel.com proxy) until all chunks are written, asserts the run is still 'running', then reads through a new sfo1-pinned workbench route (/api/e2e-stream-read/sfo1) that returns getTailIndex() plus its VERCEL_REGION. The reader's region served none of the stream's writes, so the reported chunk count must come from the backend's cross-region stream metadata. The test fails loudly if the route isn't actually executing in sfo1. Also bumps the explicit-region test timeout to 120s: the first case in the file absorbs every cold start at once (fresh workbench instances in up to three regions plus a cold backend preview) and was observed just over the 60s default. BRANCH-ONLY (revert before merge, lint enforces): WORKFLOW_SERVER_URL_OVERRIDE points at a multi-region backend preview that includes the fix, so this validates cross-region stream visibility end-to-end before multi-region serving is re-enabled.
Points the suite at an all-regions backend preview and widens coverage from the wave-1 trio to every provisioned region: - Explicit path: a single concurrent all-regions case starts one tagged run per region (one shared cold-start window instead of 19 sequential ones) and aggregates per-region failures so a single region's breakage reports alongside the full picture. The trio keeps its detailed per-region cases and the 9-way concurrent-isolation case. - Implicit path: workbench gains a region-pinned /api/e2e-region-implicit/<region> route per provisioned region (19 total, shared handler), the workbench itself now deploys to all of them, and the test.each covers the full set with per-case timeouts for regional cold starts. - Multi-region CI job timeout 20m -> 35m for the sequential implicit cases. BRANCH-ONLY (revert before merge, lint enforces): WORKFLOW_SERVER_URL_OVERRIDE now targets the all-regions backend preview instead of the previous (stale, since-merged) fix preview.
The first all-regions run surfaced a subtle execution-locality behavior: queue delivery is guaranteed to the tagged region's dataplane and the delivery callback egresses from that region, but the consumer invocation's execution region is chosen by where that callback enters Vercel's edge — and adjacent regions can geo-resolve to each other's functions. Observed live: kix1-tagged runs (callback egressing from Osaka) deterministically executing in hnd1/Tokyo on both the explicit and implicit paths, with tagging, data placement, and completion all still strictly kix1. expectRunInRegion now asserts execution lands in the tagged region OR one of its geographic neighbors (EXECUTION_ADJACENCY), while run-ID tagging and server-side completion remain strictly the requested region. Gross misrouting (e.g. kix1 -> iad1) still fails.
… production The all-regions workflow-server rollout is deployed and serving production traffic from every Vercel region, so this branch's e2e no longer needs to target a branch preview. Restores the empty override the No Test Overrides lint enforces for merge. With this the PR is complete: region-tagged run IDs, region-aware queue routing, and the multi-region e2e suite (explicit + implicit + all-regions + cross-region streams) all validate against the production-default backends.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
- start.ts: StartOptionsBase.region fallback is iad1, not the unknown
sentinel (createRunId always mints a concrete routable region)
- queue.ts: example used a nonexistent start({ runIdInput }) API; the
real option is start({ region })
- events.ts: decode() clears only the tag bit (top bit of the 48-bit
timestamp field) — it does not restore the original untagged ULID;
reword to say what actually matters for timestamp validationHooks are resolved by opaque token, which carries no region hint, so lookup and resume must work regardless of which region owns the run's data. Exercises the full follow-up-message path on sfo1- and fra1-tagged runs: create inside the workflow, resolve by token from the test process, resume twice sequentially, and assert payload order and completion. Regression coverage for the failure mode where the first message to a hook-driven app on a non-iad1 run worked but every follow-up failed with 'Hook not found'.
VaguelySerious
approved these changes
Jul 10, 2026
This was referenced Jul 13, 2026
Conflict in packages/core/src/runtime/start.test.ts: both sides appended a new top-level describe block at the same location — this branch's 'createRunId' tests and main's 'queue namespace' tests. Resolved by keeping both blocks verbatim (each is self-contained with its own beforeEach/mocks). Lockfile reconciled with pnpm install. Verified post-merge: core tsc clean, start/helpers tests 73 passed, world-vercel 263 passed, full build 27/27.
Contributor
Backport to This is usually an infrastructure problem (e.g. the configured AI model could not be found, an AI Gateway error, or an opencode crash) rather than a merge conflict. Check the job logs linked above for details. Once the underlying issue is fixed, re-run the Backport to stable workflow manually via |
This was referenced Jul 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes#1931 and helps with #2708 (and possibly more)
Wires the tagged-ULID encoding introduced in #1978 into the workflow run lifecycle:
@workflow/world— adds two extension points to theWorldinterface:createRunId?(input?: Record<string, unknown>): string— opt-in hook for worlds to mint custom run IDs (the returned value is the bare ULID; the core attaches thewrun_prefix).QueueOptions.region?: string— per-message routing hint that worlds can use to dispatch messages to a specific region.@workflow/core—start()now:regionoption (StartOptionsBase.region).world.createRunId(opts)when the world provides one (forwarding the full options bag so worlds can read whichever fields they recognise), falling back to a monotonic ULID otherwise.opts.regiononto the queue options so the initial workflow message is dispatched to the matching region.@workflow/world-vercel— implements both extension points:createRunId(options?)mints region-tagged ULIDs. Region resolution order:options.region→VERCEL_REGIONenv var →iad1default. A run ID is always tagged with a concrete, routable region; theunknown(0) sentinel is never minted.opts.region, the region embedded in the payload's tagged run ID, theVERCEL_REGIONenv var, and finally a hardcodediad1default — replacing the previous unconditional'iad1'passed to@vercel/queue. In proxy mode the resolved region is forwarded as a routing header.Monotonicity
encode()writes the 11 bits of region/version metadata into the top of the ULID's randomness section (all of byte 6 + the top 3 bits of byte 7), leaving the low 69 bits untouched. The underlyingulidmonotonic factory's same-millisecond bottom-bit increments therefore survive encoding, and consecutive IDs with the same region/version metadata are naturally monotonic — no correction needed in the common case.The one remaining hazard is the metadata changing between same-millisecond calls (a lower-numbered region sorts below a higher one at the same timestamp). For that,
createRunIdtracks the last emitted ID and, on collision, bumps the bit immediately above the metadata window (the lowest bit of the 48-bit timestamp) before re-stamping the requested region/version. The test suite covers 4096 same-ms mints plus same-ms region changes.E2E coverage
Adds
packages/core/e2e/e2e-region.test.ts(CI jobe2e-vercel-multi-region, deployed to all 19 routable regions):start({ region })for iad1 / sfo1 / fra1, verifying run-ID tags, data placement, and execution region.regionoption —VERCEL_REGIONdrives the tag).Execution-region assertions use a small adjacency tolerance for geo-elastic execution (e.g. kix1 → hnd1); run-ID tags and data placement are always asserted strictly.