Uh oh!
There was an error while loading. Please reload this page.
feat(ocap-kernel): launch subcluster vats in parallel - #983
Conversation
Previously #launchVatsForSubcluster awaited each vat's initVat
handshake serially, making startup O(sum of vat init times).
Now all vats launch concurrently via Promise.all. Each vat gets a
kernel promise pre-allocated for its root object; the bootstrap
message is queued immediately targeting the bootstrap vat's
(unresolved) promise. KernelRouter parks the send until that promise
resolves. As each vat's handshake completes its root promise is
resolved via resolvePromises('kernel', ...), making startup O(max).
The bootstrap vat receives kernel promises for all other vats' roots
in its bootstrap() call, so it can pipeline calls to those vats while
they are still initializing.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>grypez
commented
Jul 27, 2026
Benchmark: serial vs parallel subcluster startupMeasured on real worker processes (
Serial scaled linearly (~865 ms × N). Parallel is bounded by the slowest vat; the ~120 ms overhead above 1-vat for the 5-vat case is kernel promise allocation and resolution bookkeeping. Raw numbersSerial (HEAD^) Parallel (this PR) |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
…nchVatsForSubcluster Both loops iterate vatEntries identically. Building roots[vatName] from the locally-scoped kpid in the same pass eliminates the second loop, the ! non-null assertion, and the eslint-disable comment that accompanied it. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ise.all result Instead of mutating an outer variable via a .then() side-effect, collect the resolved root refs as the return value of Promise.all and extract the bootstrap vat's ref by index. Promise.all preserves insertion order, so the index is stable. config.bootstrap is guaranteed present (validated before this method is called), so the as-KRef cast is correct. Also removes the dead-code guard that followed — if Promise.all resolves, every vat (including bootstrap) launched successfully, so rootRefs[idx] is always defined. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When a vat in a multi-vat subcluster fails to launch, its pre-allocated kernel promise was left unresolved. Any bootstrap message pipelined through that promise (E(roots.failingPeer).method()) would park forever. Add a .catch() to the launchVat chain that calls resolvePromises with rejected=true and a VAT_TERMINATED kernel error, then re-throws so the outer Promise.all still rejects and launchSubcluster surfaces the error. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…r rejection Add a peer-rejection-bootstrap vat that calls E(peer).ping() and logs whether the call resolves or rejects, making peer-vat-launch failures observable from the bootstrap vat. Add an integration test that launches a two-vat cluster where the peer vat (error-build-throw) always fails to build its root object. The test asserts that launchSubcluster rejects and that, after the kernel run loop drains, the bootstrap vat has logged a rejection message containing 'VAT_TERMINATED' — confirming that the rejected kernel promise propagates to the bootstrap vat via E() pipelining. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
…omise] failures Switch #launchVatsForSubcluster from Promise.all with pre-allocated kp<N> refs to Promise.allSettled. Wait for all vats to settle, then send bootstrap with real ko<N> refs for succeeded vats and immediately-rejected kp<N> for failed peers. The previous approach sent bootstrap before any vat launched, so bootstrap received pending kp<N> refs. If bootstrap returned a value pipelined through those refs, the result CapData contained kp<N> slots, and kunser() would fail with "value is not durable: '[Promise]'" when trying to deserialize them. The new approach preserves concurrent vat launch (allSettled, not allSettled- then-serial) while ensuring bootstrap always gets concrete refs. Failed peer vats still get an immediately-rejected kernel promise so bootstrap can observe the failure via E(roots.peer).method() pipelining. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…t launch With concurrent subcluster startup (PR #983), ko<N> refs are assigned in the order vat workers respond — not the order vats are declared. Tests that hardcoded ko4/ko5/ko6 for alice/bob/carol broke because those refs are now non-deterministic. - Add `vatRootKrefs: Record<string, KRef>` to `SubclusterLaunchResult` so callers can look up the root KRef of any vat by name - persistence.test.ts: use `rootKref` from `launchSubcluster` instead of hardcoded ko4 for the multi-vat coordinator test - resume.test.ts: use `vatRootKrefs.alice/bob/carol` instead of ko4/ko5/ko6 - rejection.test.ts: sort before comparing `getVatIds()` output; Map insertion order is non-deterministic with concurrent launch Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
The local Fail tag is declared as returning Error (not never) to work around a TypeScript control-flow analysis bug. Using `x ?? Fail` therefore widens the type to `T | Error`, breaking tsc on CI even though ts-jest's transpile-only mode accepts it. Replace each occurrence with an explicit undefined guard and throw. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…edVat When BOYD (bringOutYourDead) runs for an importing vat before terminateAllVats, dropImports/retireImports decrements the exported object's refcount to (0,0). cleanupTerminatedVat then tries to decrement the baseline (1,1) that was set at object creation, causing an underflow. Guard: check the current reachable count before the baseline decrement and skip it if reachable is already zero. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Coverage Report
File Coverage
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
…ction
vatName and service names come from ClusterConfig (user-provided). Using
a plain {} as the accumulator allowed prototype pollution if a name like
'__proto__' or 'constructor' was supplied. Object.create(null) removes
the prototype chain, eliminating the risk.
vatRootKrefs is spread into a plain {} at the return site so callers
that use toStrictEqual continue to work.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>Validate that no vat name shadows Object.prototype built-ins (e.g. __proto__, constructor) before using vatName as a property key on the roots/vatRootKrefs maps. Adds // lgtm[js/remote-property-injection] suppressions on the three write sites as a belt-and-suspenders measure for CodeQL, which does not track the null-prototype path through the TypeScript cast. Object.create(null) was considered but rejected: @endo/marshal requires Object.prototype-chained objects for CopyRecord serialization. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
GitHub Code Scanning requires '// lgtm [rule]' with a space, not '// lgtm[rule]' without one. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Accumulate roots and vatRootKrefs into entry arrays and call Object.fromEntries at the end, eliminating the obj[taintedKey] = value write pattern that CodeQL's js/remote-property-injection rule flags. @endo/marshal requires Object.prototype-chained CopyRecords, so Object.create(null) was not viable. The // lgtm suppression syntax is also not active on this repo. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…e arg - Move the Object.prototype name check into the main loop body, eliminating a separate pre-pass over vatEntries. - Drop the 'vatRoot' iface arg from kslot(kpid): kslot ignores iface for kp-prefixed refs (returns makeStandinPromise early), so the arg was dead code that implied it had an effect. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
grypez
commented
Jul 30, 2026
Design note: |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
The parallel launch path built kslot(result.value) without an iface argument, so vat root remotes were branded 'Alleged: undefined' instead of 'Alleged: vatRoot'. Pass 'vatRoot' as the second argument to match the previous serial behaviour. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When a non-bootstrap vat failed to launch, the catch block in launchSubcluster only destroyed IO channels and removed the subcluster record from the store, leaving any successfully-started vat workers running as orphans with no subcluster membership. The fix iterates getSubclusterVats and calls terminateVat/collectGarbage for each vatId where hasVat returns true (i.e., the worker is alive). Vats whose launch failed are not in the VatManager's live map so hasVat returns false and they are skipped. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…cluster-startup # Conflicts: # packages/ocap-kernel/CHANGELOG.md # packages/ocap-kernel/src/vats/SubclusterManager.test.ts
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 1ff7a6a. Configure here.
Uh oh!
There was an error while loading. Please reload this page.
The launch rollback terminated vats in a loop wrapped in a single try/catch, so one vat that failed to die abandoned every vat after it while deleteSubcluster still removed the record — leaving vats running with no subcluster handle to terminate them by. Extract the per-vat teardown into #terminateVatQuietly so each vat's cleanup is independently best-effort. Terminating one vat at a time is also the shape the eventual retry path needs, where a failed vat is reaped on its own and the subcluster survives. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Uh oh!
There was an error while loading. Please reload this page.
The `cluster initialization` suite capped every test at 4s, a budget that also has to cover the kernel each test constructs in `beforeEach`. That construction costs ~0.4s on an idle machine but ~2s under CPU contention, so a loaded runner — the merge queue, or a local full-suite run — blows the cap in the hook and fails a test that never ran. Fall back to the package's 30s default for both suites in the file. Nothing here needs a tighter bound; a genuine hang still fails, just 26s later. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Uh oh!
There was an error while loading. Please reload this page.
Recording the vat's death only saves the deliveries that come after it. The one in flight when the worker died stays parked on an RPC client with no timeout, so its crank never completes — the same hang `onCriticalFailure` exists to prevent, one delivery earlier. The worker was left running too, since nothing else would stop it once the handle was off the books. Found by Cursor Bugbot on #1023. Also reverts this branch's additions to the extension control-panel e2e test. They asserted `ko6.refCount` directly, and which vat owns `ko6` is not stable: #983 launches subcluster vats in parallel, so the root krefs fall in completion order. The behaviour they checked is covered by the refcount audit, which runs on every kernel `kernel-test` builds. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…hem (MetaMask#1024) Fixes a pre-existing e2e flake on `main`, surfaced while rebasing the MetaMask#1020–MetaMask#1023 stack. Small and self-contained so the whole stack inherits it. ## The defect `control-panel.test.ts` › `should collect garbage` asserted that Carol's root object is `ko6` and Bob's is `ko5`, and that their promises are `kp4` and `kp3`: ```js '{"key":"ko6.owner","value":"v3"}', '{"key":"v3.c.ko6","value":"R o+0"}', ``` Since MetaMask#983, subcluster vats launch **in parallel**. Each vat's root is exported when its own launch finishes, so which of `ko5`/`ko6` belongs to Bob and which to Carol changes between runs. When they come back the other way round, the test fails — and `database-inspector.test.ts` fails alongside it, because it reads the same kv dump. Observed directly: a failing run had `ko6.owner = v2` and `ko5.owner = v3`, the exact inverse of what is asserted. The vat ids themselves are stable — they are handed out in config order, so alice is always `v1` — so only the object and promise krefs need deriving. ## Approach Three small helpers read the dump and look up what the assertions used to hardcode: `rootKrefOf(dump, vatId)` by owner, `promiseKrefOf(dump, vatId)` by c-list entry, and `erefOf(dump, vatId, kref)`. The erefs are derived in **full** rather than matched by prefix. A c-list entry's reverse direction is keyed by eref and valued by kref, so a loose `,"value":"ko5"}` also matches the *owning* vat's own `v2.c.o+0` entry. That passed while both vats were alive and broke the negative assertions the moment one outlived the other — which is what the test checks after terminating v3. ## Testing `yarn lint` clean. Extension e2e run three times: the kref failure is gone, and the two clean runs finish in ~50s rather than ~2.7m because no retries are needed. **What this does not fix.** The extension e2e suite has separate instability that this change does not touch and does not claim to: `object-registry.test.ts` failures, and a UI timing flake where `Terminated vat "v1"` does not render because the panel is still showing query output. One of the three runs hit those. They are unrelated to kref assignment and were present before this change. ## Checklist - [x] I've updated the test suite for new or updated code as appropriate - [x] I've updated documentation (JSDoc, `README.md`, `CHANGELOG.md`) as appropriate — test-only change, no changelog entry <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Low Risk** > Test-only change to e2e assertions and helpers; no production or runtime behavior is modified. > > **Overview** > Fixes flaky **`should collect garbage`** assertions in `control-panel.test.ts` that assumed fixed kernel refs (`ko5`/`ko6`, `kp3`/`kp4`) for Bob and Carol. Parallel subcluster launches mean those object and promise krefs can swap between runs while vat ids (`v2`/`v3`) stay stable. > > Adds helpers to parse the Database Inspector kv dump and **derive** root krefs (via `.owner`), promise krefs (via c-list), and v1’s **erefs** (full c-list lookup so reverse entries don’t false-match). The garbage-collection expectations are built from those values instead of literals. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit d8e81f7. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY --> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

Explanation
Subcluster startup was O(sum of vat init times) because
#launchVatsForSubclusterused a serialfor...ofloop — each vat'sinitVatRPC handshake had to complete before the next one began.This PR changes startup to O(max vat init time) by launching all vats concurrently via
Promise.all. The mechanism:await, one kernel promise (kp<N>) is pre-allocated per vat and marked as kernel-decided.kp<N>.KernelRouterparks the send on the promise viaenqueuePromiseMessage.initVathandshake completes, its root kernel promise is resolved viaresolvePromises('kernel', ...), and the run loop forwards any queued messages.bootstrap(roots, services)call — it can pipeline calls to peers while they are still initializing.Changes
SubclusterManager.ts— rewrote#launchVatsForSubclusterto pre-allocate kernel promises, queue bootstrap immediately, and launch all vats withPromise.all.SubclusterManager.test.ts— updated mocks (initKernelPromise,setPromiseDecider,resolvePromises) and assertions to match the new flow.Kernel.test.ts— addedresolvePromises = vi.fn()to theKernelQueuemock class.Checklist
yarn workspace @metamask/ocap-kernel test:dev:quiet)yarn workspace @metamask/ocap-kernel build)Note
Medium Risk
Changes core subcluster launch ordering, failure propagation, and cleanup in the kernel; mistakes could leak workers or break bootstrap peer wiring, though coverage is expanded in unit and integration tests.
Overview
Parallel subcluster startup replaces serial vat launch in
SubclusterManagerwithPromise.allSettled, so subcluster bring-up time tracks the slowest vat instead of the sum of init times. Kernel service resolution still runs before any vat starts.After launches settle, bootstrap is invoked with real root
korefs for successful vats. Failed peer vats get an immediately rejected kernel promise (VAT_TERMINATED) in therootsmap so bootstrap can observe the failure via pipelinedE(roots.peer)calls;launchSubclusterstill rejects once bootstrap has run.SubclusterLaunchResultnow includesvatRootKrefs(name → root kref for vats that launched successfully).Failed launches tear down any vats that did start (
#terminateVatQuietlyin reverse order) before IO/subcluster rollback. Vat cleanup skips a baseline refcount decrement when GC already zeroed reachability (vat.ts).Integration tests drop hardcoded
ko4/ko5/ko6and usevatRootKrefs/rootKreffromlaunchSubcluster; a new peer rejection integration test and bootstrap vat bundle were added. Changelog documents concurrent launch and peer rejection behavior.Reviewed by Cursor Bugbot for commit 6a2492a. Bugbot is set up for automated code reviews on this repo. Configure here.