Skip to content

feat(ocap-kernel): launch subcluster vats in parallel - #983

Merged
grypez merged 21 commits into
mainfrom
grypez/concurrent-subcluster-startup
Aug 11, 2026
Merged

feat(ocap-kernel): launch subcluster vats in parallel#983
grypez merged 21 commits into
mainfrom
grypez/concurrent-subcluster-startup

Conversation

@grypez

@grypezgrypez commented Jul 27, 2026

Copy link
Copy Markdown
Member

Explanation

Subcluster startup was O(sum of vat init times) because #launchVatsForSubcluster used a serial for...of loop — each vat's initVat RPC handshake had to complete before the next one began.

This PR changes startup to O(max vat init time) by launching all vats concurrently via Promise.all. The mechanism:

  1. Before any await, one kernel promise (kp<N>) is pre-allocated per vat and marked as kernel-decided.
  2. The bootstrap message is queued immediately, targeting the bootstrap vat's unresolved kp<N>. KernelRouter parks the send on the promise via enqueuePromiseMessage.
  3. All vats launch in parallel. As each vat's initVat handshake completes, its root kernel promise is resolved via resolvePromises('kernel', ...), and the run loop forwards any queued messages.
  4. The bootstrap vat receives kernel promise KRefs for all peer vats in its bootstrap(roots, services) call — it can pipeline calls to peers while they are still initializing.

Changes

  • SubclusterManager.ts — rewrote #launchVatsForSubcluster to pre-allocate kernel promises, queue bootstrap immediately, and launch all vats with Promise.all.
  • SubclusterManager.test.ts — updated mocks (initKernelPromise, setPromiseDecider, resolvePromises) and assertions to match the new flow.
  • Kernel.test.ts — added resolvePromises = vi.fn() to the KernelQueue mock class.

Checklist

  • Tests pass (yarn workspace @metamask/ocap-kernel test:dev:quiet)
  • Build passes (yarn workspace @metamask/ocap-kernel build)
  • Changelog updated

Note

Medium Risk
Changes core subcluster launch ordering, failure propagation, and cleanup in the kernel; mistakes could leak workers or break bootstrap peer wiring, though coverage is expanded in unit and integration tests.

Overview
Parallel subcluster startup replaces serial vat launch in SubclusterManager with Promise.allSettled, so subcluster bring-up time tracks the slowest vat instead of the sum of init times. Kernel service resolution still runs before any vat starts.

After launches settle, bootstrap is invoked with real root ko refs for successful vats. Failed peer vats get an immediately rejected kernel promise (VAT_TERMINATED) in the roots map so bootstrap can observe the failure via pipelined E(roots.peer) calls; launchSubcluster still rejects once bootstrap has run. SubclusterLaunchResult now includes vatRootKrefs (name → root kref for vats that launched successfully).

Failed launches tear down any vats that did start (#terminateVatQuietly in reverse order) before IO/subcluster rollback. Vat cleanup skips a baseline refcount decrement when GC already zeroed reachability (vat.ts).

Integration tests drop hardcoded ko4/ko5/ko6 and use vatRootKrefs / rootKref from launchSubcluster; a new peer rejection integration test and bootstrap vat bundle were added. Changelog documents concurrent launch and peer rejection behavior.

Reviewed by Cursor Bugbot for commit 6a2492a. Bugbot is set up for automated code reviews on this repo. Configure here.

Previously #launchVatsForSubcluster awaited each vat's initVat
handshake serially, making startup O(sum of vat init times).
Now all vats launch concurrently via Promise.all. Each vat gets a
kernel promise pre-allocated for its root object; the bootstrap
message is queued immediately targeting the bootstrap vat's
(unresolved) promise. KernelRouter parks the send until that promise
resolves. As each vat's handshake completes its root promise is
resolved via resolvePromises('kernel', ...), making startup O(max).
The bootstrap vat receives kernel promises for all other vats' roots
in its bootstrap() call, so it can pipeline calls to those vats while
they are still initializing.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@grypez

Copy link
Copy Markdown
MemberAuthor

Benchmark: serial vs parallel subcluster startup

Measured on real worker processes (NodejsPlatformServices), 5 iterations each. Single-vat launch ≈ 865 ms (worker spawn + initVat handshake).

VatsSerial (before)Parallel (after)Speedup
1865 ms868 ms1.0×
32599 ms880 ms3.0×
54398 ms987 ms4.5×

Serial scaled linearly (~865 ms × N). Parallel is bounded by the slowest vat; the ~120 ms overhead above 1-vat for the 5-vat case is kernel promise allocation and resolution bookkeeping.

Raw numbers

Serial (HEAD^)

[bench] 1-vat: avg=865ms min=851ms max=889ms (851, 889, 865, 866, 857ms)
[bench] 3-vat: avg=2599ms min=2581ms max=2636ms (2582, 2602, 2592, 2581, 2636ms)
[bench] 5-vat: avg=4398ms min=4363ms max=4424ms (4424, 4411, 4363, 4376, 4416ms)

Parallel (this PR)

[bench] 1-vat: avg=868ms min=852ms max=892ms (881, 892, 853, 852, 862ms)
[bench] 3-vat: avg=880ms min=852ms max=906ms (906, 863, 852, 872, 905ms)
[bench] 5-vat: avg=987ms min=974ms max=1002ms (979, 987, 1002, 994, 974ms)

Comment threadpackages/ocap-kernel/src/vats/SubclusterManager.ts Fixed
Comment threadpackages/ocap-kernel/src/vats/SubclusterManager.ts Fixed
grypezand others added 4 commits July 27, 2026 10:55
…nchVatsForSubcluster
Both loops iterate vatEntries identically. Building roots[vatName] from the
locally-scoped kpid in the same pass eliminates the second loop, the
! non-null assertion, and the eslint-disable comment that accompanied it.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ise.all result
Instead of mutating an outer variable via a .then() side-effect, collect the
resolved root refs as the return value of Promise.all and extract the bootstrap
vat's ref by index. Promise.all preserves insertion order, so the index is
stable. config.bootstrap is guaranteed present (validated before this method
is called), so the as-KRef cast is correct.
Also removes the dead-code guard that followed — if Promise.all resolves,
every vat (including bootstrap) launched successfully, so rootRefs[idx] is
always defined.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When a vat in a multi-vat subcluster fails to launch, its pre-allocated
kernel promise was left unresolved. Any bootstrap message pipelined
through that promise (E(roots.failingPeer).method()) would park forever.
Add a .catch() to the launchVat chain that calls resolvePromises with
rejected=true and a VAT_TERMINATED kernel error, then re-throws so the
outer Promise.all still rejects and launchSubcluster surfaces the error.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…r rejection
Add a peer-rejection-bootstrap vat that calls E(peer).ping() and logs
whether the call resolves or rejects, making peer-vat-launch failures
observable from the bootstrap vat.
Add an integration test that launches a two-vat cluster where the peer
vat (error-build-throw) always fails to build its root object. The test
asserts that launchSubcluster rejects and that, after the kernel run
loop drains, the bootstrap vat has logged a rejection message containing
'VAT_TERMINATED' — confirming that the rejected kernel promise
propagates to the bootstrap vat via E() pipelining.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Comment threadpackages/ocap-kernel/src/vats/SubclusterManager.ts Fixed
Comment threadpackages/ocap-kernel/src/vats/SubclusterManager.ts Fixed
grypezand others added 2 commits July 27, 2026 12:55
…omise] failures
Switch #launchVatsForSubcluster from Promise.all with pre-allocated kp<N> refs
to Promise.allSettled. Wait for all vats to settle, then send bootstrap with
real ko<N> refs for succeeded vats and immediately-rejected kp<N> for failed
peers.
The previous approach sent bootstrap before any vat launched, so bootstrap
received pending kp<N> refs. If bootstrap returned a value pipelined through
those refs, the result CapData contained kp<N> slots, and kunser() would fail
with "value is not durable: '[Promise]'" when trying to deserialize them.
The new approach preserves concurrent vat launch (allSettled, not allSettled-
then-serial) while ensuring bootstrap always gets concrete refs. Failed peer
vats still get an immediately-rejected kernel promise so bootstrap can observe
the failure via E(roots.peer).method() pipelining.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…t launch
With concurrent subcluster startup (PR #983), ko<N> refs are assigned in
the order vat workers respond — not the order vats are declared. Tests that
hardcoded ko4/ko5/ko6 for alice/bob/carol broke because those refs are now
non-deterministic.
- Add `vatRootKrefs: Record<string, KRef>` to `SubclusterLaunchResult` so
callers can look up the root KRef of any vat by name
- persistence.test.ts: use `rootKref` from `launchSubcluster` instead of
hardcoded ko4 for the multi-vat coordinator test
- resume.test.ts: use `vatRootKrefs.alice/bob/carol` instead of ko4/ko5/ko6
- rejection.test.ts: sort before comparing `getVatIds()` output; Map
insertion order is non-deterministic with concurrent launch
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Comment threadpackages/ocap-kernel/src/vats/SubclusterManager.ts Fixed
Comment threadpackages/ocap-kernel/src/vats/SubclusterManager.ts Fixed
Comment threadpackages/ocap-kernel/src/vats/SubclusterManager.ts Fixed
grypezand others added 3 commits July 28, 2026 11:01
The local Fail tag is declared as returning Error (not never) to work around
a TypeScript control-flow analysis bug. Using `x ?? Fail` therefore widens
the type to `T | Error`, breaking tsc on CI even though ts-jest's
transpile-only mode accepts it.
Replace each occurrence with an explicit undefined guard and throw.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…edVat
When BOYD (bringOutYourDead) runs for an importing vat before
terminateAllVats, dropImports/retireImports decrements the exported
object's refcount to (0,0). cleanupTerminatedVat then tries to
decrement the baseline (1,1) that was set at object creation, causing
an underflow.
Guard: check the current reachable count before the baseline decrement
and skip it if reachable is already zero.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@github-actions

github-actionsBot commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Coverage Report

StatusCategoryPercentageCovered / Total
🔵Lines72.06%
⬆️ +0.05%
9353 / 12979
🔵Statements71.9%
⬆️ +0.05%
9507 / 13222
🔵Functions72.87%
⬇️ -0.07%
2227 / 3056
🔵Branches65.82%
⬆️ +0.05%
3788 / 5755
File Coverage
FileStmtsBranchesFunctionsLinesUncovered Lines
Changed Files
packages/kernel-test/src/vats/peer-rejection-bootstrap.ts0%0%0%0%14-24
packages/ocap-kernel/src/types.ts100%
🟰 ±0%
100%
🟰 ±0%
100%
🟰 ±0%
100%
🟰 ±0%
packages/ocap-kernel/src/store/methods/vat.ts97.29%
⬆️ +0.03%
81.81%
⬇️ -1.52%
100%
🟰 ±0%
97.27%
⬆️ +0.03%
224, 297, 304-305
packages/ocap-kernel/src/vats/SubclusterManager.ts96.19%
⬇️ -0.43%
90.12%
⬇️ -2.18%
100%
🟰 ±0%
96.13%
⬇️ -0.44%
155-158, 236-239, 293, 368, 388, 405
Generated in workflow #4625 for commit 6a2492a by the Vitest Coverage Report Action

grypezand others added 2 commits July 28, 2026 12:50
…ction
vatName and service names come from ClusterConfig (user-provided). Using
a plain {} as the accumulator allowed prototype pollution if a name like
'__proto__' or 'constructor' was supplied. Object.create(null) removes
the prototype chain, eliminating the risk.
vatRootKrefs is spread into a plain {} at the return site so callers
that use toStrictEqual continue to work.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Validate that no vat name shadows Object.prototype built-ins (e.g.
__proto__, constructor) before using vatName as a property key on the
roots/vatRootKrefs maps. Adds // lgtm[js/remote-property-injection]
suppressions on the three write sites as a belt-and-suspenders measure
for CodeQL, which does not track the null-prototype path through the
TypeScript cast.
Object.create(null) was considered but rejected: @endo/marshal requires
Object.prototype-chained objects for CopyRecord serialization.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Comment threadpackages/ocap-kernel/src/vats/SubclusterManager.ts Fixed
Comment threadpackages/ocap-kernel/src/vats/SubclusterManager.ts Fixed
Comment threadpackages/ocap-kernel/src/vats/SubclusterManager.ts Fixed
grypezand others added 3 commits July 28, 2026 13:10
GitHub Code Scanning requires '// lgtm [rule]' with a space, not
'// lgtm[rule]' without one.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Accumulate roots and vatRootKrefs into entry arrays and call
Object.fromEntries at the end, eliminating the obj[taintedKey] = value
write pattern that CodeQL's js/remote-property-injection rule flags.
@endo/marshal requires Object.prototype-chained CopyRecords, so
Object.create(null) was not viable. The // lgtm suppression syntax is
also not active on this repo.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…e arg
- Move the Object.prototype name check into the main loop body,
eliminating a separate pre-pass over vatEntries.
- Drop the 'vatRoot' iface arg from kslot(kpid): kslot ignores iface
for kp-prefixed refs (returns makeStandinPromise early), so the arg
was dead code that implied it had an effect.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@grypez

Copy link
Copy Markdown
MemberAuthor

Design note: Promise.allSettled vs. early-bootstrap with peer kernel promises

During review we considered an alternative to the current Promise.allSettled approach: start bootstrap as soon as its own launch resolves, passing a kernel promise (kpid) for each peer vat and resolving/rejecting it when that peer's launch settles. This would let bootstrap overlap with slow peer startups.

Resolved-promise inlining: verified absent. To assess the per-call cost of passing kpid instead of ko<N>, we traced the kernel's message routing. KernelQueue.enqueueSend never checks promise state — routing happens only in KernelRouter.#routeMessage at dequeue time. So even for an already-resolved kpid, a send targeting it re-enters the run queue and takes one extra crank before reaching ko<N>. There is no inlining shortcut.

But the extra crank only hurts awaited calls. The kpid approach also changes the natural programming style: bootstrap has no reason to await E(peer).method() — it can fire-and-forget, letting the kernel queue those messages against the still-unresolved kpid and deliver them once the peer is up. For that pattern the extra crank is essentially free. The cost is real only when code needs a return value from a peer, and it would apply to all callers of those peer refs, not just bootstrap.

Current choice:Promise.allSettled is simpler, has zero per-call overhead for successful peers, and startup latency hasn't been a measured bottleneck. Worth revisiting if that changes.

@grypez
grypez marked this pull request as ready for review July 30, 2026 18:13
@grypez
grypez requested a review from a team as a code ownerJuly 30, 2026 18:13
Comment threadpackages/ocap-kernel/src/vats/SubclusterManager.ts
Comment threadpackages/ocap-kernel/src/vats/SubclusterManager.ts
grypezand others added 2 commits August 3, 2026 10:31
The parallel launch path built kslot(result.value) without an iface
argument, so vat root remotes were branded 'Alleged: undefined' instead
of 'Alleged: vatRoot'. Pass 'vatRoot' as the second argument to match
the previous serial behaviour.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When a non-bootstrap vat failed to launch, the catch block in
launchSubcluster only destroyed IO channels and removed the subcluster
record from the store, leaving any successfully-started vat workers
running as orphans with no subcluster membership.
The fix iterates getSubclusterVats and calls terminateVat/collectGarbage
for each vatId where hasVat returns true (i.e., the worker is alive).
Vats whose launch failed are not in the VatManager's live map so hasVat
returns false and they are skipped.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…cluster-startup
# Conflicts:
#	packages/ocap-kernel/CHANGELOG.md
#	packages/ocap-kernel/src/vats/SubclusterManager.test.ts

@cursorcursorBot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 1ff7a6a. Configure here.

Comment threadpackages/ocap-kernel/src/vats/SubclusterManager.ts
The launch rollback terminated vats in a loop wrapped in a single
try/catch, so one vat that failed to die abandoned every vat after it
while deleteSubcluster still removed the record — leaving vats running
with no subcluster handle to terminate them by.
Extract the per-vat teardown into #terminateVatQuietly so each vat's
cleanup is independently best-effort. Terminating one vat at a time is
also the shape the eventual retry path needs, where a failed vat is
reaped on its own and the subcluster survives.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
sirtimid
sirtimid previously approved these changes Aug 11, 2026
@grypez
grypez added this pull request to the merge queueAug 11, 2026
@github-merge-queue
github-merge-queueBot removed this pull request from the merge queue due to failed status checks Aug 11, 2026
grypezand others added 2 commits August 11, 2026 11:16
The `cluster initialization` suite capped every test at 4s, a budget that
also has to cover the kernel each test constructs in `beforeEach`. That
construction costs ~0.4s on an idle machine but ~2s under CPU contention,
so a loaded runner — the merge queue, or a local full-suite run — blows
the cap in the hook and fails a test that never ran.
Fall back to the package's 30s default for both suites in the file.
Nothing here needs a tighter bound; a genuine hang still fails, just 26s
later.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@grypez
grypez added this pull request to the merge queueAug 11, 2026
Merged via the queue into main with commit 494fb5eAug 11, 2026
33 checks passed
@grypez
grypez deleted the grypez/concurrent-subcluster-startup branch August 11, 2026 15:43
sirtimid added a commit that referenced this pull request Aug 13, 2026
Recording the vat's death only saves the deliveries that come after it. The one
in flight when the worker died stays parked on an RPC client with no timeout, so
its crank never completes — the same hang `onCriticalFailure` exists to prevent,
one delivery earlier. The worker was left running too, since nothing else would
stop it once the handle was off the books.
Found by Cursor Bugbot on #1023.
Also reverts this branch's additions to the extension control-panel e2e test.
They asserted `ko6.refCount` directly, and which vat owns `ko6` is not stable:
#983 launches subcluster vats in parallel, so the root krefs fall in completion
order. The behaviour they checked is covered by the refcount audit, which runs
on every kernel `kernel-test` builds.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
SherfeyInv pushed a commit to SherfeyInv/ocap-kernel that referenced this pull request Aug 17, 2026
…hem (MetaMask#1024)
Fixes a pre-existing e2e flake on `main`, surfaced while rebasing the
MetaMask#1020MetaMask#1023 stack. Small and self-contained so the whole stack inherits
it.
## The defect
`control-panel.test.ts` › `should collect garbage` asserted that Carol's
root object is `ko6` and Bob's is `ko5`, and that their promises are
`kp4` and `kp3`:
```js
'{"key":"ko6.owner","value":"v3"}',
'{"key":"v3.c.ko6","value":"R o+0"}',
```
Since MetaMask#983, subcluster vats launch **in parallel**. Each vat's root is
exported when its own launch finishes, so which of `ko5`/`ko6` belongs
to Bob and which to Carol changes between runs. When they come back the
other way round, the test fails — and `database-inspector.test.ts` fails
alongside it, because it reads the same kv dump.
Observed directly: a failing run had `ko6.owner = v2` and `ko5.owner =
v3`, the exact inverse of what is asserted.
The vat ids themselves are stable — they are handed out in config order,
so alice is always `v1` — so only the object and promise krefs need
deriving.
## Approach
Three small helpers read the dump and look up what the assertions used
to hardcode: `rootKrefOf(dump, vatId)` by owner, `promiseKrefOf(dump,
vatId)` by c-list entry, and `erefOf(dump, vatId, kref)`.
The erefs are derived in **full** rather than matched by prefix. A
c-list entry's reverse direction is keyed by eref and valued by kref, so
a loose `,"value":"ko5"}` also matches the *owning* vat's own `v2.c.o+0`
entry. That passed while both vats were alive and broke the negative
assertions the moment one outlived the other — which is what the test
checks after terminating v3.
## Testing
`yarn lint` clean. Extension e2e run three times: the kref failure is
gone, and the two clean runs finish in ~50s rather than ~2.7m because no
retries are needed.
**What this does not fix.** The extension e2e suite has separate
instability that this change does not touch and does not claim to:
`object-registry.test.ts` failures, and a UI timing flake where
`Terminated vat "v1"` does not render because the panel is still showing
query output. One of the three runs hit those. They are unrelated to
kref assignment and were present before this change.
## Checklist
- [x] I've updated the test suite for new or updated code as appropriate
- [x] I've updated documentation (JSDoc, `README.md`, `CHANGELOG.md`) as
appropriate — test-only change, no changelog entry
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Low Risk**
> Test-only change to e2e assertions and helpers; no production or
runtime behavior is modified.
> > **Overview**
> Fixes flaky **`should collect garbage`** assertions in
`control-panel.test.ts` that assumed fixed kernel refs (`ko5`/`ko6`,
`kp3`/`kp4`) for Bob and Carol. Parallel subcluster launches mean those
object and promise krefs can swap between runs while vat ids (`v2`/`v3`)
stay stable.
> > Adds helpers to parse the Database Inspector kv dump and **derive**
root krefs (via `.owner`), promise krefs (via c-list), and v1’s
**erefs** (full c-list lookup so reverse entries don’t false-match). The
garbage-collection expectations are built from those values instead of
literals.
> > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
d8e81f7. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@grypez@sirtimid@github-advanced-security