Runtime: prove generator and iterator side-store heap-cap recovery under no-GIL

[chrisbbreuer]

Parent: #30

Goal

Define and implement the safe recovery policy for GC-backed generator / iterator-helper side-store allocation pressure under live no-GIL peers.

Why this is separate

#30 has proven no-GIL emergency recovery for GC-cell slabs and safepoint-owned ArrayBuffer byte slabs, while trace-sensitive Object / Promise / Environment / async-generator request critical sections intentionally fail closed.

Generator and iterator-helper side stores are a different seam: the parallel tracer defers their mutable storage to the world-stopped finish path because a running generator's exec stack, async-generator request state, resumable handler buffers, and iterator-helper backing state are not sound to trace directly while mutators run. The allocation-failure collector refuses to sweep while those deferred edges remain, so treating these as an ordinary side-store retry class would either fail to reclaim or risk weakening the tracing invariant.

Current status

The negative/fail-closed policy is now executable for the deferred generator/iterator family:

• Suspended generator stack / execution storage fails closed while no-GIL peers are live.
• Resumable generator handler buffers fail closed while no-GIL peers are live.
• Pending async-generator request buffers fail closed while no-GIL peers are live.
• Iterator-helper backing state fails closed while no-GIL peers are live.
• Deferred-cell aborts are reported separately as gc_par_deferred_aborts / def-abort, while still preserving the existing round-limit abort accounting identity.
• docs/threads/limits.md and docs/threads/testing.md distinguish trace-sensitive locks from deferred generator/iterator tracing.

The current design evidence points to a conservative rule: deferred generator/iterator state should not be recovered by the live no-GIL allocation-failure collector unless a future allocation site can prove no deferred edges remain, or the collector can finish through a sound world-stopped path. No positive recoverable generator/iterator class has been proven yet.

Closure policy

Decision after the combined main gate for d0092f5e passed (https://github.com/zig-utils/zig-js/actions/runs/29201642565): no current generator/iterator side-store allocation class is sound to recover under live no-GIL peers. The implemented policy is intentionally all-fail-closed while deferred generator/iterator edges remain pending.

This preserves the tracer invariant: a parallel allocation-failure collector may not claim a sweep while suspended generator stacks, resumable handler buffers, async-generator request buffers, or iterator-helper backing state have been deferred to the world-stopped finish path.

Future work should reopen this tracker or file a narrower follow-up only if a specific allocation site can prove either that no deferred edges remain or that recovery can finish through a sound world-stopped path. Any such positive class needs a focused witness plus the existing negative fail-closed coverage.

Acceptance criteria

• No-GIL TSan and semantic gates remain suppression-free.
• Every newly recoverable generator/iterator allocation class has a bounded focused test.
• Deferred generator/iterator tracing still prevents unsafe parallel sweep; no generic side-store retry bypasses that invariant.
• If no positive class is sound, Runtime: extend heap-cap emergency recovery beyond current safe allocation classes #30 documents the fail-closed policy as intentional rather than incomplete.

[chrisbbreuer]

Landed the first #36 policy witness in zig-js@9a10bb7c (test(oom): cover deferred generator recovery guard).

What changed:

• Added a focused no-GIL heap-cap recovery test that roots a suspended generator, then directly attempts allocation-failure recovery while a peer is live.
• The test asserts the abort-safe parallel collector does not report recovery/sweep when generator tracing is deferred to a world-stopped finish.
• Telemetry now pins the intended boundary: attempts increase, collections do not, round-limit aborts increase, and gc_par_deferred_rounds is observed.
• Updated docs/threads/limits.md and docs/threads/testing.md to distinguish trace-sensitive side-store locks from deferred generator/iterator tracing.

Validation:

• /Users/chris/.local/bin/zig build test -Dtest-filter="deferred generator tracing" --summary all
• /Users/chris/.bun/bin/bun run docs:build
• git diff --check

CI for this push is pending: https://github.com/zig-utils/zig-js/actions/runs/29200654888

This does not implement a positive recoverable generator/iterator class yet; it makes the fail-closed invariant executable before we try to broaden recovery.

[chrisbbreuer]

Follow-up #36 policy witness landed in zig-js@97c511c3 (test(oom): cover iterator helper recovery guard).

What changed:

• Added the iterator-helper sibling to the deferred-generator recovery guard.
• The new test roots a mid-helper Iterator.from(...).map(...).filter(...) state, then attempts no-GIL allocation-failure recovery while a peer is live.
• It asserts the parallel collector still fails closed: attempts increase, collections do not, abort/round-limit telemetry increases, and gc_par_deferred_rounds is observed.
• Updated the testing guide to say both deferred generator and deferred iterator-helper witnesses cover this boundary.

Validation:

• /Users/chris/.local/bin/zig build test -Dtest-filter="deferred iterator helper tracing" --summary all
• /Users/chris/.bun/bin/bun run docs:build
• git diff --check

CI for this push is pending: https://github.com/zig-utils/zig-js/actions/runs/29200732250

The negative/fail-closed half of #36 now covers both deferred cell kinds called out by the parallel tracer. The remaining #36 work is to decide whether any generator/iterator state has a positive recoverable class under live no-GIL peers.

[chrisbbreuer]

Follow-up #36 request-buffer witness landed in zig-js@5b5f310d (test(oom): cover async generator recovery guard).

What changed:

• Added a focused no-GIL heap-cap recovery guard for async-generator request side stores.
• The test creates an async generator suspended on an unresolved await, queues later .next() / .return() requests, and asserts the request backing store is live before recovery is attempted.
• Allocation-failure recovery is then attempted while a peer is live; the expected fail-closed telemetry is pinned: attempt increases, collection count does not, round-limit aborts increase, and gc_par_deferred_rounds is observed.
• Updated the testing guide so Runtime: prove generator and iterator side-store heap-cap recovery under no-GIL #36's fail-closed coverage lists suspended generators, pending async-generator requests, and iterator helpers.

Validation:

• /Users/chris/.local/bin/zig build test -Dtest-filter="deferred async-generator requests" --summary all
• /Users/chris/.bun/bin/bun run docs:build
• git diff --check

CI for this push is pending: https://github.com/zig-utils/zig-js/actions/runs/29200819725

The negative/fail-closed witnesses now cover the deferred generator cell family called out by #36: suspended generator stack, pending async-generator requests, and iterator-helper state. Remaining work is the design/proof for any positive recoverable state under live no-GIL peers.

[chrisbbreuer]

Follow-up #36 handler-buffer witness landed in zig-js@8c137d53 (test(oom): cover generator handler recovery guard).

What changed:

• Added a focused no-GIL heap-cap recovery guard for generator handler storage.
• The test suspends a generator inside try/finally, proving GC-backed resumable handler/execution buffers are live before recovery is attempted.
• Allocation-failure recovery is attempted while a peer is live; it must fail closed with attempts/round-limit/deferred telemetry increasing and no collection reported.
• Updated the testing guide so Runtime: prove generator and iterator side-store heap-cap recovery under no-GIL #36's fail-closed coverage now explicitly lists suspended stack, resumable handler, pending async-generator requests, and iterator-helper state.

Validation:

• /Users/chris/.local/bin/zig build test -Dtest-filter="deferred generator handlers" --summary all
• /Users/chris/.bun/bin/bun run docs:build
• git diff --check

CI for this push is pending: https://github.com/zig-utils/zig-js/actions/runs/29200885804

This fills the generator handler-storage part of the #36 reproduction checklist while preserving the current fail-closed policy.

[chrisbbreuer]

Telemetry follow-up landed in 4b20837b: deferred generator/iterator blocked finishes now increment gc_par_deferred_aborts, the #36 fail-closed witnesses assert it, and midgc-profile prints it as def-abort. Current CI: https://github.com/zig-utils/zig-js/actions/runs/29201026809

[chrisbbreuer]

Routing note: the previous #36 head CI run for 4b20837b was cancelled by the newer #15 profiler push. The #36 code remains on main; use the combined-head CI run for 03092fb0 as the current full gate: https://github.com/zig-utils/zig-js/actions/runs/29201263212 (queued at the time of this note).

[chrisbbreuer]

Routing note: the combined-head CI run for 03092fb0 was superseded by the newer #13 workflow-support push. The #36 fail-closed generator/iterator recovery witnesses and deferred-abort telemetry remain on main; use current combined-head CI for 36c0159a: https://github.com/zig-utils/zig-js/actions/runs/29201371564 (queued at the time of this note).

[chrisbbreuer]

Routing note: the combined-head CI run for 36c0159a was cancelled by the newer #16 nursery-profile push. The #36 fail-closed recovery witnesses and deferred-abort telemetry remain on main; use current combined-head CI for b79f9774: https://github.com/zig-utils/zig-js/actions/runs/29201561685 (queued at the time of this note).

[chrisbbreuer]

Routing note: current combined-head CI moved with the #12 memory-model docs push. The #36 recovery witnesses and deferred-abort telemetry remain on main; use current combined-head CI for d0092f5e: https://github.com/zig-utils/zig-js/actions/runs/29201642565 (queued at the time of this note).

[chrisbbreuer]

Closed as the deliberate all-fail-closed policy after the combined main gate for d0092f5e passed: https://github.com/zig-utils/zig-js/actions/runs/29201642565

Evidence now covered by the green gate:

• suspended generator execution storage fails closed under live no-GIL peers;
• resumable generator handler buffers fail closed;
• pending async-generator request buffers fail closed;
• iterator-helper backing state fails closed;
• deferred-cell abort telemetry (gc_par_deferred_aborts / def-abort) remains separate from round-limit abort accounting;
• docs distinguish trace-sensitive locks from deferred generator/iterator tracing.

No positive recoverable generator/iterator class is sound today. If one appears later, it should be reopened/narrowed with proof that no deferred edges remain or that recovery can complete through a sound world-stopped path.