Sandbox exec is uncapped and blocks the event loop, stalling all tenants

[Hazzng]

Summary

Sandbox exec has no file-size cap of any kind, and ordinary shell commands over modest files block the Node event loop for seconds — stalling every other tenant on the replica, because it is one process and one loop.

This is worse than the write API, which at least has MAX_FILE_WRITE_BYTES.

Measured stalls (probe = tiny GET on a different sandbox, different owner)

Script	1 MiB	8 MiB	16 MiB	49 MiB
sed s///g	85 ms	706 ms	2238 ms	—
tr a-m n-z	—	—	1135 ms	5563 ms
wc -l	40 ms	335 ms	623 ms	—
cat f > g	—	—	554 ms	—

For comparison, the capped HTTP paths at the 50 MiB limit: PATCH 254 ms, PUT 158 ms.

CPU attribution (CDP profiler, sed over 49 MiB): 62.8% is garbage collection, from allocation pressure in just-bash's string-building text utilities — not CPU work. Yielding will not help; the allocations have to leave the main heap.

This is not only a latency problem — it breaks another tenant's correctness

src/redis/client.ts:40 sets commandTimeout: 2000. A stall past 2 s times out in-flight Redis commands belonging to other sandboxes. Observed in the log during an exec run:

{"event":"rw_lock_writer_release_error","key":"vfs:default:rwlock:{d6dab544-...}:writer","error":"Command timed out"}

d6dab544 is an unrelated tenant's sandbox. Thresholds: sed ~14 MiB, tr ~28 MiB, wc ~50 MiB.

Cross-tenant impact (mixed workload during one large operation)

	p50	p95	p99	mean
Baseline	3.5 ms	64.5 ms	102.7 ms	14 ms
During	3.3 ms	270 ms	307 ms	47 ms

p95 4.2x, p99 3.0x — p50 unchanged, so an average-latency dashboard shows nothing.

Related gap: bulk write is 2.5x wider than the per-file cap

MAX_BULK_WRITE_BYTES defaults to 128 MiB (routes/files.ts:68) against a 50 MiB per-file cap, and POST /writeFiles carries no writeBodyLimit middleware. A 120 MiB bulk write stalled the loop 669 ms. (The per-entry cap was closed in aa60a5a; the total and the missing body limit were not.)

Monitoring will not catch any of this

• event_loop_lag p50/p99 never move — DEFAULT_RESOLUTION_MS = 20 puts a 20 ms floor on the histogram, so one 2.2 s stall in a 10 s window is below p99. Only maxMs responds.
• eventLoopLagSnapshot() has zero non-test callers — not wired to any route, /readyz, or metrics endpoint. The only egress is a console.log.
• heartbeat_gap compares against REDIS_EXEC_LOCK_RENEW_MS (20 s) so it can never fire below a 20 s stall.

Options

1. Cap it — a file-size ceiling inside SqlFs so a script cannot read/write past a bound. Cheap, bounds the currently unbounded surface, costs sandbox capability.
2. Move bash.exec to a worker thread — the only thing that actually fixes it, since the cost is GC. Large effort; JUST_BASH_DEFENSE_IN_DEPTH monkey-patching and the IFileSystem bridge both need rethinking across the boundary. Note the existing worker-bridge plan covers the python3/js-exec SAB transport, a different limit, so it does not cover this.
3. Admission control — a semaphore on concurrent large operations, mirroring MAX_CONCURRENT_PYTHON. Does not fix head-of-line blocking but bounds the degraded window.
4. Alerting — export maxMs as a gauge and alert on it; add a threshold at the 2 s Redis commandTimeout, the point where stalls become correctness bugs.

Provenance

Found during pre-merge load testing of #162. Not introduced by that PR.