Spike: run Cloudflare Worker Shell and Worker JavaScript natively in Deno

[taras]

Story

As an Executable.md host author, I want to know how much of Cloudflare Computer's Worker Shell and Worker JavaScript execution models can run over the Deno-local DOFS Workspace without workerd, so xmd workflow can optionally add useful imperative execution without making arbitrary native commands a prerequisite for durable workflows.

Context

#349 / PR #350 proves that Deno can own Cloudflare's SQLite-backed DOFS filesystem directly. Its current limitation is execution: arbitrary native subprocesses cannot coherently access the SQLite filesystem without FUSE, materialization, or a container.

#347 / PR #348 proves the other end of the tradeoff: bundled workerd runs Cloudflare Computer's real Worker Shell and Worker JavaScript backends over the same Workspace, but adds a second runtime, a supervised process, platform constraints, and substantial artifact size.

Cloudflare's Worker Shell is implemented with the JavaScript just-bash interpreter and a Workspace filesystem adapter. Its portable core may be reusable in Deno even though the published backend is hosted by a Dynamic Worker and imports cloudflare:workers.

Worker JavaScript is more coupled to Cloudflare's runtime. It expects a WorkspaceRuntimeLoader, Dynamic Worker isolation, Workers RPC, module maps, streaming, and disposal-based cancellation. Deno has Workers, but not that loader or RPC environment.

This spike determines what can be reused and what must be replaced. Native in Deno means that the proof runs without workerd, Wrangler, Docker, or another JavaScript runtime. A Deno Worker isolate is allowed.

Product boundary

xmd workflow is intentionally a constrained durable environment. It does not need arbitrary native commands to be viable.

The primary workflow operations are declarative components such as <Dir>, <File>, <Branch>, and <Commit>. They execute through dedicated contextual capabilities against the durable Workspace; they are not translated into shell commands. An operation is not journaled as successful until the installed capability has completed its state change.

The same components work in xmd run through ordinary host capabilities. That mode still guarantees operation-level correctness, but it does not promise Workspace retention, reattachment, or continuation after interruption. Host filesystem persistence is incidental rather than an XMD restoration guarantee.

Worker Shell and Worker JavaScript are optional imperative extensions layered over that declarative core. Limited or rejected results for either backend do not reject the Deno-local DOFS topology. An unsupported operation in xmd workflow fails explicitly; it never silently falls back to an unrelated host filesystem or process.

Questions

1. Which constrained Worker Shell or Worker JavaScript capabilities can safely and usefully be layered over the Deno-local DOFS Workspace?
2. Should each backend be included initially, deferred, or omitted?
3. For use cases that require greater imperative compatibility, does bundled workerd remain an optional execution topology?

The answer may be adopt, limit, or reject, independently for each backend. Full API.Process compatibility and arbitrary native subprocess execution are not success criteria.

Slice 1: Worker Shell

• Pin the same Cloudflare Computer source/version used by Spike: host Cloudflare DOFS directly in Deno with SQLite and FUSE #349 and record provenance for every reused file.
• Instantiate the portable just-bash shell engine against the Deno-local DOFS filesystem without WorkerEntrypoint, Workers RPC, or a Dynamic Worker loader.
• Prefer a Deno Worker boundary if the filesystem capability can cross it without materializing the Workspace.
• Demonstrate cwd, environment variables, pipelines, redirection, exit status, stdout/stderr streaming, and cancellation against the Workspace.
• Prove that an API.Fs write is immediately visible to shell execution and that a shell write is immediately visible through API.Fs, including after a full host restart.
• Record the supported command set. Distinguish shell built-ins/emulated commands and registered commands from arbitrary native executables.
• Test the security boundary. Determine whether untrusted shell code can access Deno, host files, network, environment secrets, or unrestricted module loading outside the installed capabilities.
• Determine whether Cloudflare's defense-in-depth mechanism works in Deno or must be replaced.

Slice 2: Worker JavaScript

• Identify the smallest WorkspaceRuntimeLoader-equivalent boundary needed to run a module graph in a Deno Worker.
• Demonstrate at least an entry module plus one dependency, filesystem capability calls against DOFS, stdout/stderr, a returned result, thrown errors, timeout, and cancellation.
• Keep arbitrary user modules out of the host isolate.
• Record how modules are supplied: in-memory graph, generated URLs, or temporary materialization. If materialization is required, measure what coherence and cleanup guarantees it changes.
• Compare Deno behavior with Cloudflare's backend for module access, capability transfer, streams, cancellation, compatibility settings, and retained execution state.
• If a faithful loader is not viable, reduce the experiment to the smallest failing proof and name the missing Deno or Cloudflare primitive precisely. Do not recreate Cloudflare's runtime merely to force a positive result.

Required evidence

• A self-contained, reproducible spike excluded from production package and verification scopes.
• A compiled-artifact test proving that the selected paths do not depend on an installed workerd, Node, Wrangler, or Docker runtime.
• Restart-persistence and identity-isolation tests using the DOFS database as the only filesystem source of truth.
• Cancellation tests that show whether execution stops and whether committed filesystem changes remain coherent.
• A compatibility matrix for Worker Shell and Worker JavaScript covering execution, isolation, filesystem coherence, streaming, cancellation, retention/reattachment, platform support, artifact size, and startup/operation latency.
• A source-reuse assessment covering package imports, vendoring or upstream exports, licensing, pins, and the maintenance delta from Cloudflare Computer.
• A recommendation for Define the local Workspace host topology #346 that separates the viability of the Deno-local declarative Workspace from the optional imperative backends, classifies each backend as include, defer, or omit, and states which optional use cases still require bundled workerd.

Acceptance

• The proof runs from one documented command on a prepared worktree.
• Worker Shell either executes directly over DOFS with the demonstrated contract above, or the evidence identifies a concrete blocker.
• Worker JavaScript either runs in a Deno Worker with the demonstrated capability bridge, or the evidence identifies a concrete blocker.
• The result states exactly which constrained execution semantics the Deno-local provider can claim and which operations fail as unsupported.
• The result does not describe Worker Shell as arbitrary native command execution.
• A failure or limitation in an imperative backend is not used to reject Deno-local DOFS for the declarative durable workflow core.
• Security and isolation findings are measured rather than inferred.
• The comparison is added to the evidence from Spike: bundle workerd into the compiled XMD host #347 and Spike: host Cloudflare DOFS directly in Deno with SQLite and FUSE #349, and Define the local Workspace host topology #346 receives the resulting layered topology recommendation.

Non-goals

• Production <Workspace> or provider implementation.
• Implementing <Dir>, <File>, <Branch>, <Commit>, or other declarative components.
• Full compatibility with every Cloudflare Computer command, module feature, or execution backend.
• Running arbitrary native executables against DOFS.
• FUSE, container, or materialization-cache implementation beyond measuring any requirement exposed by the JavaScript loader.
• A new public XMD process API.
• Hosted Cloudflare deployment or transport.

Dependencies

• Follow-up evidence to Spike: host Cloudflare DOFS directly in Deno with SQLite and FUSE #349 / PR 🔬 Spike: host Cloudflare DOFS directly in Deno with SQLite (#349) #350.
• Compared with Spike: bundle workerd into the compiled XMD host #347 / PR 🔬 Spike: bundle workerd into the compiled XMD host (#347) #348.
• Informs the layered topology decision in Define the local Workspace host topology #346 and the first provider in Quest: Resume workflows with their Workspace, repositories, and Agent state #218.
• End goal: ✨ Compose and certify the supervised adversarial implementation workflow #181.

[taras]

Spike complete — PR #353 (spikes/351-worker-backends/, runs via deno task spike:351), stacked on #350. Both backends run natively over the Deno-local DOFS Workspace from one compiled artifact — no workerd, Wrangler, Docker, or second JavaScript runtime. Full detail in evidence/EVIDENCE.md; probe ledgers with every command and escape attempt in evidence/probes/.

Worker Shell — include initially, scoped

Cloudflare's WorkspaceFsAdapter is reused byte-identical (sha1 c9ec5356…, zero cloudflare:* imports); the WorkerEntrypoint, Workers-RPC, and Dynamic-Worker glue exist only to reach a filesystem across an isolate boundary and simply disappear when the workspace is in-process. The only new code is a ~60-line shim for the three stub-only methods (exists/statOrNull/lstatOrNull).

• Coherence proven both ways, across restarts — a host API.Fs write is visible to shell execution and a shell write is visible through API.Fs, each in a separate process against the same database. Pipelines, redirection, exit status, env, cwd, stdout/stderr separation and error cases all behave. [test]
• Command set: 83 registered built-ins. This is an interpreter, not a process launcher — no PATH resolution, and /bin/sh, id, uname are all command not found [test]. It must never be described as arbitrary native command execution.
• Security measured, not inferred: canaries planted, ~50 escape attempts, zero leaks. /etc/passwd, ../../.. traversal, find /, grep -r / all resolve inside DOFS only; writes to /tmp and /etc land in the DOFS namespace with no host file created; the environment is fabricated; the QuickJS guest has no Deno, a blocked Function constructor, and a workspace-scoped require("fs"); network is off by default and, when enabled, enforces an allowlist plus private/loopback rejection plus DNS-rebinding pinning.
• Defense-in-depth: just-bash's own DefenseInDepthBox must be disabled under Deno (on by default, hard-fails Module._load) — the same one-line opt-out Cloudflare already ships for workerd. Containment holds without it, because it is architectural: injected filesystem, fabricated environment, no launcher.
• The one real hazard is availability, not capability: a 6.4 s CPU-bound loop starved the host event loop to 0 of 127 expected ticks — the abort timer never fired, so AbortSignal only works where the interpreter already awaits. Recommendation: run it in a Deno Worker — the adapter tolerates async RPC unmodified (10/10 parity incl. symlinks, chmod, sed -i, grep -r), 39.6 ms boot, ~0.04 ms/call, and terminate() reclaims an otherwise unrecoverable loop.

Worker JavaScript — defer

The smallest WorkspaceRuntimeLoader equivalent is an in-memory module graph supplied as blob URLs with each specifier rewritten to the dependency's absolute blob URL, in topological order, run in a Deno Worker with the filesystem capability crossing as async postMessage RPC.

• Module supply needs no materialization. Relative imports do not resolve from a blob: or data: base (invalid URL: relative URL with a cannot-be-a-base base); rewriting fixes it and survives deno compile. Static import maps are inherited by workers but there is no per-Worker importMap option. Temp-dir materialization was never needed, so its coherence and cleanup costs are avoided rather than measured.
• Contract works [test]: entry + dependency, DOFS capability access, stdout/stderr separation, returned value, thrown error (exit 1), timeout, cancellation; the isolate's write is committed and readable by a later process. Output is live-streamed (frames at 6/410/812 ms). Cold spawn→exit 13 ms; capability round-trip 0.065 ms.
• Isolation holds only in the compiled artifact — permissions: "none" governs Deno ops, not the module loader: under deno run a locked worker still executes import("jsr:…")/import("npm:…"), arbitrary code plus egress that net: false does not stop; import: false is inert. deno compile's frozen graph closes it. A deno run host must never be presented as a sandbox.
• Missing primitive, named: Worker.terminate() cannot preempt CPU-spinning user JS — the timeout outcome is right but the thread runs on and blocks process exit. Deno needs a CPU/wall budget on new Worker(...), or a terminate() that interrupts running JavaScript; Cloudflare's limits.cpuMs has no equivalent. The faithful workaround is an out-of-process worker that can be SIGKILLed, forfeiting the 13 ms cold start. Also not expressible: compatibilityDate/compatibilityFlags and retained execution (getExec/resume).

Explicit refusal, never fallback

An operation the host does not install fails where it is called, at both refusal points — unsupported filesystem op: <name> for something the runner exposes but the host withheld, and not a function for anything outside the capability surface [test]. The isolate has no path to host execution, so there is nothing to fall back to.

API.Process semantics a Deno-local provider could claim

Claimable: run a command string in a workspace-scoped shell; supply cwd and environment; receive stdout, stderr, exit status; run a JavaScript module graph with a workspace capability and a returned value; refuse unsupported operations explicitly. Not claimable: arbitrary native execution, preemptive cancellation of CPU-bound work, live incremental shell streaming, retained/reattachable executions, or POSIX process semantics.

Artifact 139.7 MB carrying both backends plus DOFS, versus 191 MB for the workerd host (#347). Upstream defects found that affect Cloudflare equally: relative symlinks are broken in @cloudflare/dofs; a redirect to a missing parent throws out of bash.exec(); ls -l reports a stale mode.

Recommendation posted to #346.

[taras]

Spike complete. PR #353 preserves the Worker Shell and Worker JavaScript proofs. The decision is to defer Worker JavaScript and admit Worker Shell only if the remaining transaction proof in #357 succeeds.