Spike: make Worker Shell effects transactional with the workflow journal

[taras]

Story

As an Executable.md workflow host implementor, I want one Worker Shell invocation to share the WorkflowRun's SQLite effect transaction, so the first production release can include a useful interpreted shell without weakening the rule that a Workspace mutation and its journal result publish atomically.

Context

#351 / PR #353 proves that Cloudflare Computer's Worker Shell model runs natively in Deno over the #349 DOFS Workspace:

• just-bash is the interpreter;
• Cloudflare's WorkspaceFsAdapter supplies filesystem and environment behavior;
• a Deno Worker prevents CPU-bound shell code from starving the XMD host;
• the interpreter has no host PATH or native-execution route; and
• the measured containment suite produced no host filesystem escape.

That spike proves containment, availability isolation and cross-operation DOFS coherence. It does not prove the workflow durability boundary selected after the spike:

one expansion -> one effect -> one SQLite transaction

In PR #353, filesystem operations commit independently. Production Worker Shell can ship in xmd workflow only if every filesystem request in one invocation participates in the same operation-scoped transaction or unpublished copy-on-write root as its journal result.

Product contract under test

“Worker Shell” names the Workspace-scoped capability. just-bash is its initial engine, not a promise of native Bash or a permanent public engine choice.

XMD host
└── effect transaction + journal
 └── filesystem RPC boundary
 └── Deno Worker
 └── just-bash
 └── Cloudflare WorkspaceFsAdapter

The shell has no native executable, host PATH, writable FUSE or workerd dependency.

Successful execution

BEGIN
 shell filesystem RPCs mutate one transaction
 append the successful shell result
COMMIT

The next Workspace effect observes the committed mutations and journal result together.

Known failed execution

A nonzero exit, thrown error, timeout, cancellation or Deno Worker termination rolls filesystem mutations back to an effect-local savepoint, then records the failed result in the same outer transaction:

BEGIN
 SAVEPOINT shell_mutations
 shell filesystem RPCs
 ROLLBACK TO shell_mutations
 append the failed shell result
COMMIT

For example, this leaves no result.txt under xmd workflow:

echo partial > result.txt
false

Ordinary xmd run keeps its host-shell behavior and is not changed by this contract.

Host interruption

If the XMD host disappears with the SQLite transaction open, SQLite recovery rolls back the whole transaction. Restart finds neither published filesystem mutations nor a journal result, so replay may execute the effect again under the same identity.

Proof slices

1. Operation transaction — extend the Deno-local DOFS storage boundary so one caller owns an explicit transaction or unpublished root across multiple filesystem operations and can atomically append the journal result.
2. Worker routing — route every Workspace filesystem RPC from one Deno Worker invocation to that exact transaction. Reject calls with a missing, completed, cancelled or foreign effect identity.
3. Success publication — prove files, deletes, renames, modes, symlinks and the shell result become visible together after a zero exit.
4. Failure rollback — prove nonzero exit, interpreter error, timeout, explicit cancellation and Worker.terminate() publish no filesystem mutation while retaining the failed result.
5. Host-crash recovery — kill the compiled proof after at least one shell write but before result publication, restart against the same database and prove both mutation and result are absent.
6. Isolation regression — retain PR 🔬 Spike: run Cloudflare Worker Shell and Worker JavaScript natively in Deno (#351) #353's host-file escape, missing-native-command, fabricated-environment, network refusal and CPU-spin probes at the new boundary.
7. Replay shape — prove a committed result restores without starting a Worker and an uncommitted crash reruns against the pre-effect Workspace root.

Acceptance

• One shell invocation has one stable workflow effect identity and one operation-scoped SQLite transaction.
• All filesystem adapter calls for that invocation are fenced to its transaction.
• Successful mutations and the filtered journal result commit atomically.
• Every known failure path rolls mutations back and records the failure without opening a second top-level effect transaction.
• A host crash rolls the entire open transaction back on restart.
• A stale or late Worker message cannot mutate a completed, cancelled or different effect.
• The Deno Worker still preempts a CPU-bound interpreter without blocking the host event loop.
• The capability still exposes no native execution, host PATH, host filesystem or default network route.
• The proof runs from one documented command and records reproducible evidence.
• The issue reports a binary verdict: include Worker Shell in the first production release, or defer it without weakening workflow durability.

Intentionally excluded

• Production <Workspace> or shell-component implementation.
• Worker JavaScript.
• Native subprocesses or writable FUSE.
• Bundled workerd or Cloudflare Containers.
• External side-effect atomicity; Git push, forge APIs and Agent providers reconcile separately.
• Changing ordinary xmd run shell semantics.

Dependencies

• Spike: host Cloudflare DOFS directly in Deno with SQLite and FUSE #349 / PR 🔬 Spike: host Cloudflare DOFS directly in Deno with SQLite (#349) #350 supplies the Deno-local DOFS/SQLite proof.
• Spike: run Cloudflare Worker Shell and Worker JavaScript natively in Deno #351 / PR 🔬 Spike: run Cloudflare Worker Shell and Worker JavaScript natively in Deno (#351) #353 supplies the contained Deno Worker Shell proof.
• Define the local Workspace host topology #346 records the local topology selection.
• Quest: Resume workflows with their Workspace, repositories, and Agent state #218 owns the production provider-backed Workflow Workspace contract.
• End goal: ✨ Compose and certify the supervised adversarial implementation workflow #181.

[taras]

Verdict: INCLUDE Worker Shell

One just-bash Worker Shell invocation can satisfy one expansion → one effect → one SQLite transaction without weakening the workflow durability contract.

The proven boundary is:

BEGIN IMMEDIATE
 SAVEPOINT shell_mutations
 transaction-bound Worker filesystem RPCs
 success: RELEASE + append already-filtered ok result
 failure: ROLLBACK TO + RELEASE + append already-filtered failed result
COMMIT

A separate compiled host was killed with SIGKILL after its write RPC reached the open transaction and before result publication. Restart found neither the mutation nor a journal row. Missing, foreign, stale, completed, and cancelled Worker routes were all fenced.

All 11 focused probes pass: success atomicity; nonzero exit; interpreter error; timeout; explicit cancellation; Worker.terminate(); CPU-bound preemption; real host crash/recovery; identity fencing; isolation regression; and replay without starting a Worker.

• PR: 🔬 Spike: make Worker Shell effects transactional (#357) #362
• Evidence: https://github.com/taras/executable.md/blob/spike/issue-357-worker-shell-transactions/spikes/357-worker-shell-transactions/evidence/EVIDENCE.md
• Raw output: https://github.com/taras/executable.md/blob/spike/issue-357-worker-shell-transactions/spikes/357-worker-shell-transactions/evidence/raw-proof-output.txt
• Provenance delta: https://github.com/taras/executable.md/blob/spike/issue-357-worker-shell-transactions/spikes/357-worker-shell-transactions/evidence/UPSTREAM-DELTA.md

Production #218 must preserve forceful Worker termination for CPU-bound code, one authoritative host-owned DOFS connection per workflow database, just-bash-only isolation, and the existing prefiltered journal boundary.

The issue remains open for review.

[taras]

Verdict: INCLUDE

The spike is complete. PR #362 proves that one Worker Shell invocation satisfies one expansion → one effect → one SQLite transaction, including success publication, rollback with one failed result, host-crash recovery, stale-message fencing, committed replay with zero Worker starts, and forceful preemption of CPU-bound just-bash.

PR #358 merged the resulting architecture and observable contract. Production integration is tracked by #363 and must preserve:

• one authoritative host-owned DOFS connection per workflow database;
• serialized caller-owned effect transactions and nested savepoints;
• already-filtered journal insertion at the same transaction boundary;
• effect-ID plus per-invocation-token fencing;
• forceful Worker termination with pinned patch/fork provenance until upstream support exists; and
• no native execution, host PATH, host filesystem, inherited environment, or default network authority.

PR #362 remains unmerged experimental evidence. Closing this spike does not make its source production code and does not authorize merging the evidence PR.