feat(sandbox): add main restart policy

[drew]

Summary

Add a gateway-owned restart policy for the canonical sandbox main process introduced by #2726. Sandboxes default to Never and may opt into Kubernetes-style OnFailure or Always replacement with durable exponential backoff.

This is the second PR in the stack and must be reviewed after #2726.

Related Issue

Relates to #2710.

Depends on #2726.

Changes

• Add Never, OnFailure, and Always restart policies plus the Restarting sandbox phase.
• Persist restart count, scheduling deadline, and main-process start time on sandbox status.
• Implement gateway-owned replacement, exponential backoff, HA-safe reconciliation, readiness recovery, and manual stop cancellation.
• Clear VM stop and terminal tombstones on explicit replacement start so a prior main exit cannot poison later driver recovery.
• Preserve the original main-process start time when the same supervisor instance reconnects; reset it only for a new instance.
• Keep native runtime restart disabled across Docker, Podman, Kubernetes, and VM drivers.
• Surface policy and restart status through the CLI, TUI, SDKs, published docs, and agent guidance.

Testing

• mise run pre-commit
• mise run test with an isolated system-gateway directory
• Focused VM tombstone and gateway reconnect stability-clock tests
• Docker sandbox lifecycle E2E: 6 passed, covering retained reconnect/history and Ready to Restarting to replacement Ready with workspace preservation
• Agent infrastructure inventory and cross-reference consistency checks

The unmodified host environment contains /etc/openshell/gateways/default, so the no-config completer test requires an isolated OPENSHELL_SYSTEM_GATEWAY_DIR; the full suite passes with that isolation.

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture and user documentation updated
• Related CLI and TUI agent guidance updated

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[github-actions]

🌿 Preview your docs:https://nvidia-preview-pr-2798.docs.buildwithfern.com/openshell

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[drew]

/ok to test a5f39e2

[sjenning]

Do we want the distinction between starting and restarting?

[drew]

no, i think sticking with starting makes more sense.

[sjenning]

review-github-pr skill with 5.6-sol

Potential Concerns

• P1 — Early replacement exits are discarded. After starting a replacement, status retains the previous instance ID and exit code. If the replacement exits before connecting its supervisor session, its new UUID is rejected as stale (compute/mod.rs:2747, :3180). This can incorrectly restart an OnFailure workload that exited successfully and prevents crash-loop accounting.
• P1 — HA stop/delete race. The controller releases its process-local guard before driver operations and starts unconditionally afterward (compute/mod.rs:2722-2761). Another gateway can concurrently stop or delete the sandbox, leaving persisted state Stopped while compute is running—or creating an orphan runtime.
• P2 — Readiness deadline starts too early. The 120-second watchdog is recorded before both driver stop and start (compute/mod.rs:2701). A slow Kubernetes termination can consume the deadline and trigger an immediate second replacement.
• Documentation drift. The CLI skills incorrectly imply OnFailure restarts after every exit; several existing docs still say all main-process exits are terminal. The architecture phase pseudocode also omits the new gateway Restarting override.
• Agent-infrastructure drift. The maintenance map indicates debug-openshell-cluster should cover the new distinct Restarting state and gateway-versus-native restart behavior, but it was not updated.
• TUI scope mismatch. The TUI styles the Restarting phase but does not display restart policy, count, deadline, or exit status, despite the PR description claiming restart status is surfaced there.