fix(compute): recover Error-phase sandboxes on gateway startup by r3v5 · Pull Request #2269 · NVIDIA/OpenShell

r3v5 · 2026-07-14T17:56:31Z

Summary

After a Podman/Docker machine restart, sandbox containers exit with SIGTERM (code 143) but remain on disk. The gateway marked them Error and on next startup the resume sweep skipped Error-phase sandboxes entirely — leaving them stuck until the user deleted and recreated them, losing state.
resume_persisted_sandboxes() now attempts Error-phase sandboxes: if the container still exists it is restarted and the sandbox transitions back to Provisioning → Ready; if the container is gone or the driver fails, the sandbox stays in Error without overwriting the original error reason.
Adds resume_sandbox() to the Podman driver and wires StartupResume for PodmanComputeDriver so the resume sweep runs on Podman-backed gateways (previously Docker-only).

Related Issue

Closes#2179

Changes

crates/openshell-driver-podman/src/driver.rs — Added resume_sandbox() method that inspects container state and calls start_container if the container exists but is not running.
crates/openshell-server/src/compute/mod.rs:
- Modified phase filter in resume_persisted_sandboxes() to only skip Deleting (not Error)
- Added clear_sandbox_error() method (mirrors mark_sandbox_error()) to reset phase to Provisioning and set Ready condition to "Resumed"
- Guarded Ok(false) and Err arms to avoid overwriting existing error state on already-Error sandboxes
- Implemented StartupResume for PodmanComputeDriver and wired it into new_podman()
- Added recovered counter to summary log line for observability
- Removed unused sandbox_phase_should_be_running() function
- Updated doc comments to reflect new behavior

Testing

Unit tests (3 new + 1 updated)

resume_persisted_sandboxes_recovers_error_phase_when_container_exists — Ok(true) → phase becomes Provisioning, Ready condition reason = "Resumed"
resume_persisted_sandboxes_leaves_error_when_container_missing — Ok(false) → phase stays Error, no overwrite
resume_persisted_sandboxes_leaves_error_when_resume_fails — Err → phase stays Error, no overwrite
Updated resume_persisted_sandboxes_resumes_running_phases to expect Error-phase sandbox in called_ids

All 1032 tests pass across openshell-server and openshell-driver-podman. Clippy clean.

Manual verification

Full end-to-end reproduction of issue #2179:

Created sandbox with Vertex AI provider: openshell sandbox create --name my-podman-sandbox --provider vertex-prod
Verified sandbox Ready and container Up (healthy)
Stopped Podman machine: podman machine stop
Restarted Podman machine: podman machine start
Verified container Exited (143) and sandbox stuck in Error (before fix — screenshot 1)
Restarted gateway with fix — logs show Resumed sandbox ... phase=Error recovered=true and Sandbox resume sweep complete resumed=1 recovered=1 (screenshot 2)
Verified sandbox recovered to Ready and container Up (healthy) (screenshot 2)
Verified sandbox usable via openshell term TUI (screenshot 3)

Screenshot 1 — Before fix: sandbox stuck in Error after Podman machine restart

Screenshot 2 — After fix: gateway restart recovers sandbox to Ready

Screenshot 3 — Sandbox usable in OpenShell TUI after recovery

Checklist

Follows Conventional Commits format
Signed off (DCO)
Unit tests added and passing
Pre-commit checks passing
Manual end-to-end verification complete
No proto changes, no new gRPC RPCs, no CLI changes

🤖 Generated with Claude Code

copy-pr-bot · 2026-07-14T17:56:34Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

r3v5 · 2026-07-14T17:56:39Z

Manual Verification Screenshots

Screenshot 1 — Before fix: sandbox stuck in Error after Podman machine restart (exit code 143)

Screenshot 2 — After fix: gateway restart recovers sandbox, logs show recovered=true, sandbox Ready

Screenshot 3 — Sandbox usable in OpenShell TUI after recovery

r3v5 · 2026-07-14T18:08:48Z

I have read the DCO document and I hereby sign the DCO.

r3v5 · 2026-07-14T18:08:53Z

recheck

r3v5

PR Review — Claude Code (Opus 4.6)

Overview

After Podman/Docker machine restart, sandbox containers exit with SIGTERM but remain on disk. Gateway previously skipped Error-phase sandboxes during resume sweep, leaving them stuck. This PR makes the resume sweep attempt Error-phase sandboxes — restarting containers that still exist, leaving Error untouched if the container is gone or the driver fails. Also wires StartupResume for Podman (was Docker-only).

Key Design Decisions

Include Error in resume sweep, only skip Deleting (crates/openshell-server/src/compute/mod.rs:787) — Replaces sandbox_phase_should_be_running() filter with simple phase == Deleting check. Error-phase sandboxes now get a recovery attempt instead of being permanently stuck.
Guard against overwriting existing error state (mod.rs:810-828) — When an already-Error sandbox fails resume (Ok(false) or Err), the original error reason is preserved. Only Ok(true) clears error state. Prevents losing diagnostic info.
clear_sandbox_error() resets to Provisioning, not Ready (mod.rs:895-930) — Recovered sandboxes go to Provisioning with Ready condition reason "Resumed". Watch loop then drives them to Ready naturally, same path as fresh sandboxes.
Podman resume_sandbox() uses inspect-then-start (crates/openshell-driver-podman/src/driver.rs:695-712) — Inspects container first; if running returns Ok(true) without restart. NotFound mapped to Ok(false) at both inspect and start points for race safety.

Notable Code

crates/openshell-driver-podman/src/driver.rs:695:

pubasyncfnresume_sandbox(&self,sandbox_name:&str) -> Result<bool,ComputeDriverError>{let name = container::container_name(sandbox_name);let inspect = matchself.client.inspect_container(&name).await{Ok(i) => i,Err(PodmanApiError::NotFound(_)) => returnOk(false),Err(e) => returnErr(ComputeDriverError::from(e)),};if inspect.state.running{returnOk(true);}matchself.client.start_container(&name).await{Ok(()) => Ok(true),Err(PodmanApiError::NotFound(_)) => Ok(false),Err(e) => Err(ComputeDriverError::from(e)),}}

Potential Concerns

TOCTOU between inspect and start — Container could be removed between inspect_container and start_container. Handled by catching NotFound on start, so no actual bug, but worth noting the race is accounted for.
update_message_cas with generation 0 in clear_sandbox_error() (mod.rs:904) — Passing 0 as CAS generation means no optimistic concurrency check. Safe here since this runs at startup before watchers spawn (per doc comment), but fragile if ever called later. Matches existing mark_sandbox_error() pattern though.

Verdict

Solid fix. Tests cover all three outcomes (container exists, gone, driver error). Manual E2E verification thorough. Clean removal of now-unnecessary sandbox_phase_should_be_running(). No blocking concerns.

r3v5 · 2026-07-14T18:34:05Z

-/// silently revived. `Unspecified` is included because it is the proto
-/// default value; persisted rows with that value should be reconciled
-/// from the live driver state rather than skipped forever.
-fn sandbox_phase_should_be_running(phase: SandboxPhase) -> bool {


Unused anymore after the changes.

maxamillion · 2026-07-14T23:01:50Z

@r3v5 This is great, thank you for looking into this and proposing a fix.

Two points I was hoping to get some clarification on or possibly some changes.

As I understand it, the current logic retries every sandbox in Error, which could revive containers that failed for legitimate terminal reasons and replace the original error with Resumed. Would it make sense to restrict recovery to the expected runtime-restart case, such as SIGTERM/exit 143 or an explicitly recoverable stored reason?
It looks as though clear_sandbox_error() swallows store/CAS failures, but the caller still logs recovered=true and increments the recovery count. Ideally we could count/log recovery only after the state transition succeeds in order to not mask failures unintentionally.

r3v5 · 2026-07-16T13:44:50Z

 /// Container/pod label carrying the sandbox namespace.
 pub const LABEL_SANDBOX_NAMESPACE: &str = "openshell.ai/sandbox-namespace";

+// ---------------------------------------------------------------------------


Made constants public and reusable.

r3v5 · 2026-07-16T13:45:43Z

 }
 ContainerSummaryStateEnum::EXITED => {
- ("False", "ContainerExited", "Container exited", false)
+ ("False", openshell_core::driver_utils::CONDITION_EXITED, "Container exited", false)


Use public constant here

r3v5 · 2026-07-16T14:00:10Z

@r3v5 This is great, thank you for looking into this and proposing a fix.
Two points I was hoping to get some clarification on or possibly some changes.
As I understand it, the current logic retries every sandbox in Error, which could revive containers that failed for legitimate terminal reasons and replace the original error with Resumed. Would it make sense to restrict recovery to the expected runtime-restart case, such as SIGTERM/exit 143 or an explicitly recoverable stored reason?
It looks as though clear_sandbox_error() swallows store/CAS failures, but the caller still logs recovered=true and increments the recovery count. Ideally we could count/log recovery only after the state transition succeeds in order to not mask failures unintentionally.

Hi, @maxamillion ! Thanks for the review! I have addressed your two points in the new commit:

Recovery is now restricted to ContainerExited and ContainerStopped reasons only — sandboxes that failed for terminal reasons (BackendResourceMissing, ResumeFailed, etc.) are skipped entirely. Added a test for this case.
clear_recoverable_error() now returns bool — the caller only logs recovered=true and increments the counter recovered after the CAS write actually succeeds.

r3v5 · 2026-07-23T13:19:42Z

Hey @elezar ! Could you please take a look on my PR? Thanks!

r3v5 · 2026-08-20T09:07:32Z

Rebased PR from main and solved merge conflicts.

r3v5 · 2026-08-20T09:58:02Z

@pimlock @krishicks can I get some eyes?

Sandboxes whose container exited on its own (e.g. SIGTERM from a Podman or Docker machine restart) were left stuck in the Error phase even though the container could be restarted. The startup sweep skipped every Error-phase sandbox unconditionally. Include Error-phase sandboxes in the startup sweep when their Ready condition reason marks a container exit or stop. If the driver restarts the container, move the sandbox back to Provisioning with a Resumed condition; if the container is gone or the start fails, leave the Error state untouched. Genuine (non-container-exit) errors are still skipped. Container-exit restart is handled by the drivers' existing idempotent start_sandbox path, which already restarts stopped/exited containers. The ContainerExited/ContainerStopped condition reasons are promoted to shared constants in openshell-core so the gateway and drivers agree on the recovery signal. FixesNVIDIA#2179 Signed-off-by: Ian Miller <milleryan2003@gmail.com>

r3v5 · 2026-08-24T19:19:47Z

Hi @johntmyers ! Can you PTAL?

github-actions · 2026-08-24T19:30:22Z

Label test:e2e applied, but pull-request/2269 does not exist yet. A maintainer needs to comment /ok to test 428b85924b0cb2aad3dba4e3055b55a59e6ddfc3 to mirror this PR. Once the mirror exists, re-apply the label or re-run Branch E2E Checks from the Actions tab.

johntmyers · 2026-08-24T19:30:45Z

/ok to test 428b859

johntmyers

gator-agent

PR Review Status

Thanks @maxamillion, I checked both concerns you raised. The recovery counter now advances only after the store transition succeeds, but the reason filter still groups ordinary application exits with machine/runtime restarts, so one blocking correctness issue remains.

Action required: distinguish runtime-restart recovery from generic container exits and add a regression test showing ordinary crashes remain terminal.

Blocking findings:

GATOR-428b8592-01: generic ContainerExited errors are restarted during gateway startup.

Carried findings:

None

Non-blocking suggestions:

None

Gator metadata

Validation: Project-valid concentrated fix for linked issue #2179; duplicate search found no competing issue or PR.
Docs: No Fern update needed because this corrects internal startup recovery without changing a CLI, API, configuration, or documented workflow.
Checks: Current-head Branch E2E Checks are queued/running; review feedback takes precedence.
E2E: test:e2e applied; /ok to test 428b85924b0cb2aad3dba4e3055b55a59e6ddfc3 accepted and current-head workflow run 32768589180 is active.
Head SHA: 428b85924b0cb2aad3dba4e3055b55a59e6ddfc3
Base SHA: de4c1fecf564cd16854447ec9c08659e7b64061b
Merge base SHA: de4c1fecf564cd16854447ec9c08659e7b64061b
Patch ID: 5451afaf080976211094feaf58264a271681a97b
Gator payload: 7
Review mode: initial
Previous reviewed SHA: none
Review budget exhausted: no
Maintainer decision required: no
Next state: gator:in-review

…ecovery Startup recovery treated every non-OOM container exit as recoverable, lumping application crashes and non-zero exits together with machine/daemon-restart signal kills under the shared `ContainerExited` Ready-condition reason. On the next gateway startup it restarted those crashed workloads and replaced the terminal error with `Resumed`, erasing the failure signal and repeatedly reviving crash-prone sandboxes. Introduce a distinct `ContainerRuntimeRestart` reason for containers terminated by an external signal (exit 137/143 = SIGKILL/SIGTERM), which is the signature of a machine/daemon restart. The Podman inspect-based classification emits it; the startup recovery gate now recovers only `ContainerRuntimeRestart` and `ContainerStopped`. Ordinary `ContainerExited` records stay terminal, so genuine failures keep their error signal. Because only the new reason is recoverable and older gateways never persisted it, exits recorded before this change also stay terminal after an upgrade. Add regression coverage: a generic `ContainerExited` error is not relaunched, a signal-kill classifies as `ContainerRuntimeRestart`, and the startup sweep recovers only the runtime-restart record. Signed-off-by: Ian Miller <milleryan2003@gmail.com>

r3v5 · 2026-08-25T11:55:18Z

Addressed the blocking finding GATOR-428b8592-01 in b58ecee.

Root cause: startup recovery treated every non-OOM exit as recoverable because Podman classified both application exits and signal kills under the shared ContainerExited reason.

Fix: added a distinct ContainerRuntimeRestart reason for external-signal terminations (exit 137/143 = SIGKILL/SIGTERM, the machine/daemon-restart signature). The recovery gate now recovers only ContainerRuntimeRestart and ContainerStopped; ordinary ContainerExited records stay terminal and keep their error signal. Since older gateways never persisted the new reason, pre-upgrade exits also stay terminal.

Coverage added:

condition_signal_kill_is_runtime_restart — 137/143 classify as ContainerRuntimeRestart (137/143 vs. condition_normal_exit for exit 1 → ContainerExited).
start_persisted_sandboxes_leaves_generic_container_exit_terminal — a ContainerExited crash is not relaunched and stays in Error.
start_persisted_sandboxes_starts_running_phases — the sweep recovers only the ContainerRuntimeRestart record; ContainerExited and BackendResourceMissing stay terminal.

johntmyers

gator-agent

PR Review Status

Thanks @r3v5. I checked your new runtime-restart reason, the recovery allowlist, and the regression coverage. GATOR-428b8592-01 is resolved for ordinary Podman exits, and I resolved its review thread. One related current-head blocker remains: Docker still records every exited container as ContainerExited, so the tightened allowlist can no longer recover Docker machine-restart victims.

Action required: classify Docker exit 137/143 records distinctly from ordinary exits and cover both recovery and terminal behavior.

Blocking findings:

GATOR-b58ecee3-01: Docker runtime-restart recovery is unreachable under the new reason gate.

Carried findings:

None

Gator metadata

Validation: Project-valid concentrated fix for linked issue #2179.
Docs: No Fern update needed because this remains an internal startup-recovery correction without a CLI, API, configuration, or documented-workflow change.
Checks: Current-head required checks are pending and test dispatch is deferred while review feedback remains.
E2E: test:e2e remains applied; current-head E2E has not yet been confirmed queued, running, or complete.
Head SHA: b58ecee398d26a49816edb74852a74c90cddb8f0
Base SHA: de4c1fecf564cd16854447ec9c08659e7b64061b
Merge base SHA: de4c1fecf564cd16854447ec9c08659e7b64061b
Patch ID: a0286570c067ddc08898e7be0fd7a431c24653e7
Gator payload: 7
Review mode: follow_up
Previous reviewed SHA: 428b85924b0cb2aad3dba4e3055b55a59e6ddfc3
Review budget exhausted: no
Maintainer decision required: no
Next state: gator:in-review

Mirror the Podman classification in the Docker driver so that externally signal-killed containers (exit 137/143, non-OOM) are persisted with the ContainerRuntimeRestart ready reason and become eligible for startup recovery, while ordinary exits and OOM kills stay terminal. current_snapshots now inspects EXITED containers to obtain the exit code and OOM flag, then applies the shared exit classification. Without the inspect step the list-summary path lacks an exit code, so the recovery gate could never fire for Docker. Adds unit tests covering signal-kill reclassification (137/143), ordinary exit staying terminal (exit 1), and OOM staying terminal despite exit 137. Signed-off-by: Ian Miller <milleryan2003@gmail.com>

r3v5 · 2026-08-25T12:16:28Z

Addressed GATOR-b58ecee3-01 (Docker restart recovery unreachable) in b778f01.

The Docker driver now mirrors the Podman classification. current_snapshots inspects EXITED containers to recover the exit code and OOM flag (the list-summary path has no exit code), then apply_docker_exit_classification reclassifies non-OOM signal kills (137/143) to the ContainerRuntimeRestart ready reason. Ordinary exits and OOM kills stay terminal (ContainerExited).

Verify: an exited container with code 143 and oom_killed=false persists ContainerRuntimeRestart and is recovered at startup; code 1 (or OOM+137) stays ContainerExited and terminal.

New unit tests: docker_signal_kill_reclassified_as_runtime_restart, docker_ordinary_exit_stays_terminal, docker_oom_kill_stays_terminal_despite_137. Full suite green (docker 116, server 405, core 1412; pre-commit clean).

johntmyers · 2026-08-25T12:37:47Z

/ok to test b778f01

johntmyers

gator-agent

PR Review Status

Thanks @r3v5. I checked the Docker exit inspection and classification you added at b778f01f, including the ordinary-exit and OOM safeguards and their regression coverage. GATOR-b58ecee3-01 is resolved, its review thread is closed, and no blocking findings remain.

Blocking findings:

No blocking findings remain

Carried findings:

None

Gator metadata

Validation: Project-valid concentrated fix for linked issue #2179.
Docs: No Fern update needed because this remains an internal startup-recovery correction without a CLI, API, configuration, or documented-workflow change.
Checks: Current-head mirror setup is pending; required Branch Checks, Helm Lint, and E2E have not yet been confirmed queued, running, or complete.
E2E: test:e2e is applied and /ok to test b778f01f568f747923c8c1b4cf1c252e6d4d80e0 was posted; mirror setup is pending before E2E dispatch can be confirmed.
Head SHA: b778f01f568f747923c8c1b4cf1c252e6d4d80e0
Base SHA: de4c1fecf564cd16854447ec9c08659e7b64061b
Merge base SHA: de4c1fecf564cd16854447ec9c08659e7b64061b
Patch ID: 23efc9c3b51e15bd31581e2e1173f3749ba6b9ed
Gator payload: 7
Review mode: follow_up
Previous reviewed SHA: b58ecee398d26a49816edb74852a74c90cddb8f0
Review budget exhausted: no
Maintainer decision required: no
Next state: gator:in-review

r3v5 requested review from a team, derekwaynecarr, maxamillion and mrunalp as code owners July 14, 2026 17:56

r3v5 force-pushed the fix-sandbox-errors-after-podman-machine-restarts branch from 594ab91 to 4150476Compare July 14, 2026 18:02

r3v5 commented Jul 14, 2026
View reviewed changes

r3v5 commented Jul 16, 2026
View reviewed changes

johntmyers assigned maxamillion Jul 21, 2026

r3v5 force-pushed the fix-sandbox-errors-after-podman-machine-restarts branch from a705b0e to 48d84b4Compare July 21, 2026 15:08

elezar mentioned this pull request Jul 22, 2026
refactor(compute): define driver shutdown and restart lifecycle contract #2417
Open
6 tasks

r3v5 force-pushed the fix-sandbox-errors-after-podman-machine-restarts branch from 48d84b4 to 0e27573Compare August 20, 2026 09:00

r3v5 requested a review from sjenning as a code owner August 20, 2026 09:00

r3v5 force-pushed the fix-sandbox-errors-after-podman-machine-restarts branch from 0e27573 to 5d6f952Compare August 20, 2026 09:06

r3v5 force-pushed the fix-sandbox-errors-after-podman-machine-restarts branch from 5d6f952 to bf0adaaCompare August 20, 2026 09:52

r3v5 force-pushed the fix-sandbox-errors-after-podman-machine-restarts branch from bf0adaa to bd649a6Compare August 21, 2026 16:17

r3v5 force-pushed the fix-sandbox-errors-after-podman-machine-restarts branch from bd649a6 to 428b859Compare August 21, 2026 16:29

johntmyers added the test:e2e Requires end-to-end coverage label Aug 24, 2026

johntmyers reviewed Aug 24, 2026
View reviewed changes

Comment threadcrates/openshell-server/src/compute/mod.rs

johntmyers added the gator:in-review Gator is reviewing or awaiting PR review feedback label Aug 24, 2026

r3v5 requested a review from johntmyers August 25, 2026 12:04

johntmyers reviewed Aug 25, 2026
View reviewed changes

Comment threadcrates/openshell-server/src/compute/mod.rs

r3v5 requested a review from johntmyers August 25, 2026 12:17

johntmyers reviewed Aug 25, 2026
View reviewed changes

Uh oh!

Conversation

r3v5 commented Jul 14, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Related Issue

Changes

Testing

Unit tests (3 new + 1 updated)

Manual verification

Screenshot 1 — Before fix: sandbox stuck in Error after Podman machine restart

Screenshot 2 — After fix: gateway restart recovers sandbox to Ready

Screenshot 3 — Sandbox usable in OpenShell TUI after recovery

Checklist

Uh oh!

copy-pr-botBot commented Jul 14, 2026

Uh oh!

r3v5 commented Jul 14, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Manual Verification Screenshots

Uh oh!

r3v5 commented Jul 14, 2026

Uh oh!

r3v5 commented Jul 14, 2026

Uh oh!

r3v5 left a comment

Choose a reason for hiding this comment

PR Review — Claude Code (Opus 4.6)

Overview

Key Design Decisions

Notable Code

Potential Concerns

Verdict

Uh oh!

r3v5Jul 14, 2026

Choose a reason for hiding this comment

Uh oh!

maxamillion commented Jul 14, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

r3v5Jul 16, 2026

Choose a reason for hiding this comment

Uh oh!

r3v5Jul 16, 2026

Choose a reason for hiding this comment

Uh oh!

r3v5 commented Jul 16, 2026

Uh oh!

r3v5 commented Jul 23, 2026

Uh oh!

r3v5 commented Aug 20, 2026

Uh oh!

r3v5 commented Aug 20, 2026

Uh oh!

r3v5 commented Aug 24, 2026

Uh oh!

github-actionsBot commented Aug 24, 2026

Uh oh!

johntmyers commented Aug 24, 2026

Uh oh!

johntmyers left a comment

Choose a reason for hiding this comment

PR Review Status

Uh oh!

Uh oh!

r3v5 commented Aug 25, 2026

Uh oh!

johntmyers left a comment

Choose a reason for hiding this comment

PR Review Status

Uh oh!

Uh oh!

r3v5 commented Aug 25, 2026

Uh oh!

johntmyers commented Aug 25, 2026

Uh oh!

johntmyers left a comment

Choose a reason for hiding this comment

PR Review Status

Uh oh!

r3v5 commented Jul 14, 2026•
edited
Loading

r3v5 commented Jul 14, 2026•
edited
Loading

maxamillion commented Jul 14, 2026•
edited
Loading