Skip to content

feat(server): add bundled docker compute driver - #888

Merged
drew merged 17 commits into
mainfrom
drew/creating-a-docker-driver-like-the-vm-driver
Apr 28, 2026
Merged

feat(server): add bundled docker compute driver#888
drew merged 17 commits into
mainfrom
drew/creating-a-docker-driver-like-the-vm-driver

Conversation

@drew

@drewdrew commented Apr 20, 2026•
edited
Loading

Copy link
Copy Markdown
Collaborator

Summary

Add a bundled Docker compute driver to the gateway on top of the supervisor-relay base. This lets the gateway provision sandboxes directly through the local Docker daemon without exposing sandbox ports or spawning a separate Docker driver binary.

Related Issue

N/A

Changes

  • add docker as a supported compute driver kind
  • add an in-process Docker backend in openshell-server using Bollard
  • wire Docker CLI and env config for supervisor and TLS bind mounts
  • keep Docker sandboxes as long-lived containers with no /sandbox volume
  • fix Docker review follow-ups for namespace-scoped discovery, inherited image CMD, and crash-looping sandbox error reporting
  • update architecture and support docs for the Docker backend

Testing

  • mise run pre-commit passes
  • Unit tests added/updated
  • E2E tests added/updated (if applicable)

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated (if applicable)

@drewdrew self-assigned this Apr 20, 2026
@drew
drew requested a review from a team as a code ownerApril 20, 2026 00:48
@copy-pr-bot

copy-pr-botBot commented Apr 20, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions

github-actionsBot commented Apr 20, 2026

Copy link
Copy Markdown

🌿 Preview your docs:https://nvidia-preview-pr-888.docs.buildwithfern.com/openshell

@drew
drew marked this pull request as draft April 20, 2026 02:16
Base automatically changed from feat/supervisor-session-grpc-data to mainApril 21, 2026 15:38
@drew
drewforce-pushed the drew/creating-a-docker-driver-like-the-vm-driver branch from 2f26563 to 8bf06dbCompareApril 22, 2026 04:01
drew added 9 commits April 24, 2026 15:30
@drew
feat(server): add bundled docker compute driver
79f4935
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
@drew
test(sandbox): make procfs unlinked-path test kernel-tolerant
7f60654
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
@drew
pr feedback
be2520f
@drew
fix(server): address PR review feedback for docker compute driver
d6c9543
- Preserve sandbox id suffix in container name when sandbox name is long,
preventing collisions and confusing 'already exists' errors.
- Use container id (not name) in delete_sandbox_inner so transient
ContainerSummary entries without names still get cleaned up.
- Subscribe to the watch broadcast before snapshotting so events that fire
between snapshot and subscribe aren't missed by new watchers.
- Apply exponential backoff to the Docker poll loop on consecutive
failures, capping at 30s to avoid log floods on daemon outages.
- Reject --docker-tls-* flags when the gRPC endpoint is plaintext http://
instead of silently discarding them.
- Add e2e:docker mise task and e2e/rust/e2e-docker.sh harness that boots a
standalone gateway with the docker driver and runs the existing smoke
test against it.
@drew
fix(server): flip docker sandbox Ready=True when supervisor relay con…
ce370c7
…nects
The docker compute driver mapped RUNNING containers to Ready=False with
reason DependenciesNotReady indefinitely, so sandboxes never transitioned
to the Ready phase and ExecSandbox was gated out behind a 180s timeout.
Introduces a SupervisorReadiness trait that the driver polls on every
watch tick. The gateway's SupervisorSessionRegistry implements it via a
new is_connected(sandbox_id) method. When a ConnectSupervisor session is
live for a sandbox, the driver emits Ready=True with reason
SupervisorConnected; the condition falls back to DependenciesNotReady
if the supervisor disconnects.
Also:
- Wires the registry through run_server/ServerState so the docker driver
can be constructed before ServerState exists.
- Adds a host.openshell.internal / host.docker.internal SAN to the
e2e-docker.sh generated mTLS cert so supervisor TLS handshakes succeed.
- Points the e2e harness at the community sandbox base image (which has
the required 'sandbox' user) and preserves container logs on failure
for post-mortem debugging.
- Passes mise run e2e:docker end to end.
@drew
fix(server): harden docker sandbox reconciliation
ea76b39
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
@drew
feat(tasks): add standalone docker gateway task
d07c25d
@drew
chore(docs): drop docker gateway task references
365ab51
@drew
chore(docs): drop docker driver documentation
215f8b0
@drew
drewforce-pushed the drew/creating-a-docker-driver-like-the-vm-driver branch from 8214ca2 to 215f8b0CompareApril 25, 2026 00:11
@drew
drew marked this pull request as ready for review April 27, 2026 05:54
johntmyers
johntmyers reviewed Apr 27, 2026
View reviewed changes
Comment threade2e/rust/e2e-docker.sh Outdated
johntmyers
johntmyers reviewed Apr 27, 2026
View reviewed changes
Comment threade2e/rust/e2e-docker.sh Outdated
drew added 2 commits April 27, 2026 08:21
@drew
test(e2e): scope docker e2e cleanup to its own sandbox namespace
462051f
Pass a unique per-run --sandbox-namespace to the gateway and filter
docker ps cleanup queries on both managed-by and that namespace label,
so concurrent test runs and unrelated local sandboxes are not removed
by the trap.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
@drew
Merge remote-tracking branch 'origin/main' into drew/creating-a-docke…
8691d0d
…r-driver-like-the-vm-driver
johntmyers
johntmyers previously approved these changes Apr 27, 2026
View reviewed changes
pimlock
pimlock reviewed Apr 27, 2026
View reviewed changes
Comment threadarchitecture/gateway.md
Comment threadcrates/openshell-driver-docker/src/lib.rs Outdated
Comment threadcrates/openshell-driver-docker/src/lib.rs
@drew
fix(driver-docker): use runtime namespace label and validate identifiers
895a615
Two related correctness issues surfaced in PR review:
* build_container_create_body wrote DriverSandbox.namespace to the
sandbox-namespace label, but the gateway elides that field. The
list/get/find code paths filter by config.sandbox_namespace, so the
driver could not observe sandboxes it had just created. The label now
uses the same runtime config value as the filters, with a regression
test that exercises a non-default namespace.
* delete_sandbox accepted requests with both sandbox_id and
sandbox_name empty. Empty filters degenerate to 'match every managed
container in the namespace', so an arbitrary sandbox would be
removed. Centralized the empty-identifier guard into
require_sandbox_identifier and applied it to delete_sandbox,
stop_sandbox, and get_sandbox for consistency, with a unit test on
the helper.
@drew
drew merged commit 5975805 into mainApr 28, 2026
21 checks passed
@drew
drew deleted the drew/creating-a-docker-driver-like-the-vm-driver branch April 28, 2026 00:38
@miyoungcmiyoungc mentioned this pull request May 4, 2026
Closed
8 tasks
@gburachasgburachas mentioned this pull request May 13, 2026
Closed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pimlockpimlockpimlock approved these changes
@johntmyersjohntmyersjohntmyers left review comments

Assignees

@drewdrew

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@drew@pimlock@johntmyers@kirit93