test(e2e): run gpu workloads from manifest

[elezar]

Summary

This PR adds manifest-driven GPU workload execution tests on top of the workload image artifacts from #1484. It keeps the existing GPU device-selection coverage, adds workload execution coverage under the umbrella gpu target, and documents how to build workload images locally before running the GPU e2e suite.

This branch is now rebased on the local e2e stabilization fixes from #1935, so the Docker GPU test path also includes the supervisor-image and host SSH linker-environment fixes needed for local Nix/devenv runs.

Related Issue

Closes#1472

Changes

• Switch GPU workload execution tests from a single image env var to a YAML workload manifest consumed by the Rust e2e harness.
• Run the manifest-defined workloads through openshell sandbox create --gpu --from <image> -- <command> and enforce declared pass or fail expectations.
• Load the local manifest from e2e/gpu/images/.build/workloads.yaml by default, with OPENSHELL_E2E_WORKLOAD_MANIFEST available for external manifests.
• Update the Docker GPU e2e wrapper to point users at the workload manifest flow when no local manifest exists.
• Add serde_yaml to the e2e crate for manifest parsing.
• Include the local e2e fixes from fix(docker): honor configured supervisor image #1935: configured Docker supervisor image handling and host SSH linker-environment isolation.

Testing

• mise run pre-commit passes
• Unit tests added/updated
• E2E tests added/updated (if applicable)

Validation status:

• mise run e2e:docker:gpu
• mise run pre-commit was run after rebasing onto main; Rust format/check/clippy, markdown lint, Python format, license checks, and docs checks completed successfully.
• mise run pre-commit currently fails in helm:lint because the local chart dependency directory is missing the postgresql dependency. This is unrelated to the GPU workload changes.

GPU validation commands for future runs:

• mise run e2e:workloads:build
• mise run e2e:docker:gpu

Notes:

• Build workload images and generate the local manifest with mise run e2e:workloads:build before running mise run e2e:docker:gpu locally.
• External catalogs can be exercised by setting OPENSHELL_E2E_WORKLOAD_MANIFEST=/abs/path/to/workloads.yaml.

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture docs updated (if applicable)

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[github-actions]

🌿 Preview your docs:https://nvidia-preview-pr-1709.docs.buildwithfern.com/openshell

[github-actions]

Label test:e2e-gpu applied for 2f36b22. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute GPU E2E after building the required supervisor image once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

[github-actions]

Label test:e2e applied for 2f36b22. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

[elezar]

gator-agent

Blocked

Gator is blocked by merge conflicts with the base branch. GitHub reports mergeable_state=dirty for head dbdc8134896c94f5f9babeae6fd8b5fb649494ab.

Next action: @elezar, please rebase or merge the base branch and resolve the conflicts, then push an updated head so gator can validate and review the PR.

[elezar]

gator-agent

Re-check After Author Update

I re-evaluated latest head 6c30486aaa9df6a1f63608cfea6f98266eca214e after the branch update since the prior merge-conflict blocker, and ran a bounded independent review of the current diff.

Disposition: partially resolved.

Remaining items:

• Resolved: GitHub now reports mergeable=true and mergeable_state=blocked, so the previous dirty merge-conflict blocker is cleared.
• Still unresolved: .github/workflows/e2e-gpu-test.yaml builds workload images without pinning the container engine. tasks/scripts/container-engine.sh auto-detects Podman first when present, so this Docker GPU workflow can build workload images into Podman storage and then run e2e:docker:gpu against Docker, where those image refs are unavailable. Please run the workload build with CONTAINER_ENGINE=docker, for example: CONTAINER_ENGINE=docker mise run --no-deps --skip-deps e2e:workloads:build.
• Nonblocking suggestion: tasks/test.toml now runs the full gpu suite for Podman too; consider adding matching manifest setup guidance to e2e/rust/e2e-podman.sh so users do not accidentally run Podman GPU e2e with a Docker-built manifest.
• OpenShell / Branch Checks and OpenShell / Helm Lint are passing. OpenShell / E2E and OpenShell / GPU E2E are still pending for this head.

Docs: GPU e2e documentation and gateway config reference updates are present; no docs navigation change appears necessary.

Next state: gator:in-review

[elezar]

gator-agent

Re-check After CI Update

I re-evaluated latest head 6c30486aaa9df6a1f63608cfea6f98266eca214e after the required gates completed.

Disposition: partially resolved, but not ready to leave review.

Remaining items:

• Resolved: OpenShell / Branch Checks, OpenShell / Helm Lint, DCO, OpenShell / E2E, and OpenShell / GPU E2E are passing for this head.
• Still unresolved: .github/workflows/e2e-gpu-test.yaml builds workload images without pinning the container engine. The workflow should run the workload build with CONTAINER_ENGINE=docker, for example CONTAINER_ENGINE=docker mise run --no-deps --skip-deps e2e:workloads:build, or explain why Podman-first auto-detection cannot affect this Docker GPU path.
• Nonblocking suggestion remains: add matching manifest setup guidance to e2e/rust/e2e-podman.sh for Podman GPU e2e users.

Next state: gator:in-review

[elezar]

gator-agent

Re-check After Author Update

I re-evaluated latest head 0e9f9fc605e36400cf32fbd9370e88d77019ec44 after the branch update since the prior gator:in-review feedback, and ran a bounded independent re-review of the current diff.

Disposition: resolved for review; CI is still in progress.

Remaining items:

• Resolved: .github/workflows/e2e-gpu-test.yaml now sets CONTAINER_ENGINE=docker before building workload images, so the Docker GPU workflow builds workload images into the same engine used by e2e:docker:gpu.
• No blocking gator review findings remain.
• Nonblocking suggestions from the independent review: mirror the missing-manifest guidance in the Podman GPU wrapper, update the stale workflow comment path from e2e/rust/tests/gpu_device_selection.rs to e2e/rust/tests/gpu/device_selection.rs, refresh the TESTING.md Docker GPU description, and keep the serde_yaml/unsafe-libyaml dependency scoped to e2e-only use.
• OpenShell / Branch Checks and OpenShell / Helm Lint are passing. OpenShell / E2E and OpenShell / GPU E2E are still pending for this head.

Docs: GPU e2e documentation is updated under e2e/gpu/README.md; no Fern docs or navigation change appears required for this internal test harness change.

Next state: gator:watch-pipeline

[elezar]

gator-agent

Blocked

Gator is blocked from completing the required independent re-review for current head a351e3c14b75a842ca08396e34e7c0fca47d2821 because the principal-engineer-reviewer sub-agent failed before producing a review result due to a Codex token refresh/authentication error.

The PR also has new CI in progress for this head, so it is not ready to stay in gator:approval-needed yet.

Next action: OpenShell sandbox operator should refresh or relaunch gator with a working reviewer sub-agent, then re-run gator so the current diff can be independently reviewed and the pending checks can be reconciled. No PR author action is requested by this blocker.

[elezar]

gator-agent

Re-check After Reviewer Update

I re-evaluated latest head a351e3c14b75a842ca08396e34e7c0fca47d2821 after the prior gator blocker, and the required principal-engineer-reviewer sub-agent completed successfully for the current diff.

Disposition: blocker resolved, but review follow-up is still needed.

Remaining items:

• Resolved: the previous process blocker was only the failed reviewer sub-agent run; the independent review has now completed.
• Still unresolved: tasks/scripts/e2e-gpu-build-images.sh now depends directly on GNU sha256sum for file and input hashing. OpenShell supports macOS developer workflows, and nearby project tasks use a sha256sum / shasum -a 256 fallback. Please add a small hash helper so local GPU workload image builds work on non-GNU userlands too.
• Nonblocking suggestions: mirror the missing-manifest guidance in e2e/rust/e2e-podman.sh, refresh the TESTING.md Docker GPU description for manifest workload validation, and keep the deprecated serde_yaml / unsafe-libyaml dependency scoped to the e2e crate only.

Checks: OpenShell / Branch Checks, OpenShell / Helm Lint, DCO, OpenShell / E2E, and OpenShell / GPU E2E are passing for this head.

Docs: GPU E2E documentation is updated under e2e/gpu/README.md; no Fern docs or navigation update appears required for this internal test harness change.

Next state: gator:in-review

[elezar]

gator-agent

Blocked

Gator is blocked from completing the required independent re-review for current head 87ea21e479ae5c52ba1c4278677f549ea548407a because the principal-engineer-reviewer sub-agent failed before producing a review result due to a Codex token refresh/authentication error.

The required CI gates are passing for this head, but gator cannot move the PR forward until the current diff has a successful independent review.

Next action: OpenShell sandbox operator should refresh or relaunch gator with a working reviewer sub-agent, then re-run gator so the current diff can be independently reviewed and reconciled. No PR author action is requested by this blocker.

[elezar]

gator-agent

PR Review Status

Validation: maintainer-authored, project-valid GPU E2E test-harness work linked to #1472.
Head SHA: 6938613dd5f91e5bdcb54e9d34c03be449e2d92f

Review findings:

• e2e/rust/tests/gpu/workloads.rs:104 and e2e/rust/tests/gpu/workloads.rs:138: workload sandbox creation is unbounded. A hung image pull, GPU runtime deadlock, or bad external manifest command can stall the GPU job until the outer CI timeout. Please add the same timeout pattern used by the existing device-selection tests around both expected-pass and expected-fail SandboxGuard::create calls. Security mapping: CWE-400.
• e2e/rust/Cargo.toml:120 / e2e/rust/Cargo.lock:886: serde_yaml 0.9.34+deprecated adds unsafe-libyaml for parsing external manifests. Even in e2e-only code, this expands the parser and supply-chain surface for local external input. Please prefer a maintained parser, or explicitly constrain this as a temporary test-only dependency. Security mapping: CWE-1104.
• e2e/gpu/README.md:127: the documented raw cargo test command does not start the Docker gateway or set the same environment as e2e:docker:gpu, so local users can follow the README and hit confusing failures. Please document the task flow instead, for example mise run e2e:workloads:build followed by mise run e2e:docker:gpu.

Nonblocking suggestions:

• In e2e/rust/tests/gpu/workloads.rs, capture output and call guard.cleanup().await before asserting the success marker, so assertion failures do not leave a sandbox behind.
• Mirror the Docker missing-manifest hint in e2e/rust/e2e-podman.sh, since e2e:podman:gpu now runs the same gpu test target.

Docs: GPU E2E documentation is updated under e2e/gpu/README.md; no Fern docs or navigation update appears required for this internal test harness change. The README command issue above is ordinary review feedback.

Checks: OpenShell / Helm Lint and DCO are passing. OpenShell / Branch Checks, OpenShell / E2E, and OpenShell / GPU E2E are still pending for this head.

Next state: gator:in-review

[elezar]

gator-agent

PR Review Status

Validation: maintainer-authored, project-valid GPU E2E test-harness work linked to #1472.
Head SHA: e63040b452ce702d05ba5520d176a4ec232971b5

Review findings:

• tasks/scripts/e2e-gpu-build-images.sh:116: the script still uses GNU-only sort -z. The prior macOS/BSD portability feedback is only partially resolved by the sha256sum / shasum -a 256 fallback; the documented mise run e2e:workloads:build path can still fail on non-GNU userlands. Please replace the find ... -print0 | sort -z path with a portable helper that preserves null-delimited sorting, or otherwise avoid GNU-only sort options.

Nonblocking suggestions:

• serde_yml / libyml is scoped to the e2e crate, which is acceptable here. Please keep it test-harness-only and do not reuse it in runtime crates without a separate dependency/security review.

Docs: GPU E2E documentation is updated under e2e/gpu/README.md; no Fern docs or docs/index.yml update appears required for this internal test harness change.

Checks: OpenShell / Branch Checks, OpenShell / Helm Lint, and DCO are passing. OpenShell / E2E and OpenShell / GPU E2E are still pending for this head.

Next state: gator:in-review

[elezar]

Follow-up on the serde_yml note:

I checked the current autosquashed branch against origin/main. PR 1709 adds serde_yml / libyml only to the standalone e2e/rust crate for GPU workload manifest parsing.

serde_yml is already part of the project dependency surface and is used elsewhere in the repository. This PR keeps the new usage scoped to the e2e test harness, so I do not think it needs a separate dependency/security review for that scoped use.

A separate review would make sense for introducing a brand-new YAML parser, changing runtime YAML parsing behavior, or expanding this dependency into new runtime paths. That is not what this PR is doing.

[elezar]

gator-agent

PR Review Status

Validation: maintainer-authored, project-valid GPU E2E test-harness work linked to #1472.
Head SHA: f84ea0f1f65eee0d9a23d999b0d62f80156fc7ca

Review findings:

• No blocking findings remain. The independent reviewer found no Critical or Warning issues for this head.
• Resolved: the prior GNU-only sort -z portability finding in tasks/scripts/e2e-gpu-build-images.sh is resolved by the portable find -print0 plus Bash sorting flow.
• Nonblocking suggestion: consider a small parser-focused unit test around the workload manifest loader for empty manifests, non-GPU workloads, and malformed expect values.

Docs: GPU E2E documentation is updated under e2e/gpu/README.md; no Fern docs or docs/index.yml update appears required for this internal test harness change.

Checks: OpenShell / Helm Lint and DCO are passing. OpenShell / Branch Checks, OpenShell / E2E, and OpenShell / GPU E2E are still pending for this head.

Next state: gator:watch-pipeline

[elezar]

/ok-to-test e776a13