feat(kubernetes): add sidecar supervisor topology

[TaylorMutch]

Summary

Adds an experimental Kubernetes sidecar supervisor topology after the combined-topology base from #2074. The default combined topology remains unchanged; sidecar is opt-in and may change as we continue validating the permission model, pod layout, and configuration surface across Kubernetes runtimes.

The goal of sidecar is to move pod-level network enforcement, gateway authentication, gateway session handling, and SSH relay ownership out of the agent container and into a dedicated network sidecar. The agent container runs as the resolved sandbox UID/GID with runAsNonRoot, allowPrivilegeEscalation: false, and all Linux capabilities dropped.

Outcome

• Keeps combined as the default topology and full OpenShell enforcement path.
• Adds topology = "sidecar" under [openshell.drivers.kubernetes] and nests sidecar-only settings under [openshell.drivers.kubernetes.sidecar]; Helm renders the same shape from supervisor.topology and supervisor.sidecar.*.
• Splits sidecar-mode pods into a root network init container, an isolated network supervisor sidecar, and a low-privilege agent container.
• Enables shareProcessNamespace: true for sidecar pods so the network sidecar can resolve workload process and binary identity through /proc.
• Keeps process/binary-aware network policy enabled by default in sidecar mode. Setting supervisor.sidecar.processBinaryAwareNetworkPolicy=false drops the sidecar SYS_PTRACE capability and downgrades sidecar network policy to endpoint/L7 enforcement without matching policy.binaries.
• Moves gateway credentials and gateway callback/session ownership into the network sidecar. The agent container receives policy and provider environment state over the sidecar control socket instead of mounting gateway authentication material. Both supervisors exit if their authenticated control connection closes, coupling their restart lifecycle.
• Adds sidecar Helm overlays, Skaffold profiles, docs, and a branch Kubernetes E2E matrix entry for the sidecar topology.
• Adds deploy/helm/openshell/ci/values-sidecar-kata.yaml as the local/CI-dev overlay used for Kata validation.

Enforcement Model

combined remains the complete OpenShell sandbox contract: network policy, filesystem policy, process controls, supervisor privilege drop, supervisor identity mount isolation, gateway relay, SSH, exec, and file sync all run through the existing agent-container supervisor path.

sidecar is network-focused but no longer disables all process-side enforcement:

• Network endpoint and L7 policy are enforced by the network sidecar.
• Process/binary-aware network policy is enforced by default through the shared PID namespace and sidecar /proc inspection.
• The process supervisor in the agent container still launches the workload and applies Landlock filesystem policy and child seccomp filters where the kernel/runtime supports them.
• The process supervisor does not perform root-to-sandbox privilege dropping in sidecar mode because Kubernetes starts the agent container directly as the sandbox UID/GID.
• Supervisor identity mount isolation is not applied in the agent container because gateway credentials are not mounted there in the default sidecar path.

Runtime Validation Status

• GKE Autopilot: sidecar mode has been validated on Autopilot clusters created with gcloud container clusters create-auto --workload-policies=allow-net-admin; the Google Cloud CLI reference lists allow-net-admin as the supported Autopilot workload policy: https://docs.cloud.google.com/sdk/gcloud/reference/container/clusters/create-auto#--workload-policies
• Kata Containers: sidecar mode has been validated on our Kata-enabled Kubernetes cluster using branch-built gateway/supervisor images and deploy/helm/openshell/ci/values-sidecar-kata.yaml. This validation passed without using the supervisor sideload init container.
• gVisor: this PR does not claim gVisor support. The sidecar topology still depends on pod-local network rule setup, which was previously observed to fail under gVisor.

Related Issue

References #1827, #981, #899, #1305.

Related PRs: #1973, #2074, #2016.

Changes

• Include prerequisite sandbox UID/GID support: numeric sandbox identities, configurable Kubernetes sandbox UID/GID values, and OpenShift SCC annotation fallback.
• Add the experimental sidecar topology and sidecar-specific configuration.
• Rename Kubernetes driver supervisor_topology to topology and move sidecar settings under openshell.drivers.kubernetes.sidecar.
• Render sidecar-mode sandbox pods with a network init container, a privilege-scoped supervisor sidecar, shared PID namespace, and unprivileged agent container.
• Add sidecar control-socket bootstrap for policy/provider environment state so the agent container does not need gateway credentials.
• Add process-supervisor network-only behavior for sidecar mode while preserving workload launch, Landlock/seccomp application, and SSH/session relay behavior.
• Add a sidecar flag to disable process/binary-aware network policy and drop the sidecar SYS_PTRACE capability when users accept the downgrade.
• Add IPv4-only fallback handling for sidecar network setup so Kata clusters without IPv6 support can start sidecar pods.
• Add sidecar and sidecar-Kata Helm overlays plus Skaffold profile support.
• Add a sidecar topology row to the branch Kubernetes E2E matrix.
• Document topology choice, architecture diagrams, permission model, RuntimeClass validation status, and sidecar tradeoffs.

Testing

• git diff --check origin/main..feat/kubernetes-sidecar-topology-v2
• cargo check -p openshell-driver-kubernetes -p openshell-sandbox -p openshell-supervisor-process -p openshell-supervisor-network
• cargo test -p openshell-driver-kubernetes --lib
• cargo test -p openshell-supervisor-process --lib
• mise run helm:test
• markdownlint-cli2 docs/kubernetes/topology.mdx docs/kubernetes/setup.mdx
• ruby -e 'require "yaml"; YAML.load_file(".github/workflows/branch-e2e.yml"); puts "ok"'
• mise run rust:lint
• CI container: mise run rust:format:check && mise run rust:lint
• CI container: cargo test -p openshell-supervisor-process
• CARGO_BUILD_JOBS=1 mise run pre-commit
• GitHub OpenShell / Branch Checks passing on this branch, including Linux amd64 and arm64 Rust jobs.
• Prior validation from feat(kubernetes): add sidecar and proxy-pod topology configurations #2016: HELM_K3S_LB_HOST_PORT=18080 mise run e2e:kubernetes:sidecar
• Sidecar topology validated on GKE Autopilot using --workload-policies=allow-net-admin.
• Sidecar topology validated on a Kata-enabled Kubernetes cluster with branch-built images and deploy/helm/openshell/ci/values-sidecar-kata.yaml; validation passed without the supervisor sideload init container.

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[github-actions]

🌿 Preview your docs:https://nvidia-preview-pr-2076.docs.buildwithfern.com/openshell

[github-actions]

Label test:e2e applied, but pull-request/2076 is at {"messa while the PR head is 6276de3. A maintainer needs to comment /ok to test 6276de38ac92b65b0b24a68b8d09a5eed06f50d7 to refresh the mirror. Once the mirror catches up, re-run Branch E2E Checks from the Actions tab.

[mrunalp]

Some comments with help of agent

1. Central trust boundary — the workload can use the sandbox's gateway identity .
Client mTLS and the SA token render 0440 with fsGroup=sandbox_gid, the process leaf skips supervisor-identity isolation in network-only, and the 127.0.0.1:18080 forward is always open. Since the workload child also runs with sandbox_gid, it can read those credentials and reach the gateway directly.

2. nftables fence only covers TCP/UDP.
The output chain is policy accept and only rejects TCP/UDP from non-proxy_uid UIDs. ICMP (e.g. unprivileged ping sockets) and other L4 protocols fall through. CAP_NET_RAW is dropped so raw sockets aren't available, but this is a narrower egress boundary than combined mode's per-netns setup. Low severity — suggest a brief comment in generate_sidecar_bypass_ruleset noting the intentional scope.

3. processEnforcement=full does not restore network binary-identity.
supervisor_sidecar_env hard-codes NETWORK_BINARY_IDENTITY=relaxed regardless of process_enforcement, so full restores agent-container Landlock/privilege-drop but network policy stays endpoint/L7-scoped. The docs are technically accurate ("combined-mode process/filesystem controls"), but a reader could reasonably infer full ≈ combined. A one-line note that network policy remains endpoint-scoped in sidecar mode regardless of processEnforcement would prevent that.

4. Doc drift in the debug skill.
debug-openshell-cluster/SKILL.md says the sidecar runs "with primary GID 0" to read the token, but the implementation uses runAsGroup: sandbox_gid + fsGroup: sandbox_gid (which re-owns the projected volume to sandbox_gid). Please update to match.

[TaylorMutch]

1. Central trust boundary — the workload can use the sandbox's gateway identity .
   Client mTLS and the SA token render 0440 with fsGroup=sandbox_gid, the process leaf skips supervisor-identity isolation in network-only, and the 127.0.0.1:18080 forward is always open. Since the workload child also runs with sandbox_gid, it can read those credentials and reach the gateway directly.

Addressed this by moving the supervisor and sandbox to run with different GIDs.

Updated the other areas with clarifying comments.

[mrunalp]

Deployed this branch on an OpenShift 4.22 cluster (K8s 1.35) with supervisor.topology=sidecar and TLS + mTLS enabled (default PKI init job). The sidecar pod renders correctly, but openshell-network-init is stuck in CrashLoopBackOff:

Error: × failed to create sidecar directory /etc/openshell-tls/proxy/client
╰─▶ Permission denied (os error 13)

$ oc -n openshell get pod test-sidecar
NAME READY STATUS RESTARTS
test-sidecar 0/2 Init:CrashLoopBackOff 6

Root cause

In run_network_init (crates/openshell-sandbox/src/main.rs) the ordering is:

1. prepare_sidecar_directory(sidecar_tls_dir /* /etc/openshell-tls/proxy */, proxy_uid, proxy_gid, 0o755) — this chowns the TLS dir to proxy_uid (e.g. 1337), mode 0755.
2. copy_sidecar_client_tls_if_present(...) → prepare_sidecar_directory(dest_dir /* …/proxy/client */, …) → create_dir_all.

Step 2 runs as root with capabilities.drop: ["ALL"] (the init container only adds NET_ADMIN, NET_RAW, CHOWN, FOWNER, so it has no CAP_DAC_OVERRIDE). The parent dir is now owned by uid 1337, mode 0755, so root is neither the owner nor has group-write → EACCES. Root can no longer bypass DAC checks because DAC_OVERRIDE was dropped.

Note the state dir was already changed to 0o2775 (setgid + group-write), but the TLS dir is still 0o755, and it's the one that gets a subdirectory created in it right after being chowned away from root.

[TaylorMutch]

@mrunalp I reproduced locally and pushed a fix. Please have a look when you get a chance!

[drew]

This draft intentionally includes the prerequisite UID/GID commits from #1973 for now. We expect to rebase those out after that work lands separately.

Looks like you've rebased now since that's been merged?

[mrunalp]

Discussed with @TaylorMutch:

Suggestion: keep Landlock (and seccomp) in sidecar network-only mode — they don't need privilege

In sidecar topology the process leaf runs network-only, which currently disables Landlock, seccomp, and filesystem policy along with the genuinely privileged controls:

// crates/openshell-supervisor-process/src/process.rs:591let prepared_sandbox = if enforcement_mode.enforces_process_controls(){Some(sandbox::linux::prepare(policy, workdir)?)// Landlock ruleset prep}else{None};

enforces_process_controls() is false for sidecar, so prepare() is skipped and no Landlock ruleset is built — same in the ssh.rs spawn paths. But most of what's gated here doesn't actually require privilege, so we're relaxing more than we need to.

Landlock is unprivileged by design

landlock_create_ruleset / add_rule / restrict_self need no capabilities — Landlock only ever removes access, so any process may self-restrict. The single precondition for restrict_self is no_new_privs, and we already set that unconditionally:

• harden_child_process() → set_no_new_privs() (sandbox/linux/seccomp.rs:126) runs in the child pre-exec outside the enforces_process_controls() gate (e.g. ssh.rs:1266), so it fires in sidecar mode too.

So the kernel prerequisite for Landlock is already satisfied in the capless agent container. It's gated off only incidentally:

1. Bundling — enforces_process_controls() lumps Landlock with drop_privileges (setuid/setgid — needs root) and the supervisor-identity mount isolation (needs CAP_SYS_ADMIN/userns). Landlock got swept up with controls that genuinely need privilege.
2. "Prepare as root" — prepare() opens Landlock PathFds "as root before drop_privileges" (sandbox/linux/mod.rs:15,21) so it can open mode-700 paths. That's an optimization for the root→user transition, not a Landlock requirement.

What it would take

• Ungate Landlock (and seccomp) from enforces_process_controls() so they apply even when we're not dropping privileges.
• Open the path FDs as the sandbox UID in the sidecar path. Paths that UID can't open are ones the workload can't reach anyway — skip them / treat open-failure as already-denied. "Open as root" is unnecessary when the workload is that UID.
• Keep the existing kernel-availability probe (log_sandbox_readiness) and degrade gracefully where Landlock/ABI is unavailable.

Seccomp is the same: filters install unprivileged under no_new_privs, so the syscall hardening can also run in the sidecar agent.

What legitimately stays out — and why it doesn't matter here

• Privilege dropping (root → sandbox uid): needs root, but in sidecar mode Kubernetes already starts the container as runAsUser=sandbox_uid, runAsNonRoot: true. Nothing to drop.
• Supervisor-identity mount isolation (hiding the SA token): needs CAP_SYS_ADMIN. Moot in sidecar mode — the credential redesign already keeps gateway creds out of the agent container (snapshot brokering).

Net effect

Re-enabling Landlock + seccomp shrinks the sidecar's "relaxed filesystem/process controls" tradeoff down to essentially just "network binary-identity is relaxed." The agent container stays capless and non-root, but the workload regains filesystem and syscall confinement.

Caveat: kernel availability

Landlock must be compiled in (CONFIG_SECURITY_LANDLOCK) and present in the active LSM list. RHEL 9 / RHCOS historically shipped without it enabled (added in later minors), so on OpenShift it may be unavailable even after ungating — in which case the availability probe should simply no-op it (as it does today), while it engages on upstream/newer kernels (k3d, GKE, etc.). Worth ungating regardless: it applies wherever the kernel supports it and degrades cleanly where it doesn't. Quick check on a target node: cat /sys/kernel/security/lsm should list landlock.

[TaylorMutch]

@mrunalp addressed in cd0477f - thank you for calling it out!