feat(auth): add SPIFFE supervisor authentication

[TaylorMutch]

Summary

Add optional SPIRE/SPIFFE support for Helm dev clusters and allow sandbox supervisors to authenticate to the gateway with SPIFFE JWT-SVIDs instead of gateway-minted JWTs.

Related Issue

Stacked on #1404.

Changes

• Add SPIFFE configuration and sandbox environment plumbing across core, gateway, sandbox, and Kubernetes driver code.
• Install SPIRE as an opt-in Helm dev component and mount the SPIFFE CSI Workload API socket into gateway and sandbox pods.
• Use the rust-spiffe crate for JWT-SVID fetch and validation, aligned with tonic/prost 0.14.
• Document SPIFFE/SPIRE dev usage and update local debugging guidance.
• Ignore git-ignored architecture plan scratch files in markdownlint so pre-commit does not lint local plan notes.

Testing

• RUSTC_WRAPPER= mise run pre-commit passes
• cargo check -p openshell-core -p openshell-server -p openshell-sandbox -p openshell-driver-kubernetes passes
• Focused SPIFFE unit tests passed
• Helm lint/unit tests passed
• Local /helm-dev-environment SPIRE deployment exercised sandbox list/create/delete and supervisor connect-back

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[github-actions]

🌿 Preview your docs:https://nvidia-preview-pr-1414.docs.buildwithfern.com/openshell

[mrunalp]

1. unshare(CLONE_NEWNS) runs on a shared tokio worker thread

Refs: crates/openshell-sandbox/src/process.rs, crates/openshell-sandbox/src/lib.rs

prepare_supervisor_identity_mount_namespace_from_env is called from run_sandbox and executes libc::unshare(CLONE_NEWNS) followed by a recursive MS_PRIVATE remount of /, then mounts a tmpfs over the socket directory, then setns() back to the original namespace. unshare(CLONE_NEWNS) only affects the calling thread. On a multi-threaded tokio runtime this thread can be stolen by other tasks between the unshare and the setns restore, so unrelated work transiently observes the modified mount namespace and the recursive private remount.

Suggested way forward:

• Run the namespace-creation sequence inside tokio::task::spawn_blocking (or on a dedicated single-purpose thread) so it is guaranteed to be the only work on that OS thread for the entire create-and-restore window.
• Alternatively, perform this setup from a synchronous entry point before the tokio runtime is even built — the namespace fd just needs to be available before child pre_exec.
• Add an assertion that no other tokio task can be scheduled on the namespace-mutating thread, or split into a dedicated runtime.

2. Per-RPC ValidateJWTSVID couples gateway availability to local SPIRE

Refs: crates/openshell-server/src/auth/spiffe.rs

SpiffeAuthenticator::validate_bearer calls the local SPIFFE Workload API on every authenticated RPC. This is correct from a correctness standpoint (delegation is the point) but means a SPIRE agent stall, restart, or socket disconnect translates directly into gateway-wide auth failures — including for RPCs that have nothing to do with SPIFFE if the chain reaches the authenticator. The gateway-minted JWT path validates locally with cached keys and survives identity-system outages.

Suggested way forward:

• Add a small validation cache keyed on (token-hash, exp) so a stalled SPIRE agent does not immediately fail in-flight RPCs that already validated within the last few seconds. Bound the cache TTL well below SVID lifetime.
• Surface SPIRE Workload API health as a separate readiness signal so operators can see "gateway up, SPIRE down" instead of generic 401s.
• Document the dependency explicitly in the operator-facing SPIFFE docs.

3. refresh_spiffe_token_loop sleeps twice on error

Refs: crates/openshell-sandbox/src/grpc_client.rs

On FetchJWTSVID failure the loop logs, sleeps 60 s, then continues to the top where compute_refresh_delay(&slot) sleeps again using the stale token's claims. With a fresh-but-just-expired token the second sleep collapses to its lower bound, but with a recently-rotated token it can add minutes. The cumulative effect is that recovery from a transient SPIRE blip is slower than the operator expects from the 60 s back-off alone.

Suggested way forward:

• Replace the inner sleep(60s); continue with a continue that re-enters the loop top, where compute_refresh_delay is the single point that decides backoff.
• Use a small dedicated retry-backoff helper so future authors do not reintroduce the double-sleep.
• Add a unit test that simulates a failing FetchJWTSVID and asserts the next attempt happens within one backoff window, not two.

4. Token-source precedence is silent when multiple are configured

Refs: crates/openshell-sandbox/src/grpc_client.rs

acquire_sandbox_token checks env in order: SANDBOX_TOKEN, SANDBOX_TOKEN_FILE, SPIFFE_WORKLOAD_API_SOCKET, K8S_SA_TOKEN_FILE. A misconfigured pod that has both SPIFFE_WORKLOAD_API_SOCKET and K8S_SA_TOKEN_FILE set will silently use SPIFFE and ignore the SA token — likely fine, but a config-migration mistake could leave both set and the operator would have no warning.

Suggested way forward:

• Log all detected sources at startup, even the ones not used.
• Warn loudly when more than one supervisor identity source is configured, so misconfiguration is visible at sandbox boot rather than discovered during incident response.
• Optionally treat multi-source config as a hard error behind a "strict identity" feature flag.

5. Default trust domain openshell.local ships in values.yaml

Refs: deploy/helm/openshell/values.yaml, deploy/helm/openshell/ci/values-spire-stack.yaml

The shipped default trust domain is openshell.local. Two production clusters that both adopt SPIFFE with defaults will end up minting overlapping SPIFFE IDs, which becomes a real problem the moment anyone enables SPIFFE Federation or shares an identity-aware service across clusters.

Suggested way forward:

• Make server.spiffe.trustDomain required when server.spiffe.enabled=true — no default, fail Helm install with a clear message.
• If a default is kept, derive it from .Release.Name plus a deployment-id hash so two installs collide only by deliberate choice.
• Document trust-domain naming guidance prominently in the SPIFFE section of the chart README.

6. Mount-namespace plumbing is exercised only by path-parsing tests

Refs: crates/openshell-sandbox/src/process.rs

The unshare / mount tmpfs / setns plumbing is #[cfg(target_os = "linux")] and unit-test coverage stops at supervisor_identity_mount_target_*. The PR description claims local Helm-dev exercise, which is real but not reproducible in CI.

Suggested way forward:

• Add a Linux-only integration test that runs under --cap-add=SYS_ADMIN or as root in CI, creates a sandbox child, and asserts the socket path is not visible from the child's view.
• Wire it into the sandbox e2e path (per AGENTS.md guidance about sandbox infra changes).
• If the namespace setup is moved to spawn_blocking per 1, assert no extra mount-namespace state leaks back to the supervisor's primary worker thread.

[TaylorMutch]

Closing this in favor of the direction started in #1781; we don't need the supervisor using an alternative auth mechanism at this time (e.g. delegating to SPIFFE), but the value is in allowing SPIFFE to generate tokens which can be used for OIDC auth.