feat(k8s): add namespace-per-workspace support (RFC 0011 Phase 3)

[derekwaynecarr]

Summary

Implement three workspace namespace modes for the Kubernetes compute driver: shared (default, preserves current single-namespace behavior), managed (auto-creates/deletes namespaces per workspace), and operator (pre-provisioned namespaces with dynamic discovery via label selector or drop-in allowlist file).

Related Issue

Closes#2486

Changes

Key changes:

• WorkspaceMode enum and namespace resolution in driver config
• Managed namespace lifecycle with ServiceAccount and OpenShift SCC annotation propagation
• Cluster-wide sandbox CR watchers for managed/operator modes
• NamespaceValidator (Exact/Prefix/Allowlist) for SA token auth
• Workspace-aware credential secret storage
• Helm ClusterRole for multi-namespace RBAC
• Gateway config, architecture, and reference docs

Testing

• [ x] mise run pre-commit passes
• [ x] Unit tests added/updated
• [ x] E2E tests added/updated (if applicable)

Checklist

• [x ] Follows Conventional Commits
• [ x] Commits are signed off (DCO)
• [ x] Architecture docs updated (if applicable)

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[derekwaynecarr]

/ok to test 628a4b7

[github-actions]

Label test:e2e applied, but pull-request/2656 is at 628a4b7 while the PR head is 6a067bb. A maintainer needs to comment /ok to test 6a067bbb29ef5fdde7d395fd248deef8b9d2ee69 to refresh the mirror. Once the mirror catches up, re-run Branch E2E Checks from the Actions tab.

[derekwaynecarr]

/ok to test 6a067bb

[mrunalp]

Rereview verdict: still request changes

The update fixed:

• Invalid Helm ClusterRole rendering.
• Operator namespace label/file watchers.
• Shared operator allowlist for SA authentication.
• Ownership checks before managed namespace deletion.
• Helm mode validation and several tests.

Remaining blockers

1. P1 — Managed sandboxes are still invisible after creation.
   Managed queries require openshell.ai/gateway-id, but Sandbox CR labels still omit it. selector (

   OpenShell/crates/openshell-driver-kubernetes/src/driver.rs

   Line 796 in 6a067bb

   fnsandbox_lookup_selector(&self,sandbox_id:&str) -> String{

   ), created labels
   (

   OpenShell/crates/openshell-driver-kubernetes/src/driver.rs

   Line 1184 in 6a067bb

   obj.metadata = ObjectMeta{

   ), label helper
   (

   OpenShell/crates/openshell-driver-kubernetes/src/driver.rs

   Line 1589 in 6a067bb

   fnsandbox_labels(sandbox:&Sandbox) -> BTreeMap<String,String>{

   )

2. P1 — Managed/operator namespaces lack the client TLS Secret.
   Pods mount openshell-client-tls, but it exists only in the Helm release namespace. Default TLS-enabled sandboxes cannot start. namespace provisioning
   (

   OpenShell/crates/openshell-driver-kubernetes/src/driver.rs

   Line 615 in 6a067bb

   self.ensure_service_account(&ns_name).await?;

   ), Secret volume
   (

   OpenShell/crates/openshell-driver-kubernetes/src/driver.rs

   Line 3016 in 6a067bb

   if !params.client_tls_secret_name.is_empty(){

   )

3. P1/security — Dynamic namespaces lose SSH isolation.
   The default NetworkPolicy remains only in the static sandbox namespace, leaving port 2222 reachable laterally in workspace namespaces. networkpolicy.yaml:14
   (

   OpenShell/deploy/helm/openshell/templates/networkpolicy.yaml

   Line 14 in 6a067bb

   namespace: {{ include "openshell.sandboxNamespace" . }}

   )

4. P1 — Kubernetes credential storage remains unusable.
   Credentials target workspace namespaces before those namespaces exist, are destroyed when the last sandbox deletes the namespace, and Helm grants Secret access only in the static credential namespace. credential namespace selection
   (

   OpenShell/crates/openshell-driver-kubernetes-secrets/src/lib.rs

   Line 206 in 6a067bb

   namespace:self.settings.target_namespace(&request.workspace),

   )

5. P1/security — Operator reads and watches remain cluster-wide.
   Creation is now allowlisted, but get/list/delete/watch do not filter results by the operator allowlist. One gateway can observe or operate on another gateway’s OpenShell CRs. lookup and selectors
   (

   OpenShell/crates/openshell-driver-kubernetes/src/driver.rs

   Line 780 in 6a067bb

   asyncfnsupported_sandbox_api_for_lookup(

   )

6. P1/security — Managed namespace adoption still trusts names alone.
   A create conflict is accepted without validating ownership, and SA authentication accepts any namespace with the managed prefix. 409 handling
   (

   OpenShell/crates/openshell-driver-kubernetes/src/driver.rs

   Line 598 in 6a067bb

   match tokio::time::timeout(KUBE_API_TIMEOUT, ns_api.create(&PostParams::default(),&ns))

   ), prefix authentication
   (

   OpenShell/crates/openshell-server/src/lib.rs

   Line 463 in 6a067bb

   openshell_driver_kubernetes::WorkspaceMode::Managed => {

   )

Coverage/process

• The new managed/operator E2E tests are exposed as local tasks but are not wired into GitHub Actions; the green Kubernetes jobs ran existing shared-mode tests.
• The current E2E gate is red from rootless Podman.
• Issue feat(kubernetes): map workspaces to sandbox namespaces #2486 (feat(kubernetes): map workspaces to sandbox namespaces #2486) is still state:triage-needed, not accepted.
• Required debug-openshell-cluster skill maintenance remains missing.

[mrunalp]

Rereview — changes still requested

Reviewed the latest head (b9ab04ad). Several earlier issues are fixed, but these blockers remain:

1. P1 – Second sandbox creation fails with TLS enabled

   ensure_tls_secret calls replace without the existing resourceVersion. Kubernetes rejects that update once the Secret already exists.

   OpenShell/crates/openshell-driver-kubernetes/src/driver.rs

   Lines 695 to 748 in b9ab04a

   	let target_api:Api<Secret> = Api::namespaced(self.client.clone(), namespace);
   	let copy = Secret{
   	metadata:ObjectMeta{
   	name:Some(self.config.client_tls_secret_name.clone()),
   	namespace:Some(namespace.to_string()),
   	labels:Some(BTreeMap::from([(
   	LABEL_MANAGED_BY.to_string(),
   	LABEL_MANAGED_BY_VALUE.to_string(),
   	)])),
   	..Default::default()
   	},
   	data: source.data,
   	type_: source.type_,
   	..Default::default()
   	};
   	match tokio::time::timeout(
   	KUBE_API_TIMEOUT,
   	target_api.create(&PostParams::default(),&copy),
   	)
   	.await
   	{
   	Ok(Ok(_)) => {
   	info!(
   	namespace = %namespace,
   	secret = %self.config.client_tls_secret_name,
   	"created TLS secret copy"
   	);
   	}
   	Ok(Err(KubeError::Api(api)))if api.code == 409 => {
   	match tokio::time::timeout(
   	KUBE_API_TIMEOUT,
   	target_api.replace(
   	&self.config.client_tls_secret_name,
   	&PostParams::default(),
   	&copy,
   	),
   	)
   	.await
   	{
   	Ok(Ok(_)) => {
   	debug!(
   	namespace = %namespace,
   	secret = %self.config.client_tls_secret_name,
   	"updated TLS secret copy"
   	);
   	}
   	Ok(Err(e)) => returnErr(KubernetesDriverError::from_kube(e)),
   	Err(_) => {
   	returnErr(KubernetesDriverError::Message(format!(
   	"timeout updating TLS secret in {namespace}"
   	)));
   	}
   	}

2. P1/security – Operator allowlist does not scope reads, watches, or deletes

   Operator mode uses the cluster-wide API, but gateway selectors are only added in managed mode. The gateway can therefore list or delete Sandboxes outside its allowlist.

   OpenShell/crates/openshell-driver-kubernetes/src/driver.rs

   Lines 889 to 921 in b9ab04a

   	asyncfnsupported_sandbox_api_for_lookup(
   	&self,
   	client:Client,
   	) -> Result<AgentSandboxApi,String>{
   	let sandbox_api_version = self.supported_sandbox_api_version(client.clone()).await?;
   	ifself.config.is_multi_namespace(){
   	Ok(Self::cluster_wide_sandbox_api(client, sandbox_api_version))
   	}else{
   	Ok(Self::agent_sandbox_api(
   	client,
   	sandbox_api_version,
   	&self.config.namespace,
   	))
   	}
   	}
   	fnsandbox_lookup_selector(&self,sandbox_id:&str) -> String{
   	letmut selector =
   	format!("{LABEL_MANAGED_BY}={LABEL_MANAGED_BY_VALUE},{LABEL_SANDBOX_ID}={sandbox_id}");
   	ifself.config.workspace_mode == WorkspaceMode::Managed{
   	use std::fmt::Write;
   	write!(selector,",{LABEL_GATEWAY_ID}={}",self.config.gateway_id).unwrap();
   	}
   	selector
   	}
   	fnopenshell_sandbox_selector(&self) -> String{
   	letmut selector = openshell_sandbox_label_selector();
   	ifself.config.workspace_mode == WorkspaceMode::Managed{
   	use std::fmt::Write;
   	write!(selector,",{LABEL_GATEWAY_ID}={}",self.config.gateway_id).unwrap();
   	}
   	selector

3. P1/security – Credential writes bypass the operator allowlist

   EnsureWorkspace validates only managed mode. In operator mode it succeeds without checking the allowlist, after which the credential driver uses the workspace name as the Secret namespace.

   OpenShell/crates/openshell-driver-kubernetes/src/grpc.rs

   Lines 157 to 171 in b9ab04a

   	asyncfnensure_workspace(
   	&self,
   	request:Request<EnsureWorkspaceRequest>,
   	) -> Result<Response<EnsureWorkspaceResponse>,Status>{
   	let workspace = request.into_inner().workspace;
   	if workspace.is_empty(){
   	returnErr(Status::invalid_argument("workspace is required"));
   	}
   	ifself.driver.workspace_mode() == WorkspaceMode::Managed{
   	self.driver
   	.ensure_namespace(&workspace)
   	.await
   	.map_err(|e| Status::internal(e.to_string()))?;
   	}
   	Ok(Response::new(EnsureWorkspaceResponse{}))

4. P1 – Operator credential storage lacks required RBAC

   Operator rendering grants Secret get/create/update, but credential updates use patch and deletion uses delete. Those verbs are currently added only for managed mode.

   OpenShell/deploy/helm/openshell/templates/clusterrole.yaml

   Lines 76 to 96 in b9ab04a

   	{{- if ne $workspaceMode "shared" }}
   	# TLS secret sync: read the source Secret in the release namespace and
   	# create/update copies in workspace namespaces so sandbox pods can mount
   	# client TLS material for mTLS to the gateway.
   	{{- if and (eq $workspaceMode "managed") .Values.server.credentialDrivers.kubernetesSecrets.enabled }}
   	# Managed mode with kubernetes-secrets credential driver: credentials are
   	# stored as Secrets in workspace namespaces, requiring patch+delete in
   	# addition to the TLS sync verbs.
   	{{- end }}
   	- apiGroups:
   	- ""
   	resources:
   	- secrets
   	verbs:
   	- get
   	- create
   	- update
   	{{- if and (eq $workspaceMode "managed") .Values.server.credentialDrivers.kubernetesSecrets.enabled }}
   	- patch
   	- delete
   	{{- end }}

5. P1/security – Dynamic namespaces lack the SSH NetworkPolicy

   The policy is still created only in the static sandbox namespace. Without a policy in each dynamic namespace, workloads elsewhere in the cluster can reach sandbox port 2222.

   OpenShell/deploy/helm/openshell/templates/networkpolicy.yaml

   Lines 4 to 34 in b9ab04a

   	{{- if .Values.networkPolicy.enabled }}
   	# NetworkPolicy restricting SSH ingress on sandbox pods to the gateway pod.
   	# Sandbox pods are dynamically created by the server and labelled with
   	# openshell.ai/managed-by=openshell. This policy ensures only the gateway
   	# (openshell server) pod can reach the sandbox SSH port (2222), blocking
   	# lateral movement from other in-cluster workloads.
   	apiVersion: networking.k8s.io/v1
   	kind: NetworkPolicy
   	metadata:
   	name: {{ include "openshell.fullname" . }}-sandbox-ssh
   	namespace: {{ include "openshell.sandboxNamespace" . }}
   	labels:
   	{{- include "openshell.labels" . | nindent 4 }}
   	spec:
   	podSelector:
   	matchLabels:
   	openshell.ai/managed-by: openshell
   	policyTypes:
   	- Ingress
   	ingress:
   	- from:
   	- namespaceSelector:
   	matchLabels:
   	kubernetes.io/metadata.name: {{ .Release.Namespace }}
   	podSelector:
   	matchLabels:
   	app.kubernetes.io/name: {{ include "openshell.name" . }}
   	app.kubernetes.io/instance: {{ .Release.Name }}
   	ports:
   	- protocol: TCP
   	port: 2222

6. P1 – Existing managed namespaces are adopted without validation

   A namespace-create 409 proceeds directly to ServiceAccount, Secret, and Sandbox creation without verifying ownership. Deletion validation does not make initial adoption safe.

   OpenShell/crates/openshell-driver-kubernetes/src/driver.rs

   Lines 607 to 624 in b9ab04a

   	match tokio::time::timeout(KUBE_API_TIMEOUT, ns_api.create(&PostParams::default(),&ns))
   	.await
   	{
   	Ok(Ok(_)) => {
   	info!(namespace = %ns_name, workspace = %workspace,"created managed namespace");
   	}
   	Ok(Err(KubeError::Api(api)))if api.code == 409 => {
   	debug!(namespace = %ns_name,"managed namespace already exists");
   	}
   	Ok(Err(e)) => returnErr(KubernetesDriverError::from_kube(e)),
   	Err(_) => {
   	returnErr(KubernetesDriverError::Message(format!(
   	"timeout creating namespace {ns_name}"
   	)));
   	}
   	}
   	self.ensure_service_account(&ns_name).await?;

7. P2 – Namespace cleanup remains one-shot

   A terminating Sandbox CR causes cleanup to skip permanently. Cleanup errors are logged only after the workspace record is deleted, leaving no retry path.

   OpenShell/crates/openshell-driver-kubernetes/src/driver.rs

   Lines 761 to 788 in b9ab04a

   	/// Delete the managed namespace if it contains no sandboxes (managed mode
   	/// only). Called via the `DeleteWorkspace` RPC after workspace deletion.
   	pubasyncfndelete_namespace_if_empty(
   	&self,
   	workspace:&str,
   	) -> Result<(),KubernetesDriverError>{
   	let ns_name = managed_namespace(&self.config.gateway_id, workspace);
   	let sandbox_api_version = self
   	.supported_sandbox_api_version(self.client.clone())
   	.await
   	.map_err(KubernetesDriverError::Message)?;
   	let agent_api = Self::agent_sandbox_api(self.client.clone(), sandbox_api_version,&ns_name);
   	let lp = ListParams::default()
   	.labels(&openshell_sandbox_label_selector())
   	.limit(1);
   	let list = tokio::time::timeout(KUBE_API_TIMEOUT, agent_api.api.list(&lp))
   	.await
   	.map_err(|_| {
   	KubernetesDriverError::Message(format!("timeout listing sandboxes in {ns_name}"))
   	})?
   	.map_err(KubernetesDriverError::from_kube)?;
   	if !list.items.is_empty(){
   	debug!(namespace = %ns_name,"namespace still has sandboxes, skipping delete");
   	returnOk(());
   	}

Additional outstanding items:

• Configured image-pull Secrets are not copied into managed namespaces.
• Multi-namespace watches still omit Kubernetes Events.
• The new workspace-mode E2E tasks are not wired into GitHub Actions.
• debug-openshell-cluster remains stale for the new namespace modes.

Addressed since the previous review

• Fixed ClusterRole apiVersion rendering.
• Wired the operator allowlist into authentication and sandbox creation.
• Added gateway-ID labels to managed Sandbox CRs.
• Improved managed credential namespace lifecycle.
• Added partial ownership validation before namespace deletion.
• Fixed the Helm documentation metadata.

Validation

• Kubernetes driver: 168 tests passed.
• Kubernetes credential driver: 21 tests passed.
• Server: 1,288 tests passed.
• Helm lint and managed/operator rendering passed.
• Required PR checks were still pending at review time.

[derekwaynecarr]

/ok to test c655db1

[derekwaynecarr]

/ok to test 11d9c21

[derekwaynecarr]

/ok to test dfd3306

[mrunalp]

Rereview — changes still requested

Reviewed the latest head (dfd33068). Most earlier issues are fixed, but these blockers remain:

1. P1 – Default TLS deployments lack Secret patch RBAC

   TLS synchronization now uses server-side apply (PATCH), but the ClusterRole grants patch only when the Kubernetes credential driver is enabled. Default managed/operator deployments will receive a 403.

   The workspace E2Es do not catch this because their Skaffold values disable TLS, causing the TLS test to skip.

   OpenShell/crates/openshell-driver-kubernetes/src/driver.rs

   Lines 733 to 739 in dfd3306

   	match tokio::time::timeout(
   	KUBE_API_TIMEOUT,
   	target_api.patch(
   	&self.config.client_tls_secret_name,
   	&PatchParams::apply("openshell"),
   	&Patch::Apply(&copy),
   	),

   OpenShell/deploy/helm/openshell/templates/clusterrole.yaml

   Lines 76 to 95 in dfd3306

   	{{- if ne $workspaceMode "shared" }}
   	# TLS secret sync: read the source Secret in the release namespace and
   	# create/update copies in workspace namespaces so sandbox pods can mount
   	# client TLS material for mTLS to the gateway.
   	{{- if .Values.server.credentialDrivers.kubernetesSecrets.enabled }}
   	# kubernetes-secrets credential driver: credentials are stored as Secrets
   	# in workspace namespaces, requiring patch+delete in addition to TLS sync.
   	{{- end }}
   	- apiGroups:
   	- ""
   	resources:
   	- secrets
   	verbs:
   	- get
   	- create
   	- update
   	{{- if .Values.server.credentialDrivers.kubernetesSecrets.enabled }}
   	- patch
   	- delete
   	{{- end }}

2. P1 – Shared-mode upgrades lose existing Sandboxes

   Lookup, list, delete, and watch selectors now always require openshell.ai/gateway-id. Sandbox CRs created before this PR do not have that label, so they become invisible after an upgrade.

   Apply the gateway selector only in multi-namespace modes or provide an upgrade migration/compatibility path.

   OpenShell/crates/openshell-driver-kubernetes/src/driver.rs

   Lines 874 to 880 in dfd3306

   	fnsandbox_lookup_selector(&self,sandbox_id:&str) -> String{
   	sandbox_lookup_selector_for(sandbox_id,&self.config.gateway_id)
   	}
   	fnopenshell_sandbox_selector(&self) -> String{
   	openshell_sandbox_selector_for(&self.config.gateway_id)
   	}

3. P1/security – Dynamic namespaces still lack the SSH NetworkPolicy

   The NetworkPolicy remains limited to the static sandbox namespace. Managed namespaces therefore allow other cluster workloads to reach sandbox port 2222. Namespace separation alone does not prevent cross-namespace traffic.

   OpenShell/deploy/helm/openshell/templates/networkpolicy.yaml

   Lines 4 to 34 in dfd3306

   	{{- if .Values.networkPolicy.enabled }}
   	# NetworkPolicy restricting SSH ingress on sandbox pods to the gateway pod.
   	# Sandbox pods are dynamically created by the server and labelled with
   	# openshell.ai/managed-by=openshell. This policy ensures only the gateway
   	# (openshell server) pod can reach the sandbox SSH port (2222), blocking
   	# lateral movement from other in-cluster workloads.
   	apiVersion: networking.k8s.io/v1
   	kind: NetworkPolicy
   	metadata:
   	name: {{ include "openshell.fullname" . }}-sandbox-ssh
   	namespace: {{ include "openshell.sandboxNamespace" . }}
   	labels:
   	{{- include "openshell.labels" . | nindent 4 }}
   	spec:
   	podSelector:
   	matchLabels:
   	openshell.ai/managed-by: openshell
   	policyTypes:
   	- Ingress
   	ingress:
   	- from:
   	- namespaceSelector:
   	matchLabels:
   	kubernetes.io/metadata.name: {{ .Release.Namespace }}
   	podSelector:
   	matchLabels:
   	app.kubernetes.io/name: {{ include "openshell.name" . }}
   	app.kubernetes.io/instance: {{ .Release.Name }}
   	ports:
   	- protocol: TCP
   	port: 2222

4. P1 – Configured image-pull Secrets are not copied

   Sandbox pods in managed namespaces reference image-pull Secret names configured in the release namespace. Because Secrets are namespace-scoped, private sandbox images cannot be pulled unless those Secrets are separately provisioned in every workspace namespace.

5. P2 – Failed namespace deletion has no retry

   The workspace database record is deleted before platform cleanup. Kubernetes deletion errors are only logged, leaving no durable retry path and potentially leaking the managed namespace permanently.

   Namespace deletion also lacks a UID precondition.

6. P2 – Multi-namespace watches still omit Kubernetes Events

   Managed/operator users do not receive the scheduling and image-pull diagnostic events available in shared mode.

7. Maintenance and process gaps

   • Architecture documentation still says managed namespaces are deleted after the last sandbox rather than on workspace deletion.
   • debug-openshell-cluster still assumes a single static sandbox namespace.
   • Related issue feat(kubernetes): map workspaces to sandbox namespaces #2486 remains labeled state:triage-needed.

Addressed since the previous review

• TLS updates no longer omit resourceVersion.
• Operator credential provisioning enforces the namespace allowlist.
• Operator credential RBAC includes patch/delete when enabled.
• Cluster-wide Sandbox queries are gateway-scoped.
• Existing managed namespaces receive ownership validation.
• Finalizer timing no longer blocks normal workspace deletion.
• Managed/operator E2Es are wired into the required CI gate.

Validation

• Kubernetes driver: 174 tests passed.
• Kubernetes credential driver: 21 tests passed.
• Helm: 77 tests passed.
• Managed workspace E2E passed.
• Operator workspace E2E passed.
• Overall required E2E was still running at review time.

[drew]

gator-agent

PR Review Status

Validation: Project-valid because @derekwaynecarr is a CODEOWNER; implementation review remains required.
Head SHA: dfd33068830456ddca41334ace628c6d6ca8bc4c
Base SHA: 4cb77a900ebd6b789d2b68daaba4830866833b1c
Merge base SHA: 4cb77a900ebd6b789d2b68daaba4830866833b1c
Patch ID: 60c066e46381c7e205f9f952acdecc99461c4c8b
Gator payload: 3
Review mode: initial
Previous reviewed SHA: none

Thanks @mrunalp. I independently checked the exact-head concerns you raised. The TLS RBAC mismatch, shared-mode upgrade regression, missing managed-namespace SSH isolation, image-pull Secret gap, and non-durable namespace cleanup all satisfy the blocking evidence contract. I also checked the multi-namespace Event path and the docs/skill-maintenance notes; they did not meet the blocking contract in this normalized review.

Blocking findings:

• GATOR-dfd33068-01: default TLS-enabled multi-namespace installs lack the Secret patch verb used during every sandbox create.
• GATOR-dfd33068-02: shared-mode upgrades cannot discover legacy Sandbox CRs without the new gateway-ID label.
• GATOR-dfd33068-03: managed namespaces omit the chart's default SSH NetworkPolicy isolation.
• GATOR-dfd33068-04: managed namespaces reference configured image-pull Secrets without provisioning them.
• GATOR-dfd33068-05: workspace deletion discards the durable retry handle before managed namespace cleanup succeeds.

GATOR-dfd33068-04 — managed image-pull Secrets are not provisioned

Invariant: A driver-created managed namespace must contain configured image-pull Secrets before creating pods that reference them.

Prerequisite: An operator enables managed mode and configures server.sandboxImagePullSecrets for a private sandbox or supervisor image.

Entry point → sink: CreateSandbox in managed mode → kubelet image pull for the generated pod.

Changed location: crates/openshell-driver-kubernetes/src/driver.rs:1210. This is summarized here because the reference is an unchanged context line outside GitHub's commentable diff range; the defect is the missing provisioning step in the newly added namespace lifecycle.

Base → head: Shared mode required the named Secret only in the configured sandbox namespace → managed mode creates pods in a new namespace while still injecting the same Secret name, but ensure_namespace creates only the Namespace and ServiceAccount and the new TLS path copies only TLS material.

Impact: Private images enter ImagePullBackOff unless an operator separately pre-populates every dynamically named namespace, defeating managed lifecycle behavior.

Reproducer: Create regcred in the release/sandbox namespace, configure server.sandboxImagePullSecrets[0].name=regcred, enable managed mode, and create a private-image sandbox. The pod references regcred in openshell-<gateway>-<workspace>, where that Secret is absent.

PR ownership: This PR moves managed pods into driver-created namespaces while preserving namespace-local Secret references without provisioning them.

Requested change: Copy configured image-pull Secrets before Sandbox creation, or fail with a clear precondition requiring an operator-managed propagation mechanism; cover the supported private-image path.

Carried findings: None.

Docs: Fern gateway configuration is updated; the blocking runtime gaps above remain.

Next state: gator:in-review

[derekwaynecarr]

/ok to test 0810025

[rhuss]

cc-review Summary

What Went Well

• Fail-closed security posture (config.rs:572-584): Operator mode rejects unknown workspaces by default. NamespaceValidator uses allowlist semantics, ensuring new namespaces are denied until explicitly added.
• Comprehensive validation at config boundaries (config.rs:606-674): validate_workspace_mode() checks gateway_id DNS compliance, namespace length limits, managed SSH ingress completeness, and operator discovery source exclusivity.
• Idempotent namespace lifecycle (driver.rs:654-739): ensure_namespace handles 409 conflicts gracefully by verifying gateway ownership. Prevents namespace hijacking while supporting retry-safe operations.
• Thorough E2E coverage (workspace_namespace_managed.rs): Tests cover namespace creation, label verification, SA provisioning, Secret copying, NetworkPolicy verification, cross-namespace isolation, lifecycle, and DNS-1123 validation.
• Least-privilege RBAC (clusterrole.yaml:76-114): Secret access restricted by resourceNames to only configured TLS and image-pull secrets.

Inline Findings

ID	Severity	File	Description	Source
1	Important	kubernetes-secrets/lib.rs:51	Missing gateway_id validation in secrets driver config	coderabbit
9	Important	config.rs:748	OperatorNamespaceAllowlist panics on poisoned lock	production
14	Important	grpc.rs:228	delete_workspace fails for absent allowlist entries	coderabbit
20	Important	provider_refresh.rs:1121	Provider refresh worker has no shutdown integration	production
2	Minor	config.rs:504	Unused parameter in resolve_sandbox_gid	architecture
4	Minor	config.rs:648	Missing test: operator both label AND file	test-quality
5	Minor	config.rs:572	Missing test: operator None allowlist	test-quality
8	Minor	config.rs:692	DNS validation logic duplicated within crate	architecture
10	Minor	driver.rs:658	Duplicate API handle and silent error swallowing	coderabbit
13	Minor	grpc.rs:243	Fragile string-based error matching	architecture
18	Minor	README.md:14	README misrepresents operator namespace config	coderabbit
25	Minor	managed.rs:64	E2E timestamp modulo allows collisions	test-quality
22	Notable	clusterrole.yaml:76	Secret RBAC could be narrower for operator mode	coderabbit, security

Summary-Only Findings (outside PR diff)

ID	Severity	Location	Description	Source
15	Important	main.rs:231	No graceful shutdown in K8s driver binary. .serve() without shutdown handler; SIGTERM drops in-flight gRPC calls, leaving sandbox resources inconsistent.	production
12	Important	driver.rs:4163	Background label/file watchers spawn infinite-loop tokio tasks with no CancellationToken. Tasks run until runtime is forcefully dropped.	production
3	Minor	config.rs:452	effective_sa_token_ttl_secs has 3 code paths with zero test coverage	test-quality
6	Minor	config.rs:525	from_open_shift_supplemental_groups lacks negative tests (unlike from_open_shift_uid_range)	test-quality
7	Minor	config.rs:219	AppArmorProfile serde deserialization of "Localhost/" untested	test-quality
11	Minor	driver.rs:4209	Namespace label watcher uses fixed 2s retry without backoff or jitter	production
16	Minor	main.rs:210	3 config fields use raw std::env::var instead of clap #[arg] attributes	architecture
17	Minor	driver.rs:109, k8s_sa.rs:44	Sandbox API group/kind/version constants duplicated across crate boundary	architecture
19	Minor	provider_refresh.rs:113	list_all_refresh_states loads all records into unbounded Vec	production
21	Minor	provider_refresh.rs:973	New reqwest::Client created per token request instead of reusing	production

Notable Observations

• Handler-level auth inconsistency (workspace.rs:142,287): handle_create_workspace and handle_delete_workspace skip handler-level extract_principal(), relying solely on proto-level descriptor_authz annotations. Mitigated but inconsistent with other workspace handlers. (security)
• Rate limiting disabled by default (values.yaml): gRPC rate limiting defaults to disabled. Deployments exposed to untrusted clients should enable it. (security)

Review Details

• Findings posted: 25 (13 inline, 12 in summary body)
• Findings reviewed and not posted: 0
• Gate outcome: FAIL (6 Important findings)
• Participating agents: correctness, architecture, security, production, test-quality, goal-alignment, coderabbit

[derekwaynecarr]

/ok to test bac95bf

[derekwaynecarr]

/ok to test 6699658