feat(helm): cert-manager external issuer + OpenShift passthrough Route by jhjaggars · Pull Request #2468 · NVIDIA/OpenShell

jhjaggars · 2026-07-24T15:32:14Z

Summary

Lets cert-manager issue the gateway's server TLS certificate from a real Issuer/ClusterIssuer (for example an ACME issuer) instead of only the chart's built-in self-signed CA, and adds an OpenShift Route template with TLS passthrough so the gateway can be exposed externally with a publicly-trusted certificate while it keeps terminating its own TLS/mTLS.

Closes#2466

Changes

Dual-certificate architecture (SNI-based)

The gateway server certificate is split into two: an internal cert issued by the chart's own CA (for supervisor connections via cluster-local SANs) and an external cert issued by an operator-configured Issuer such as ACME/Let's Encrypt (for CLI and Route access via public SANs). The gateway uses SNI-based certificate selection (DualCertResolver): connections whose SNI hostname matches external_server_names receive the external cert; all others receive the internal cert. Wildcard patterns (e.g. *.example.com) are supported per RFC 6125.

Security: supervisor trust pinned to chart CA

The sandbox supervisor's gRPC client previously trusted WebPKI roots (tls-webpki-roots). This PR removes that trust and keeps supervisors pinned exclusively to the chart CA configured via OPENSHELL_TLS_CA. The UnknownCA problem (supervisor connecting to a gateway serving an ACME cert) is solved server-side: supervisors connect via internal service names, the SNI resolver presents the internal (chart CA) cert, and the TLS handshake succeeds without needing public root trust.

This is a deliberate security improvement — a user-supplied container image that ships its own root store can no longer influence the supervisor's trust decisions, closing a MITM vector where a publicly-trusted cert for an attacker-controlled name could intercept the sandbox JWT.

All four compute drivers (Docker, Podman, VM, Kubernetes) now strip OPENSHELL_GATEWAY_TLS_SERVER_NAME from the sandbox environment to prevent user-supplied overrides of the TLS server name the supervisor verifies.

Helm templates

certManager.serverIssuerRef (new value): creates a second server Certificate from the operator's Issuer/ClusterIssuer with only externally-resolvable SANs. The internal server cert always uses the chart's own CA issuer.
certManager.serverDnsNames entries are validated: internal-only names are rejected when an external issuer is configured.
New templates/route.yaml: OpenShift Route with tls.termination: passthrough, gated by openshiftRoute.enabled. Includes fail guards for TLS disabled and Route host not covered by serverDnsNames.
Empty serverDnsNames with an external issuer is rejected at install time.

Docs

Production section in docs/kubernetes/openshift.mdx with full helm install command.
New section in docs/kubernetes/managing-certificates.mdx documenting the dual-cert architecture.
docs/reference/gateway-config.mdx updated with external_cert_path, external_key_path, external_server_names TOML fields.
.agents/skills/debug-openshell-cluster/SKILL.md updated with dual-cert troubleshooting.

Testing

mise run pre-commit passes
Unit tests for DualCertResolver SNI selection (exact + wildcard), build_cert_resolver validation (partial config, empty names), and full TCP+TLS integration test
Parity tests for GATEWAY_TLS_SERVER_NAME stripping across Docker, Podman, and VM drivers
Helm lint passes all CI value variants
Helm unittest coverage for Route template and cert-manager PKI templates

Validated end-to-end against a live OpenShift (ROSA) cluster: passthrough Route serving a real Let's Encrypt certificate issued via a Route53 DNS-01 ClusterIssuer, OIDC-authenticated CLI access via Keycloak, sandbox created and exec'd successfully with the supervisor connecting back to the gateway over the internal (chart CA) TLS path.

Checklist

mise run pre-commit passes
Unit tests added/updated
E2E tests added/updated — validated manually against a live OpenShift cluster
Follows Conventional Commits
Commits are signed off (DCO)

copy-pr-bot · 2026-07-24T15:32:17Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

mrunalp · 2026-07-30T14:55:56Z

/ok to test cf02477

mrunalp · 2026-07-31T23:26:11Z

Blocking

1. Native roots are controlled by the sandbox image

crates/openshell-core/src/grpc_client.rs:176-180

The supervisor now calls both with_native_roots() and with_webpki_roots(). For Docker and Podman, the supervisor binary runs inside the user-selected sandbox image. The image's CA bundle is therefore not an operator-controlled trust source. rustls-native-certs also honors SSL_CERT_FILE and SSL_CERT_DIR, and both drivers pass user-provided template/spec environment variables into the supervisor.

If an attacker can also influence DNS or routing for the gateway connection, they can install their own CA, present a certificate for the gateway hostname, and receive the sandbox JWT that the supervisor sends on its RPCs. That can expose sandbox-scoped data and let the fake gateway return attacker-selected policy during supervisor startup.

There is also a compatibility regression: tonic 0.14 returns NativeCertsNotFound when the native store is empty before it considers the configured CA or WebPKI roots. A minimal BYOC image can therefore fail to start even though OPENSHELL_TLS_CA is valid.

Please remove with_native_roots() here and use the explicitly configured CA plus the compiled-in WebPKI roots. If enterprise/private server issuers need support, add an operator-mounted server CA bundle rather than trusting the sandbox image's native store. It would also be prudent to reserve or scrub SSL_CERT_FILE, SSL_CERT_DIR, and OPENSHELL_GATEWAY_TLS_SERVER_NAME from supervisor input.

2. `serverIssuerRef` renders a broken default client-CA configuration

deploy/helm/openshell/values.yaml:351-369
deploy/helm/openshell/templates/_gateway-workload.tpl:147-157

Setting the advertised serverIssuerRef and an external SAN while leaving the other defaults unchanged keeps clientCaFromServerTlsSecret=true. The gateway then tries to mount ca.crt from the public server certificate Secret.

ACME Secrets commonly do not contain ca.crt, leaving the gateway pod in MountVolume.SetUp failed. If an issuer does populate it, it is still not the CA that signed the chart-issued supervisor client certificate.

The chart should fail rendering when serverIssuerRef.name is combined with clientCaFromServerTlsSecret=true, or derive the correct client CA source automatically. Please add a negative Helm test for the default interaction.

3. `clientIssuerRef` is not independently usable as documented

deploy/helm/openshell/templates/cert-manager-pki.yaml:138-147
docs/kubernetes/managing-certificates.mdx:109-111

Changing only clientIssuerRef leaves the gateway trusting the built-in server CA while the supervisor receives the custom client issuer's CA. The gateway does not trust the new client certificate, and the supervisor no longer trusts the default server certificate. Every supervisor TLS connection consequently fails.

Either remove this option for now or model the server-verification and client-verification trust bundles separately and validate the complete configuration. kind should also be required or default to cert-manager's standard Issuer; silently defaulting to ClusterIssuer is surprising for an interface that advertises both.

4. The documented OIDC production command cannot connect

docs/kubernetes/openshift.mdx:102-135
crates/openshell-server/src/cli.rs:256-301

The Helm command never sets server.oidc.issuer, but the next step registers the CLI using --oidc-issuer. That flag configures only the CLI. Without server-side OIDC, the gateway requires a client certificate and rejects the OIDC-only CLI during the TLS handshake.

Please add server.oidc.issuer (and the expected audience) to the Helm command, or make completing the Access Control configuration an explicit prerequisite before registration.

5. The passthrough Route accepts a plaintext backend

deploy/helm/openshell/templates/route.yaml:4-26

openshiftRoute.enabled=true can currently be combined with server.disableTls=true, including by adding the Route settings to the existing OpenShift quickstart. Helm succeeds, but HAProxy forwards TLS handshake bytes to a plaintext gateway, resulting in resets or hangs.

The template should fail when passthrough routing is enabled while gateway TLS is disabled, with a test covering that combination.

…ssuer Addresses all five blocking review items from NVIDIA#2468: 1. Remove .with_native_roots() from supervisor gRPC client -- the supervisor runs inside the user-selected sandbox image, so the image CA bundle is not operator-controlled. Keep .with_webpki_roots() (compiled-in, not user-controlled) alongside the configured CA. 2. Fail at render time when serverIssuerRef.name is set but clientCaFromServerTlsSecret is still true. Add negative Helm test. 3. Remove clientIssuerRef -- changing only clientIssuerRef breaks both directions because trust bundles are not modeled separately. Change serverIssuerRef.kind default from ClusterIssuer to Issuer. 4. Add server.oidc.issuer and server.oidc.audience to the documented OpenShift production Helm command. Add Access Control prerequisite. 5. Fail at render time when openshiftRoute.enabled and disableTls are both true. Add negative Helm test. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

mrunalp · 2026-08-04T18:45:27Z

Follow-up review: all five blocking items addressed

Re-reviewed at 0276980. All five blocking items from my earlier review are addressed. I verified each against the code rather than the commit message and ran the chart and crate checks.

#	Item	Fix	Verification
1	Native roots controlled by the sandbox image	`.with_native_roots()` removed from `grpc_client.rs`; `tls-native-roots` dropped from `openshell-core/Cargo.toml`; the comment explains why it should not come back	`cargo check -p openshell-core --all-targets` clean
2	`serverIssuerRef` + `clientCaFromServerTlsSecret=true`	`fail` guard in `cert-manager-pki.yaml:46` plus a negative unittest	`helm template` with that combination errors with the expected message
3	`clientIssuerRef` not independently usable	Value removed from `values.yaml`, the template, the chart README, and the docs; `serverIssuerRef.kind` now defaults to `Issuer`	no remaining value or template references
4	Documented OIDC command cannot connect	`server.oidc.issuer` / `server.oidc.audience` added to the production Helm command, Access Control called out as a prerequisite, override table row added	both keys exist in `values.yaml` and render into `[openshell.gateway.oidc]`
5	Passthrough Route with a plaintext backend	`fail` guard in `route.yaml:5` plus a negative unittest	`helm template` with `openshiftRoute.enabled=true,server.disableTls=true` errors as intended

mise run helm:test passes 74/74 across 7 suites, the CI values overlay renders, and the chart still creates the internal openshell-ca-tls CA that the docs point clientCaSecretName at.

Remaining

1. The env-scrubbing half of item 1 is still open

crates/openshell-core/src/grpc_client.rs:188
crates/openshell-driver-docker/src/lib.rs:2147-2152
crates/openshell-driver-kubernetes/src/driver.rs:1982

OPENSHELL_GATEWAY_TLS_SERVER_NAME still overrides the hostname the supervisor verifies. The Kubernetes driver strips it from container env, but the Docker driver merges template.environment and spec.environment straight into the supervisor's environment and never sets or removes it. Podman is the same.

Now that the WebPKI roots are trusted, a sandbox user who can also redirect the gateway hostname inside the container can satisfy verification with a publicly valid certificate for a name they control and receive the sandbox JWT. Before this change, only the configured CA was trusted, so redirection alone was not enough.

Please strip the variable in the Docker and Podman drivers to match what the Kubernetes driver already does. SSL_CERT_FILE and SSL_CERT_DIR no longer matter for this path now that rustls-native-certs is gone from it.

2. Dangling `clientIssuerRef` reference in the docs

docs/kubernetes/managing-certificates.mdx:102

The page still reads "unless you've also overridden clientIssuerRef". That value no longer exists.

3. A half-configured client CA still renders (optional)

deploy/helm/openshell/values.yaml:248

Setting clientCaFromServerTlsSecret=false without server.tls.clientCaSecretName falls back to the default openshell-server-client-ca, which nothing in the chart creates. The pod then lands in MountVolume.SetUp failed — the same symptom item 2 was about. The new guard's message does tell operators to set both, so this is documented but unguarded. Worth a follow-up rather than a blocker.

CI

/ok to test was granted for cf02477, not the current head — Branch Checks and Helm Lint are still waiting on the mirror, so the fix commit has not been through CI yet.

mrunalp · 2026-08-04T18:46:06Z

/ok to test 0276980

… default clientCaSecretName, remove stale clientIssuerRef doc ref Address remaining review feedback from NVIDIA#2468: 1. Strip OPENSHELL_GATEWAY_TLS_SERVER_NAME from Docker and Podman driver supervisor environments to match the K8s driver. With WebPKI roots trusted, leaving this user-controllable would let an attacker inside the sandbox redirect TLS verification to a hostname they control. 2. Remove dangling 'clientIssuerRef' reference from docs/kubernetes/managing-certificates.mdx — the value was removed from the chart in the prior fix commit. 3. Add a fail guard in cert-manager-pki.yaml when clientCaFromServerTlsSecret=false but clientCaSecretName is still the default (openshell-server-client-ca), which nothing creates under cert-manager. Includes a negative Helm unittest. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

…ssuer Addresses all five blocking review items from NVIDIA#2468: 1. Remove .with_native_roots() from supervisor gRPC client -- the supervisor runs inside the user-selected sandbox image, so the image CA bundle is not operator-controlled. Keep .with_webpki_roots() (compiled-in, not user-controlled) alongside the configured CA. 2. Fail at render time when serverIssuerRef.name is set but clientCaFromServerTlsSecret is still true. Add negative Helm test. 3. Remove clientIssuerRef -- changing only clientIssuerRef breaks both directions because trust bundles are not modeled separately. Change serverIssuerRef.kind default from ClusterIssuer to Issuer. 4. Add server.oidc.issuer and server.oidc.audience to the documented OpenShift production Helm command. Add Access Control prerequisite. 5. Fail at render time when openshiftRoute.enabled and disableTls are both true. Add negative Helm test. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

… default clientCaSecretName, remove stale clientIssuerRef doc ref Address remaining review feedback from NVIDIA#2468: 1. Strip OPENSHELL_GATEWAY_TLS_SERVER_NAME from Docker and Podman driver supervisor environments to match the K8s driver. With WebPKI roots trusted, leaving this user-controllable would let an attacker inside the sandbox redirect TLS verification to a hostname they control. 2. Remove dangling 'clientIssuerRef' reference from docs/kubernetes/managing-certificates.mdx — the value was removed from the chart in the prior fix commit. 3. Add a fail guard in cert-manager-pki.yaml when clientCaFromServerTlsSecret=false but clientCaSecretName is still the default (openshell-server-client-ca), which nothing creates under cert-manager. Includes a negative Helm unittest. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

mrunalp · 2026-08-05T17:35:28Z

Follow-up review at `876959a1`

All three remaining items from my previous comment are addressed and verified. One stray file needs to come out of the commit before this merges.

Verified fixed

1. OPENSHELL_GATEWAY_TLS_SERVER_NAME stripping.environment.remove(...) in crates/openshell-driver-docker/src/lib.rs:2256 and crates/openshell-driver-podman/src/container.rs:487, both placed after the template.environment / spec.environment merge, so user-supplied env cannot win. Both are single funnels — docker's build_environment delegates to build_environment_for_oci_user, and podman's build_env has one caller — so there is no bypass path. cargo test -p openshell-driver-docker -p openshell-driver-podman passes (107 + 149).

2. Dangling doc reference. The clientIssuerRef clause is gone from docs/kubernetes/managing-certificates.mdx:98-101, and no references remain anywhere in the tree.

3. Half-configured client CA. New fail guard at deploy/helm/openshell/templates/cert-manager-pki.yaml:49 plus a negative unittest. I exercised the matrix: the default install renders, clientCaFromServerTlsSecret=false with the default secret name fails with the new message, clientCaSecretName=openshell-ca-tls renders, and clientCaSecretName="" (mTLS off) renders. The guard is correctly scoped inside if .Values.certManager.enabled. mise run helm:test passes 77/77 across 7 suites.

One behavior note on that guard: it hard-fails a combination that previously rendered (certManager.enabled + clientCaFromServerTlsSecret=false + the default secret name), which is what forced the statefulset_client_ca_test.yaml change. The old expectation was wrong — with cert-manager enabled the certgen hook runs in JWT-only mode and never creates openshell-server-client-ca — so the guard is correct. But any existing release sitting on that combination will now fail its next helm upgrade, so this is worth a release note.

Please fix: a `target` symlink was committed

876959a1 adds a symlink at the repository root:

120000 blob e58e2c52 target -> /mnt/build-artifacts/cargo-target

.gitignore has /target/ and target/; the trailing slash makes both directory-only, so neither matches a symlink named target — that is how this slipped through. Checking the branch out materializes the symlink, and /mnt/build-artifacts does not exist outside your machine, so every clone gets a dangling redirect for all cargo builds. Please git rm target, and consider adding a bare target line to .gitignore so the pattern also catches the non-directory case.

Minor

There is no regression test for the env stripping in either driver, though crates/openshell-driver-docker/src/tests.rs already has the pattern for exactly this kind of assertion (build_environment_protects_oci_identity_metadata:574, build_environment_uses_token_file_without_raw_token_env:1422). For a security-relevant strip, a small test in both drivers would keep it from silently regressing.

CI

The head is now 876959a1; the /ok to test above was for 0276980. Branch Checks and Helm Lint still read "Waiting for /ok to test mirror".

…ssuer Addresses all five blocking review items from NVIDIA#2468: 1. Remove .with_native_roots() from supervisor gRPC client -- the supervisor runs inside the user-selected sandbox image, so the image CA bundle is not operator-controlled. Keep .with_webpki_roots() (compiled-in, not user-controlled) alongside the configured CA. 2. Fail at render time when serverIssuerRef.name is set but clientCaFromServerTlsSecret is still true. Add negative Helm test. 3. Remove clientIssuerRef -- changing only clientIssuerRef breaks both directions because trust bundles are not modeled separately. Change serverIssuerRef.kind default from ClusterIssuer to Issuer. 4. Add server.oidc.issuer and server.oidc.audience to the documented OpenShift production Helm command. Add Access Control prerequisite. 5. Fail at render time when openshiftRoute.enabled and disableTls are both true. Add negative Helm test. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

mrunalp · 2026-08-05T21:57:27Z

Follow-up review at `aac65a99`

The branch was force-rewritten since my last pass (all commit hashes changed). The target symlink is gone, but the same rewrite also dropped one of the three fixes I confirmed last time.

Present and verified

Env stripping.environment.remove(GATEWAY_TLS_SERVER_NAME) in crates/openshell-driver-docker/src/lib.rs:2257 and crates/openshell-driver-podman/src/container.rs:490, both after the user-env merge. cargo test -p openshell-driver-docker -p openshell-driver-podman passes (107 + 149).
Dangling doc reference.clientIssuerRef no longer appears anywhere in the tree.
target symlink. Removed — checking the branch out no longer materializes it.

Everything from the earlier rounds also survived: with_native_roots() is still absent (only the do-not-re-add comment remains), openshell-core's tonic features are ["channel", "tls-webpki-roots"], the route.yaml TLS guard fires, the serverIssuerRef + clientCaFromServerTlsSecret=true guard fires, and the openshift.mdx OIDC additions are intact. mise run helm:test passes 76/76.

Regressed: the default `clientCaSecretName` guard is gone

The fail guard that was at deploy/helm/openshell/templates/cert-manager-pki.yaml:49 and its negative unittest ("fails when clientCaFromServerTlsSecret is false but clientCaSecretName is the default") are no longer in the branch, and tests/statefulset_client_ca_test.yaml is back to unchanged from main. Only the two original guards remain in that template. The helm test count went 74 → 77 → 76, which is that one test disappearing.

The footgun is back:

helm template ... --set certManager.enabled=true --set certManager.clientCaFromServerTlsSecret=false
-> secretName: openshell-server-client-ca # renders fine; nothing creates this Secret

That leaves the gateway pod in the MountVolume.SetUp failed state the original item 2 was about.

This may be deliberate — my last comment noted the guard would break helm upgrade for any release sitting on that combination, and dropping it is a defensible response. If so, nothing replaced it: the only thing steering operators now is the prose in managing-certificates.mdx plus the other guard's error message. Could you confirm whether this was an intentional revert or collateral damage from the force-push? It came out in the same rewrite that removed the stray symlink.

Still minor

No regression test for the env stripping in either driver. crates/openshell-driver-docker/src/tests.rs already has the pattern for it (build_environment_protects_oci_identity_metadata, build_environment_uses_token_file_without_raw_token_env).

CI

The head is now aac65a99. Branch Checks and Helm Lint still read "Waiting for /ok to test mirror".

mrunalp · 2026-08-06T19:46:05Z

/ok to test 85ec214

github-actions · 2026-08-07T19:47:47Z

All contributors have signed the DCO ✍️ ✅
_{Posted by the DCO Assistant Lite bot.}

mrunalp · 2026-08-12T20:54:45Z

Blocking

1. The old client-CA guard is now incorrect

Files:

deploy/helm/openshell/templates/cert-manager-pki.yaml:45
deploy/helm/openshell/templates/_gateway-workload.tpl:157

The internal openshell-server-tls certificate is now signed by the same chart CA that signs supervisor client certificates. Therefore, the default certManager.clientCaFromServerTlsSecret=true is correct again: its filtered ca.crt is exactly what the gateway should use to verify supervisors.

The template still rejects this configuration using an assumption from the previous single-certificate design. The documented workaround sets clientCaFromServerTlsSecret=false and mounts openshell-ca-tls without an items filter. Because that CA Secret contains its signing key, this unnecessarily exposes the CA
private key to the gateway container.

Please remove the obsolete guard and the two client-CA overrides from the production examples. Keep the existing filtered ca.crt projection from the internal server Secret.

2. Route validation conflicts with wildcard certificates

Files:

deploy/helm/openshell/templates/route.yaml:8
crates/openshell-server/src/tls.rs:289

The gateway supports wildcard names such as *.example.com, but the Helm guard uses exact list membership. It rejects gateway.example.com even though *.example.com covers it.

External-issuer mode also permits an empty Route host. OpenShift then generates a hostname absent from external_server_names, causing the gateway to serve its internal certificate.

Please:

Require an explicit Route host when an external issuer is configured.
Validate exact and single-level wildcard coverage consistently with the gateway.
Add tests for wildcard coverage and a missing host.

3. serverIssuerRef is not coupled to certManager.enabled

Files:

deploy/helm/openshell/templates/cert-manager-pki.yaml:4
deploy/helm/openshell/templates/_gateway-workload.tpl:87
deploy/helm/openshell/templates/gateway-config.yaml:84

Certificate creation requires certManager.enabled, but the external Secret mount and gateway configuration require only serverIssuerRef.name.

Setting serverIssuerRef.name with certManager.enabled=false therefore creates a workload referencing an external certificate Secret that nothing creates.

Please reject this combination or consistently gate all external-certificate consumers on both values.

Follow-ups

The integration test signs both certificates with the same CA. It verifies SNI selection but not the intended trust separation. Distinct internal and external test CAs would exercise the actual security boundary.
values.yaml and the certificate documentation still describe serverIssuerRef as replacing the server certificate. It now creates a second certificate.
architecture/gateway.md should document the SNI-based dual-certificate boundary.
Add the new Helm CI values overlay to the helm-dev-environment skill’s key-file table.

mrunalp · 2026-08-13T14:28:52Z

Needs rebase

Remaining concern

.agents/skills/debug-openshell-cluster/SKILL.md:304-310 is now stale. It still claims the chart rejects clientCaFromServerTlsSecret=true and instructs agents to set it to false and mount openshell-ca-tls—the exact CA-key exposure the latest commit fixes.

I would request one final correction to that skill before approval. The architecture and Helm skill updates can be included in the same small documentation commit.

Verification:

mise run helm:test: 81/81 passed
mise run helm:lint: all variants passed
External-issuer render confirmed tls-client-ca projects only ca.crt
git diff --check: clean
GitHub Branch Checks and Helm Lint remain pending /ok to test for 93c3473

The supervisor gRPC client only trusted the CA configured via OPENSHELL_TLS_CA, since tonic ClientTlsConfig starts with an empty root store unless with_native_roots()/with_webpki_roots() is also enabled. Deployments where the gateway server certificate is issued by a public CA (e.g. cert-manager against an ACME issuer) caused every supervisor connection to fail the TLS handshake with "UnknownCA", since the sandbox mTLS CA and the server cert issuer were no longer the same. Enable both native and webpki roots in addition to the configured CA. tonic root store is a union of all configured sources, so this does not weaken verification for existing self-signed deployments. webpki-roots (compiled in) is enabled alongside native-roots since the supervisor binary may run in minimal sandbox images without a populated system CA bundle. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

… passthrough Add certManager.serverIssuerRef/clientIssuerRef so the gateway and mTLS client certificates can be issued by a real Issuer/ClusterIssuer (e.g. ACME) instead of only the chart built-in self-signed CA. Add openshiftRoute template for exposing the gateway via a TLS passthrough Route so the gateway keeps terminating its own TLS/mTLS. The server Certificate excludes internal-only SANs (cluster-local, localhost, loopback) when an external issuer is configured, since ACME issuers reject those per CA/Browser Forum baseline requirements. A template-time fail guard catches the misconfiguration at helm install time rather than asynchronously at cert-manager issuance time. Includes Helm unittest coverage for both issuerRef overrides and Route rendering, plus a CI values overlay for lint coverage. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

Update managing-certificates.mdx with the serverIssuerRef workflow and install-time validation behavior. Add a production section to the OpenShift guide covering passthrough Route with a real certificate. Regenerate Helm README for new certManager and openshiftRoute values. Sync debug-openshell-cluster skill with new troubleshooting steps for ACME issuance failures and supervisor UnknownCA from mismatched CAs. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

…ssuer Addresses all five blocking review items from NVIDIA#2468: 1. Remove .with_native_roots() from supervisor gRPC client -- the supervisor runs inside the user-selected sandbox image, so the image CA bundle is not operator-controlled. Keep .with_webpki_roots() (compiled-in, not user-controlled) alongside the configured CA. 2. Fail at render time when serverIssuerRef.name is set but clientCaFromServerTlsSecret is still true. Add negative Helm test. 3. Remove clientIssuerRef -- changing only clientIssuerRef breaks both directions because trust bundles are not modeled separately. Change serverIssuerRef.kind default from ClusterIssuer to Issuer. 4. Add server.oidc.issuer and server.oidc.audience to the documented OpenShift production Helm command. Add Access Control prerequisite. 5. Fail at render time when openshiftRoute.enabled and disableTls are both true. Add negative Helm test. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

… tests Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

…r TLS Split the gateway server certificate into two: an internal cert issued by the chart's own CA (for supervisor connections via cluster-local SANs) and an external cert issued by an operator-configured Issuer such as ACME/Let's Encrypt (for CLI and Route access via public SANs). The gateway uses SNI-based certificate selection: connections whose SNI hostname matches external_server_names receive the external cert; all others (including those with no SNI) receive the internal cert. Security improvement: remove .with_webpki_roots() from the supervisor gRPC client so supervisors trust only the chart CA, closing a MITM vector via publicly-trusted certificates in user-supplied container images. Key changes: - Add DualCertResolver with SNI-based cert selection and full test coverage - Add external_cert_path, external_key_path, external_server_names to TlsConfig - Validate partial external cert config (error on cert-without-key or vice versa) - Validate empty external_server_names when external cert is configured - Split cert-manager templates into internal + external Certificate resources - Add Helm guards for misconfigured external issuer (empty serverDnsNames, internal-only SANs with external issuer, conflicting clientCaFromServerTlsSecret) - Update gateway-config.mdx, managing-certificates.mdx, openshift.mdx docs - Update debug-openshell-cluster skill for dual-cert troubleshooting Signed-off-by: Pi Agent <agent@openshell.local>

…comments Add the same GATEWAY_TLS_SERVER_NAME environment stripping to the VM compute driver that Docker, Podman, and Kubernetes drivers already perform. Without this, a sandbox user on the VM driver could override the TLS server name the supervisor verifies. Fix stale comments in Docker and Podman drivers that referenced 'with WebPKI roots trusted' — WebPKI roots are explicitly not trusted after the tls-webpki-roots removal. Use tls-ring instead of bare channel for tonic in openshell-core so the TLS API (ClientTlsConfig, Endpoint::tls_config) is available without pulling in any root certificate store. Signed-off-by: Pi Agent <agent@openshell.local>

Add RFC 6125 single-level wildcard matching to DualCertResolver so external_server_names entries like *.example.com correctly match SNI hostnames like gw.example.com. Previously only exact matches worked, silently falling back to the internal cert for wildcard configurations. Add a Helm fail guard in route.yaml that rejects openshiftRoute.host values not listed in certManager.serverDnsNames when an external issuer is configured — catches cert/route hostname mismatches at install time instead of at TLS connect time. Quote the host field in route.yaml for robustness. Signed-off-by: Pi Agent <agent@openshell.local>

…Route, serverIssuerRef gate 1. Remove the obsolete guard rejecting serverIssuerRef + clientCaFromServerTlsSecret=true. The internal server certificate is always signed by the chart CA (the same CA that signs the client cert), so clientCaFromServerTlsSecret=true is correct — its filtered ca.crt is exactly the right trust anchor. The old workaround (mounting openshell-ca-tls directly) unnecessarily exposed the CA private key to the gateway container. Remove the client-CA overrides from docs, CI overlay, and production examples. 2. Route host validation now supports wildcard certificates per RFC 6125: single-level wildcards like *.example.com match gateway.example.com but not deep.sub.example.com. Require an explicit openshiftRoute.host when an external issuer is configured — without one, OpenShift generates a hostname absent from serverDnsNames. 3. Reject serverIssuerRef.name when certManager.enabled is false — the external certificate, its Secret mount, and the gateway TLS config all require cert-manager to be enabled. Validated on ROSA (dev.dyee.p3) with branch-built images: - Fresh install with letsencrypt-prod ClusterIssuer - SNI dual-cert: external hostname served Let's Encrypt cert - Supervisor mTLS via internal cert path: ConnectSupervisor accepted - Client CA volume: filtered ca.crt from internal server secret (no key) - CLI connected via Route + OIDC Helm tests: 81 pass across 7 suites. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

mrunalp · 2026-08-13T19:23:43Z

The blocking concern is addressed at 74de2a6: the debug skill now keeps clientCaFromServerTlsSecret=true and verifies the filtered ca.crt mount (source
(

OpenShell/.agents/skills/debug-openshell-cluster/SKILL.md

Lines 320 to 331 in 74de2a6

 Less commonly, `UnknownCA` can occur if the gateway's client-verification CA 
 is misconfigured. The default `clientCaFromServerTlsSecret=true` is correct 
 for all configurations — the internal server certificate is always signed by 
 the chart CA (the same CA that signs the client cert), so its `ca.crt` is 
 the right trust anchor. Only override this if you intentionally mount a 
 separate client CA via `server.tls.clientCaSecretName`. Verify the mounted 
 client CA matches the CA that signed the client certificate: 
```bash
 kubectl -n openshell get statefulset openshell -o jsonpath='{.spec.template.spec.volumes[?(@.name=="tls-client-ca")]}'| jq .
# Should show items filter for ca.crt from openshell-server-tls
```

)).

Status:

All three functional blockers remain fixed.
Rebase conflict is resolved; GitHub reports the PR mergeable.
Helm tests: 93/93 passed across 8 suites.
Helm lint: all variants passed.
git diff --check: clean.
Branch Checks and Helm Lint still await /ok to test.

Not all follow-ups were addressed:

architecture/gateway.md still lacks the dual-certificate/SNI trust boundary.
helm-dev-environment still omits values-openshift-route-cert-manager.yaml.

mrunalp · 2026-08-13T19:24:09Z

/ok to test 74de2a6

…e-openshell skill Add skills/tooling/update-openshell - a repeatable, self-reinforcing skill for syncing HyperShell to upstream OpenShell releases (they ship ~daily). It bumps the version-pin footprint, triages release notes for contract-affecting changes, verifies build/config, and folds each run's lessons back into the skill + specs. First run: update 0.0.101 -> 0.0.106. - Bump defaultGatewayImage/defaultSupervisorImage (source of truth) and every copy across specs, the ROKS e2e script, kind lib, and the ibm-cluster skill. - Normalize specs/platform/openshell-gateway-database.spec.md, which pinned the gateway image to a git SHA instead of a release tag. - Preserve fixtures/historical refs (the validation_test.go regex fixture and the "v0.0.101 introduced credential drivers" historical sentence). Triage 102-106: mechanically safe pin bump. One needs-decision item recorded as a follow-up in the skill's learnings log - upstream v0.0.106 shipped a cert-manager external issuer + OpenShift passthrough Route (NVIDIA/OpenShell#2468) that overlaps HyperShell's hand-rolled per-tenant self-signed CA; evaluate adopting it separately. Also repair pre-existing "make check" failures from earlier spec commits on this branch: replace forbidden em dashes (U+2014) with " - " and refresh the line-number-based forbidden-term whitelist (Mermaid ACP nodes + vteam path) after the Terminology block shifted line numbers. Register /update-openshell in CLAUDE.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* spec: add global architecture specification Captures deployment patterns (single-node, global multi-region, multi-cloud), tooling stack decisions (CNPG, ArgoCD, Tekton, Vault, Terraform, Prometheus), namespace strategy, installer pipeline requirements, and monitoring architecture from the Aug 10 architecture meeting. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * docs: add Ingress Architecture section detailing dual-ingress strategy * docs: resolve open questions regarding IBM Gateway API availability, VPC LB scope, and control-plane reconciler bug * docs: add shared gateway rationale to Design Decisions table * docs(skills): add IBM Cloud Hub provisioning and shared-gateway ingress skills Add two deploy skills and cross-link them from deploy-cluster and CLAUDE.md: - cloud-hub-ingress-bootstrap: cloud-agnostic shared Gateway + wildcard DNS/TLS bootstrap (AWS reference / IBM parity). Encodes the OCP >= 4.19 requirement for the built-in openshift-default GatewayClass and the nlb-dns DNS+TLS path. - ibm-cluster: ROKS VPC Gen2 provisioning mirroring the reference cluster, the cluster-create command (COS CRN required, not GUID), and a registry-storage decision table (emptyDir / PVC / COS) with PVC as the chosen persistent backend. - deploy-cluster: add Cloud-Hub parameter overrides (registry host, ibmc-vpc-block storage class) and scope note pointing at the ingress bootstrap. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * spec: record IBM Gateway API version requirement in global architecture The IBM Cloud Parity Plan now documents the root cause of the tenant-gateway ingress gap: the built-in CIO-managed Gateway API (openshift-default GatewayClass) is GA only on OCP >= 4.19, and the original hypershell-cluster ran 4.17. The fix is a new >= 4.19 cluster (hysh-ibm-01, 4.21.27), not an in-place upgrade, cross-linked to the ibm-cluster skill. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(control-plane): environment-adaptive tenant ingress (Gateway API or Route) Tenant-gateway ingress is now a selectable mode, chosen per environment by configuration rather than hardcoded: emit Kubernetes Gateway API GRPCRoutes where the Gateway API is available and functional, or OpenShift Routes (HAProxy passthrough) where it is not. Motivation: IBM Cloud ROKS is HyperShift-hosted and cannot run the CIO-managed Istio (OSSM images unpullable, IDMS denied on the HostedCluster), yet ships the Gateway API CRDs. Route passthrough preserves the gateway pod's per-tenant self-signed TLS + client mTLS end-to-end, so it needs no shared Gateway, wildcard cert, ClusterIssuer, or external DNS - it works on IBM's free *.containers.appdomain.cloud wildcard. Control plane: - GATEWAY_INGRESS_MODE env var (gateway-api|route|none); auto-detects from opts.HasGatewayAPI/opts.IsOpenShift when unset. Explicit override wins, since ROKS's Gateway API CRDs are present-but-non-functional. - reconcileRouteResources/deleteRouteResources (passthrough Route to openshell-gateway:8080 + openshell-gateway-allow-router NetworkPolicy + grpcs://<host>:443 address publish); shared deriveGatewayHostname/ publishRouteAddress helpers; Route added to kindToResource and cleanup. - Table tests for mode selection and hostname derivation. Deploy: - deploy/ibm kustomize overlay (on deploy/openshift) sets GATEWAY_INGRESS_MODE=route + base domain. Controller ClusterRole already grants route.openshift.io/routes. Docs: - global-architecture.spec.md: two first-class ingress modes with mode-aware requirements/scenarios and the ROKS Route-mode section. - ibm-cluster / cloud-hub-ingress-bootstrap skills: Route mode is the ROKS path; do not run the shared-Gateway bootstrap there. - Swept em dashes from tracked files; whitelisted pre-existing ACP mermaid nodes and rosa-vteam.yaml so make check passes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(control-plane): make gateway/supervisor/sandbox images env-overridable for registry mirrors Gateway, supervisor, and sandbox default images were hardcoded to ghcr.io with no override, so any gateway created without an explicit image failed to pull on clusters whose nodes cannot reach ghcr.io (e.g. IBM ROKS). - Add GATEWAY_IMAGE, GATEWAY_SUPERVISOR_IMAGE, and GATEWAY_SANDBOX_IMAGE env overrides (mirroring the existing HYPERSHELL_DATABASE_IMAGE pattern), with a new DefaultSandboxImage() so the sandbox base is resolved the same way. - Substitute SANDBOX_IMAGE_PLACEHOLDER in the gateway configmap (ordered before IMAGE_PLACEHOLDER since the shorter token is a substring). - Allow an optional host:port in image references so the in-cluster registry service address (image-registry.openshift-image-registry.svc:5000/...) validates. - Apply supervisor_image on gateway PATCH. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(control-plane): add derived ingress hostname to gateway certificate SANs When an ingress mode is active the gateway is reachable at an external hostname (gw-<namespace>.<base-domain> or an explicit Route.Host), and both ingress modes carry the gateway pod's TLS through unmodified (Route passthrough / Gateway API BackendTLSPolicy). The server certificate must therefore list that external hostname as a SAN, or clients fail verification. The controller derives the hostname, so it injects it into the cert SANs before cert-manager mints the certificate. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(spec): expand global-architecture with ROKS ingress and image-mirror details Extend the global architecture specification with the ingress-mode and internal-registry-mirror behaviour exercised on IBM ROKS, and bump the forbidden-terms whitelist line reference to track the moved example path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(skills): ROKS deploy guidance for ibm-cluster and deploy-cluster Document the ROKS-specific deployment path: image mirroring to the internal registry, raw operator installs, and the env overrides the control plane needs on clusters that cannot reach ghcr.io. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(pr-test): ROKS-adapted e2e-openshell script and ibm deploy overlay Add e2e-openshell-roks.sh, a ROKS-layout copy of the canonical e2e that targets the hypershell/hypershell-api route names, trusts the per-gateway CA, uses the quoted SQL-like search grammar, and preserves a pre-existing gateway on cleanup. Add the components/api-server/deploy/ibm overlay with the controller cluster RBAC used on ROKS. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(ibm-cluster): worker-SG egress fix + OIDC default-secure + cloud providers Refine the registry-timeout explanation with its root cause (the kube-<clusterID> worker Security Group is default-deny outbound) and the supported fix (add an outbound 0.0.0.0/0:443 rule). Add section 5.7 covering Keycloak default-secure gateway wiring, the correct OIDC `gateway add` command (not bare edge/cloud mode), and the worker-egress requirement for cloud-model providers such as google-vertex-ai. Correct the stale gateway-add note in 5.5. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(ibm-cluster): document sandbox connect gateway-insecure env workaround `openshell sandbox connect` execs the system ssh with a ProxyCommand that re-execs `openshell ssh-proxy`, and the CLI omits `--gateway-insecure` from that generated ProxyCommand. The child ssh-proxy therefore verifies the self-signed passthrough gateway cert and fails `invalid peer certificate: UnknownIssuer`; the flag on `connect` never reaches it. The child inherits the environment, so `export OPENSHELL_GATEWAY_INSECURE=true` is the working fix. Verified live on hysh-ibm-01 (sandbox woot). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * spec: inference-routing + ROKS credential-free sandbox agent runbook Add openshell-inference-routing.spec.md documenting how sandbox agents reach cloud models with no credential in the sandbox: the inference.local router strips the client key and injects the provider token server-side, translating /v1/messages -> Vertex :rawPredict. Covers the two credential paths (per-binary sentinel rewrite vs router injection) and the request-shape compatibility requirement. Register it in the spec index. Add ibm-cluster skill section 5.8 with the ROKS runbook: `inference set`, the required non-effort `--model claude-sonnet-4-5` workaround for Vertex's strict vertex-2023-10-16 validation (adaptive-thinking / output_config.effort 400s), sandbox connect, and the ~/.claude/settings.json wiring for bare `claude`. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(spec): add --dangerously-skip-permissions to inference client example Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * style: replace em dashes with ASCII punctuation for forbidden-terms lint main tightened scripts/check_forbidden_terms.py to reject U+2014; convert the em dashes in files this PR authored to ASCII hyphens. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(control-plane): validate Route.Host and block cross-tenant route hijack An explicit Gateway Route.Host was not validated: it bypassed the DNS-name check applied to ServerDnsNames and flowed verbatim into the OpenShift Route spec.host (with the controller holding routes/custom-host) and the gateway certificate SANs. Under a shared wildcard base domain a tenant could set Route.Host to another tenant's derived host (gw-<other>.<base-domain>) and hijack its route, since OpenShift Route host claiming is first-come. - ValidateGatewayConfig now DNS-validates Route.Host (hard fail). - deriveGatewayHostname now requires an explicit host that falls under GATEWAY_API_BASE_DOMAIN to equal this tenant's own gw-<namespace>.<base> slot; foreign hosts under the shared wildcard are rejected (fail-closed: no Route is created). External/vanity hosts outside the base domain pass through unchanged. - Table tests for the DNS validation and the hijack/own-slot/vanity cases. Addresses Amber review finding #1 on PR #85. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(spec): clarify control-plane role, terminology, ownership, and LB (amarin #1-5) Address amarin's review comments on the global architecture spec: - Broaden the control-plane operational role: it provisions the full set of OpenShell resources per tenant (namespaces, PKI, RBAC, ingress, CNPG, and supporting workloads), not just OpenShell Gateways. - Terminology: add a canonical-names note and use fully-qualified names — "OpenShell Gateway" (workload), "Gateway API" (k8s API), "shared Gateway (Gateway API resource)" — instead of the overloaded bare "gateway". - Make Sandboxes explicit in the Tier 3 gateway-workloads description. - Source of truth: HyperShell's Cloud Hub PostgreSQL owns desired state for Fleet/Gateway/ManagedCluster; OpenShell Gateway runtime state (sandboxes, provider credentials, sessions) lives in the gateway's own database. - Generalize the platform-services load balancer: describe the role generically with AWS NLB and IBM Cloud VPC LB as the concrete instances today. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(spec): OIDC is the client auth mechanism; drop client mTLS (amarin #6) HyperShell authenticates callers with OIDC (Keycloak bearer tokens); client mTLS is not required or supported. The gateway pod's TLS is server-side transport encryption only. Remove client mTLS from the four clauses that previously described it as a hard, mode-independent prerequisite: - ingress overview (both modes converge on the same workload) - ROKS route-mode cert-manager prerequisite note - Requirement: Tenant Gateway Ingress via OpenShift Route (route mode) - Requirement: cert-manager Is a Mode-Independent Prerequisite cert-manager still mints the gateway's per-tenant server TLS + CA in every mode. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(deps): update OpenShell to v0.0.106; add self-reinforcing update-openshell skill Add skills/tooling/update-openshell - a repeatable, self-reinforcing skill for syncing HyperShell to upstream OpenShell releases (they ship ~daily). It bumps the version-pin footprint, triages release notes for contract-affecting changes, verifies build/config, and folds each run's lessons back into the skill + specs. First run: update 0.0.101 -> 0.0.106. - Bump defaultGatewayImage/defaultSupervisorImage (source of truth) and every copy across specs, the ROKS e2e script, kind lib, and the ibm-cluster skill. - Normalize specs/platform/openshell-gateway-database.spec.md, which pinned the gateway image to a git SHA instead of a release tag. - Preserve fixtures/historical refs (the validation_test.go regex fixture and the "v0.0.101 introduced credential drivers" historical sentence). Triage 102-106: mechanically safe pin bump. One needs-decision item recorded as a follow-up in the skill's learnings log - upstream v0.0.106 shipped a cert-manager external issuer + OpenShift passthrough Route (NVIDIA/OpenShell#2468) that overlaps HyperShell's hand-rolled per-tenant self-signed CA; evaluate adopting it separately. Also repair pre-existing "make check" failures from earlier spec commits on this branch: replace forbidden em dashes (U+2014) with " - " and refresh the line-number-based forbidden-term whitelist (Mermaid ACP nodes + vteam path) after the Terminology block shifted line numbers. Register /update-openshell in CLAUDE.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(control-plane): make ROKS e2e pass on openshell 0.0.109 Bring the IBM Cloud ROKS (hysh-ibm-01) end-to-end path to 22/22 on openshell 0.0.109. components/pr-test/e2e-openshell-roks.sh now validates the full flow (HyperShell API -> control plane -> per-tenant passthrough Route -> gateway -> OIDC admin + developer -> sandbox create + exec) for both an admin and a standard user. Three fixes, each caught only by the live sandbox e2e (pin/schema diffing missed all three): - Sandbox client TLS: restore the openshell-client cert-manager Certificate and client_tls_secret_name so runners get OPENSHELL_TLS_CA to verify the gateway server cert (required by 0.0.109 combined topology). This is internal sandbox<->gateway TLS, distinct from external-client mTLS (external clients authenticate via OIDC over the Route; no client_ca_path). - StatefulSet/Deployment collision: drop statefulset.yaml from the deploy order and remove the file so the gateway workload is a single Deployment (no orphaned crash-looping openshell-gateway-0). - Workspace membership: the e2e's developer-RBAC step now has an admin grant the standard user 'user' membership on the 'default' workspace before sandbox create. openshell 0.0.109 enforces workspace membership as a second, non-claim-derived authz layer independent of the OIDC role. The e2e defaults OPENSHELL to ~/.local/bin/openshell (>= 0.0.98, which has the `workspace` subcommand); there is no downloadable 0.0.109 CLI. Docs: ibm-cluster/SKILL.md gains a validation banner, section 5.9 (workspace membership + CLI version), and section 5.5 notes on sandbox client TLS and the Deployment-only workload; update-openshell/SKILL.md gains a 0.0.106 -> 0.0.109 learnings-log entry (v1beta1 confirmed against the running gateway). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(pr-test): REST-based gateway delete, stale-gateway re-provision, sandbox diagnostics Three fixes to the ROKS e2e (components/pr-test/e2e-openshell-roks.sh), found while diagnosing sandboxes that reached Running but never became Ready (every runner crashlooped on "OPENSHELL_TLS_CA is required"): - Delete via REST, not hsctl. hsctl exposes no `delete` subcommand (only create/get/list/login), so the cleanup trap's `hsctl delete gateway` silently no-op'd on every run - gateways and their tenant namespaces accumulated indefinitely. Added a delete_gateway helper that calls DELETE /api/hypershell/v1/gateways/{id} and use it in cleanup. - Detect and re-provision stale gateways. A gateway provisioned by a controller predating the sandbox client-TLS fix has no openshell-client-tls secret and no client_tls_secret_name in gateway.toml, so its sandboxes crashloop. The script now checks those markers on an existing gateway and, if stale, deletes and re-provisions it instead of blindly reusing it. - Surface sandbox failure root cause. dump_sandbox_diag prints the sandbox pod's phase/restart count and last container logs on any exec/not-ready/not-found failure, turning the CLI's generic "sandbox is not ready" into the actual cause. Also correct ibm-cluster/SKILL.md 5.1: it wrongly claimed HyperShell no longer issues an openshell-client certificate. It does - the client cert exists so sandbox runners get OPENSHELL_TLS_CA to verify the gateway server cert (internal sandbox->gateway TLS, not external mTLS). Validated on hysh-ibm-01: 23/23 with a stale gateway present (detected, torn down, re-provisioned; admin + developer sandbox exec both succeed), 22/22 on a clean run with cleanup actually deleting the gateway (0 leftover e2e gateways). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: user <u@example.com> Co-authored-by: Claude <noreply@anthropic.com>

jhjaggars requested review from a team, derekwaynecarr, maxamillion and mrunalp as code owners July 24, 2026 15:32

jhjaggars force-pushed the 2466-cert-manager-openshift-tls/jhjaggars branch from 87527a1 to cf02477Compare July 28, 2026 14:52

matthewgrossman mentioned this pull request Jul 29, 2026
feat(helm): cert-manager external issuer + OpenShift passthrough Route for gateway TLS #2466
Closed

jhjaggars force-pushed the 2466-cert-manager-openshift-tls/jhjaggars branch from cf02477 to 0276980Compare August 4, 2026 13:28

jhjaggars force-pushed the 2466-cert-manager-openshift-tls/jhjaggars branch from cc11d20 to 876959aCompare August 4, 2026 22:26

jhjaggars force-pushed the 2466-cert-manager-openshift-tls/jhjaggars branch from 876959a to aac65a9Compare August 5, 2026 19:18

mrunalp previously approved these changes Aug 6, 2026
View reviewed changes

mrunalp enabled auto-merge August 7, 2026 00:22

auto-merge was automatically disabled August 7, 2026 19:47
Head branch was pushed to by a user without write access

jhjaggars dismissed mrunalp’s stale review via 6225211August 7, 2026 19:47

jhjaggars force-pushed the 2466-cert-manager-openshift-tls/jhjaggars branch from 4e5ec46 to 1362c65Compare August 7, 2026 22:55

jhjaggars requested a review from mrunalp August 12, 2026 20:32

bsquizz mentioned this pull request Aug 12, 2026
feat: add gateway OIDC, OpenShift routing, and IBM Cloud deployment openshift-online/hypershell#72
Closed
6 tasks

jhjaggars added 10 commits August 13, 2026 10:42

fix(drivers): strip GATEWAY_TLS_SERVER_NAME from Docker and Podman env
a028d63
Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

fix(helm,drivers): guard default clientCaSecretName and add env-strip…
a19c563
… tests Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

jhjaggars force-pushed the 2466-cert-manager-openshift-tls/jhjaggars branch from 93c3473 to 74de2a6Compare August 13, 2026 14:45

mrunalp approved these changes Aug 14, 2026
View reviewed changes

mrunalp added this pull request to the merge queue Aug 14, 2026

Merged via the queue into NVIDIA:main with commit c4b500aAug 14, 2026
34 checks passed

prekshivyas mentioned this pull request Aug 16, 2026
chore(deps): update OpenShell to 0.0.106 NVIDIA/NemoClaw#9192
Merged
23 tasks

bsquizz mentioned this pull request Aug 19, 2026
spec: global architecture spec & IBM ROKS impl fixes openshift-online/hypershell#85
Merged
3 tasks

Uh oh!

Conversation

jhjaggars commented Jul 24, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Dual-certificate architecture (SNI-based)

Security: supervisor trust pinned to chart CA

Helm templates

Docs

Testing

Checklist

Uh oh!

copy-pr-botBot commented Jul 24, 2026

Uh oh!

mrunalp commented Jul 30, 2026

Uh oh!

mrunalp commented Jul 31, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Blocking

1. Native roots are controlled by the sandbox image

2. serverIssuerRef renders a broken default client-CA configuration

3. clientIssuerRef is not independently usable as documented

4. The documented OIDC production command cannot connect

5. The passthrough Route accepts a plaintext backend

Uh oh!

mrunalp commented Aug 4, 2026

Follow-up review: all five blocking items addressed

Remaining

1. The env-scrubbing half of item 1 is still open

2. Dangling clientIssuerRef reference in the docs

3. A half-configured client CA still renders (optional)

CI

Uh oh!

mrunalp commented Aug 4, 2026

Uh oh!

mrunalp commented Aug 5, 2026

Follow-up review at 876959a1

Verified fixed

Please fix: a target symlink was committed

Minor

CI

Uh oh!

mrunalp commented Aug 5, 2026

Follow-up review at aac65a99

Present and verified

Regressed: the default clientCaSecretName guard is gone

Still minor

CI

Uh oh!

mrunalp commented Aug 6, 2026

Uh oh!

github-actionsBot commented Aug 7, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mrunalp commented Aug 12, 2026

Blocking

1. The old client-CA guard is now incorrect

2. Route validation conflicts with wildcard certificates

3. serverIssuerRef is not coupled to certManager.enabled

Follow-ups

Uh oh!

mrunalp commented Aug 13, 2026

Remaining concern

Uh oh!

mrunalp commented Aug 13, 2026

Uh oh!

mrunalp commented Aug 13, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jhjaggars commented Jul 24, 2026•
edited
Loading

mrunalp commented Jul 31, 2026•
edited
Loading

2. `serverIssuerRef` renders a broken default client-CA configuration

3. `clientIssuerRef` is not independently usable as documented

2. Dangling `clientIssuerRef` reference in the docs

Follow-up review at `876959a1`

Please fix: a `target` symlink was committed

Follow-up review at `aac65a99`

Regressed: the default `clientCaSecretName` guard is gone

github-actionsBot commented Aug 7, 2026•
edited
Loading