Make substrate's namespace, Service names and ServiceAccount names configurable by jjamroga · Pull Request #350 · agent-substrate/substrate

Jonathan Jamroga (jjamroga) · 2026-06-29T18:59:14Z

Description

Substrate assumes the canonical install layout, and every deviation fails closed. The namespace ate-system, the Services api / atenet-router, and the ServiceAccounts atelet / atenet-router are compiled-in constants. An install in a per-developer namespace, or under a Helm release that prefixes resource names, breaks — and no failure points at the naming.

This series makes the namespace, the Service names, and the ServiceAccount names configuration. Every option defaults to the canonical value. A canonical install is byte-for-byte unaffected.

Hardcoded assumption	Failure	What it looks like instead
atelet's namespace in ateapi's SPIFFE check	ateapi rejects every atelet	an mTLS handshake failure, not a naming error
atelet's namespace in the worker's broker check	no actor obtains a certificate	`credential broker is not atelet`
NetworkPolicy's ingress namespace	the CNI drops every request to the pool	a silent network fault
ateapi Service name and namespace in the client	the client cannot authenticate	`services "api" not found`, then `invalid bearer token`
Resource names in `hack/install-ate.sh` and the e2e harness	authorities land in the wrong namespace	suites die in preflight

A SPIFFE ID breaks on two axes. It names a namespace and a ServiceAccount. Both were constants, and the failure surfaces as a rejected peer, not a missing object.

Design notes

ate-controller passes the worker-side identities. ateom already took --atunnel-client-identity as a flag but relied on its default. A new --atunnel-broker-identity flag alone would be inert: correct only where the constant was already correct. The controller knows the control plane's namespace, so it supplies both.
ateapi takes the whole expected identity, not a namespace. The dialer and ateletauth used the namespace only to build one string. One place decides how atelet's identity is spelled.
ateletauth's dedup was pre-authorised. Its comment asks for a shared package "if a third pkg that needs these constants appears". ateletdial is the third. The segments move to internal/installdefaults.
The client reads environment variables, not flags. It runs outside the cluster. It has no downward API and nothing to discover from.
Namespaces come from the downward API, not flags. Every supported topology co-locates these components. Only Service and ServiceAccount names, which a Helm release may rename, get flags.
hack/install-ate.sh and the e2e harness are in scope. A relocated install cannot be bootstrapped or exercised without them. The script refuses ATE_NAMESPACE or ATE_API_SERVICE_NAME overrides on the manifest-applying subcommands, which would half-install: manifests/ate-install/ names ate-system and api literally.

Rebase notes (2026-09-10)

The series is rebased onto current main, which refactored both identity checks while the PR was open:

ateletauth (ateapi's side) and ateletdial (the worker's side) each re-declared the hardcoded SPIFFE ID. Both now take the expected identity as a parameter.
The refactor added two consumers this series originally predates: workerservice.SetWorkerCapacity authenticates through ateletauth, and ateomcapacity dials through ateletdial. Both receive the configured identity.
main deleted the atenet DNS subsystem. The series no longer touches it, and installdefaults carries no DNS Service name.
The controller passes --atunnel-broker-identity only when it differs from the canonical default. An ateom old enough to predate the flag exits on it, and docs/upgrade.md keeps such a pool serving during a rolling upgrade. A relocated install gets the flag and necessarily runs an image that accepts it.
A new test in installdefaults fails on any new Go string literal naming the canonical namespace or a SPIFFE identity outside an explicit allowlist. The refactor above re-introduced exactly this class of constant while the series was in review, so the guard earns its place.

Testing

On kind, before the rebase: install into a non-default namespace, control plane healthy, kubectl-ate authenticates, an actor template reaches Ready, e2e suites pass.
Under a Helm release that prefixes every resource name, a worker carries --atunnel-client-identity=spiffe://cluster.local/ns/kagent/sa/kagent-atenet-router and the matching broker identity — correct on both axes. Before this change, both read ns/ate-system/sa/atelet.
After the rebase: the full unit suite passes with -race; every commit builds and vets individually; CI's kind e2e revalidates the install paths.
The new unit tests use a relocated namespace and renamed ServiceAccounts. Reintroducing each hardcoding makes them fail; the pre-existing tests passed canonical values and held against the constants they now guard.

Additional Notes

Credential contents stay canonical. actoridentity still mints credentials naming api.ate-system.svc: the JWT issuer and the certificate's Issuer CN. Relying parties validate these, so changing them is a compatibility decision, not a lookup fix. The code carries a TODO to make the issuer a globally unique, OIDC-compliant name.

Eitan Yarmush (EItanya)

Functionality makes sense. There are some duplicated constants and some env behavior I think we could take advantage of

…_NAMESPACE Addresses review comments on agent-substrate#350: - New internal/installdefaults package owns SystemNamespace, RouterServiceName, DNSServiceName. dns, router, and controlapi/informer drop their duplicate Default* constants and reference installdefaults via the matching flag declarations and tests. - Drop the --atelet-namespace flag on ateapi. The namespace is now resolved at startup from the POD_NAMESPACE env var (Kubernetes' downward API), falling back to installdefaults.SystemNamespace for non-k8s invocations (tests, local dev). atelet and ateapi share a namespace in every supported deployment topology, so a separate knob was dead weight.

* atenet: make system namespace and component Service names configurable The dns-controller (`atenet dns`) and router (`atenet router`) hardcoded the substrate namespace ("ate-system") and the component Service names ("atenet-router", "dns") from the canonical install manifests under `manifests/ate-install/`. Deployments that deviate from that layout — running in a different namespace, renaming the Services, or composing substrate into a larger install that rewrites resource names — silently break: the dns-controller can't find atenet-router, the router can't find itself for /statusz, and the cluster's actor DNS never gets patched. Expose the relevant names as flags on the cobra commands and as fields on `dns.Controller` / `router.RouterConfig`. Defaults match the values in `manifests/ate-install/` so existing deployments are unaffected: atenet dns: --system-namespace (default "ate-system") --router-service-name (default "atenet-router") --dns-service-name (default "dns") atenet router: --router-service-name (default "atenet-router") * ateapi: make atelet namespace configurable via --atelet-namespace The atelet pod informer hardcoded `ateletNamespace = "ate-system"`, so ate-api-server could only locate atelet pods in that namespace. Deployments that run atelet elsewhere — an alternative install layout or a larger composition that relocates substrate components — leave the informer's cache empty and ResumeActor fails with `found 0 atelet pods on node "<node>", expected 1`. Promote the constant to an exported default and accept the namespace as a parameter to `AteletInformer`. Add an `--atelet-namespace` flag on the ateapi binary (default DefaultAteletNamespace) that callers override when needed. * chart: pass system namespace and Service names to dns-controller and router Wire the new flags added in the previous commit through the Helm templates so the canonical-render defaults are overridden when the chart is used as a subchart (e.g. the kagent-enterprise composition where substrate.fullname prefixes all component Service names). For atenet-dns the dns-controller now receives: --system-namespace={{ .Release.Namespace }} --router-service-name={{ include "substrate.fullname" (list "atenet-router" .) }} --dns-service-name={{ include "substrate.fullname" (list "dns" .) }} For atenet-router the /statusz lookup gets: --router-service-name={{ include "substrate.fullname" (list "atenet-router" .) }} When the release name equals the chart name ("substrate") these expand to the canonical bare names, preserving existing behavior for top-level installs. * chart: pass --atelet-namespace to ate-api-server Wire the new ateapi flag from the previous commit through the chart so the atelet pod informer watches the chart's release namespace by default. Canonical render (release name "substrate" in namespace "ate-system") still produces "--atelet-namespace=ate-system", so behavior is unchanged for top-level installs. * chart: regenerate manifests/ate-install/ from current Helm chart Re-runs `make helm-template` so the checked-in render matches the chart. Brings in rustfs.yaml, the s3-backed atelet storage envvars, the trimmed valkey manifest, and drops the no-longer-templated sandboxconfig-gvisor and sandboxconfig-validation manifests. `make verify-helm-template` now passes. * review: centralize install defaults, derive atelet namespace from POD_NAMESPACE Addresses review comments on agent-substrate#350: - New internal/installdefaults package owns SystemNamespace, RouterServiceName, DNSServiceName. dns, router, and controlapi/informer drop their duplicate Default* constants and reference installdefaults via the matching flag declarations and tests. - Drop the --atelet-namespace flag on ateapi. The namespace is now resolved at startup from the POD_NAMESPACE env var (Kubernetes' downward API), falling back to installdefaults.SystemNamespace for non-k8s invocations (tests, local dev). atelet and ateapi share a namespace in every supported deployment topology, so a separate knob was dead weight. * review: derive atenet's system namespace from POD_NAMESPACE Same rationale as the prior atelet-namespace change: atenet, atenet-router, and substrate's CoreDNS live in a single namespace in every supported deployment topology, so a separate --system-namespace flag was dead weight. Resolve from the POD_NAMESPACE env var (Kubernetes' downward API) with installdefaults.SystemNamespace as the fallback for non-k8s runs. --router-service-name and --dns-service-name stay as flags because a subchart deployment renames those Services with a release prefix, and the binary can't derive that from pod metadata. * review: NamespaceFromPodEnv helper, APIServiceName const, ateclient hardcodes Three follow-ups from the self-review: - Extract the POD_NAMESPACE-with-SystemNamespace-fallback pattern into installdefaults.NamespaceFromPodEnv() so ateapi and atenet share a single implementation (also makes a third call site one line instead of four if anyone needs one). - Add installdefaults.PodNamespaceEnv ("POD_NAMESPACE") and APIServiceName ("api") so the constant set covers every name in the canonical install layout that's referenced by Go code. - Route internal/ateclient/builder.go's previously-hardcoded "ate-system" and "api" lookups through installdefaults, so kubectl-ate's port-forward no longer bypasses the new single source of truth. ate-controller (ServiceAccount), ate-api-server-deployment (Deployment), and "api.ate-system.svc" (JWT audience) are still hardcoded but their configurability needs a real flag/discovery story and is out of scope for this PR. * chart: render ate-client ServiceAccount in every mode The JWT install overlay (manifests/ate-install/jwt) references ate-client.yaml as a top-level resource, but the chart previously guarded the SA behind {{ if eq .Values.auth.mode "jwt" }} so render-manifests.sh (mtls) never emitted it. That divergence broke verify-helm-template after merging the upstream JWT fix that added a hand-maintained manifests/ate-install/ate-client.yaml. The SA is harmless in mtls installs (unused), so render it unconditionally so the chart is the single source of truth.

…figurable The control-plane components hardcoded "ate-system" and the Service names of their peers, so an install into any other namespace — a per-developer sandbox, or a chart that renames the release — silently failed to find atenet-router, ateapi, or the atelet pods. Collect the canonical install names in internal/installdefaults so there is a single place where the layout in manifests/ate-install/ is written down, and consume them from the binaries: - atenet router takes --router-service-name for the /statusz self lookup of its own ClusterIP. - ateapi resolves the atelet namespace from POD_NAMESPACE and passes it to the atelet informer and dialer. The dialer needs it for more than listing: the namespace is a segment of the SPIFFE ID it expects on the atelet serving cert, so a hardcoded value would fail the mTLS handshake wherever atelet does not run in ate-system. - ateclient runs outside the cluster, where the downward API is not available, so it uses the install defaults directly. Namespaces are read from the downward API rather than exposed as flags because every supported topology co-locates these components; only the Service names, which a chart may legitimately rename, get flags.

…rom the install namespace The audit for the previous commit turned up more places that assume substrate occupies "ate-system", all of which fail closed in a relocated install rather than degrading visibly: - The NetworkPolicy the WorkerPool controller generates admits ingress only from the "ate-system" namespace. Pointed at the wrong namespace it admits nobody, so every request to the pool's actors is dropped by the CNI with no error from substrate itself. - ateletauth, which authenticates atelet's calls to actoridentity's MintCert and workerservice's SetWorkerCapacity, re-declared the atelet SPIFFE segments to avoid depending on controlapi, with a comment asking for a shared package once a third consumer appeared. ateletdial is that third consumer, so the segments move to installdefaults and the namespace is threaded in from ateapi. - ateletdial verifies atelet's SPIFFE ID where a worker dials the node-local socket — atunnel's credential broker client and ateomcapacity's capacity report. Both run in the actor's pod, whose POD_NAMESPACE is the actor's and not atelet's, so the expected identity is passed in. atunnel already took the atenet-router identity as a flag, so the atelet identity follows that convention. Both flags previously relied on their defaults being right; the controller now passes them explicitly, since it is the component that knows which namespace the control plane occupies. Without that the new flag would be inert — correct only where the old hardcoded value was already correct. The egress MITM CA pool Secret reference was hardcoded to the same namespace and is now derived alongside it.

Benjamin Elder (BenTheElder) · 2026-09-10T17:30:30Z

Please don't put links to github issues / PRs into git commits, this is specified in our AGENTS.md.

It makes the github event history for the page absurd (see above), you can't really read the comment stream because there's pages of github pingbacks.

Jonathan Jamroga (jjamroga) · 2026-09-10T17:32:27Z

Please don't put links to github issues / PRs into git commits, this is specified in our AGENTS.md.

It makes the github event history for the page absurd (see above), you can't really read the comment stream because there's pages of github pingbacks.

Sorry about that, I think I was using my own agents.md instead of the projects. I will fix this. Aside from the PR cleanliness, do you have any feedback for the problem this PR is solving?

(Also I can totally create a fresh PR Benjamin Elder (@BenTheElder) if you'd like a clean stream, this has been sitting around for a while)

Running a relocated install end to end turned up three more places that assume substrate occupies ate-system, none of which a read of the code makes obvious. ateclient looked configurable but was not: it resolved the namespace and the ateapi DNS name from installdefaults constants, so kubectl-ate always port-forwarded to ate-system/api and always minted its token for the audience api.ate-system.svc. Against a relocated install it failed with `services "api" not found`, and had it got further the audience would not have matched what the server was configured to accept. The client runs outside the cluster, where there is no downward API to read, so the namespace comes from ATE_NAMESPACE. hack/install-ate.sh hardcoded the namespace in 52 places, so the mTLS authorities it creates landed in ate-system while the control plane looked for them in the release namespace. It now reads the same ATE_NAMESPACE. Namespace creation is the one part that is not a substitution: manifests/ate-install/ate-system-namespace.yaml names ate-system and carries labels of its own, so it stays the source of truth for the canonical install and ensure_ate_namespace falls back to a plain create for anything else. The e2e harness addressed the router, the egress CA pool and the egress gateway in ate-system regardless of where the install under test lived. It now reads E2E_SYSTEM_NAMESPACE. The egress probe is the awkward case: it runs inside the sandbox and cannot infer substrate's namespace, so the suite substitutes it into the probe manifest the way it already substitutes the probe's own namespace. Left alone, every egressauthz case failed in the tunnel stage on a DNS lookup, which reads as an authorization failure rather than a misconfigured address. Both variables default to ate-system, so existing flows are unaffected.

A SPIFFE ID names a ServiceAccount as well as a namespace, and the ServiceAccount name was still a constant. Any deployment that renames those accounts — the Helm chart prefixes every resource name for a release not called "substrate", which is every install consuming substrate as a subchart — ends up with ateapi and each worker's atunnel expecting an account that does not exist, and rejecting the peer that does. ateapi takes --atelet-service-account and ate-controller takes that plus --router-service-account, both defaulting to the canonical names so existing installs are unaffected. ateapi's dialer and actoridentity now take the whole expected identity rather than a namespace they rebuild it from. Both only ever used the namespace to construct that one string, and having a single place decide what atelet's identity is beats two places agreeing on how to spell it. ate-controller stamps both identities into each worker's args, since it is the component that knows which namespace and accounts the control plane runs under. --atunnel-broker-identity is newer than the oldest ateom a rolling upgrade still has running: docs/upgrade.md keeps the outgoing worker pool serving alongside the new one, this controller reconciles that pool's Deployment while it is still pinned to its old image, and ateom parses flags with ExitOnError — so the flag is passed only when it differs from the identity an ateom without it already assumes. That is equivalent for both binaries in a canonical install and keeps the upgrade intact, while a relocated or renamed install still gets the flag and necessarily runs an image new enough to accept it. --atunnel-client-identity needs no such care: it predates every ateom this controller ships beside. Empty ServiceAccount flag values are refused at startup. path.Join drops an empty segment, so they would produce an identity that parses and matches nothing, and every peer would be rejected with nothing pointing at the flag.

Installing substrate under a release that renames its resources — which a Helm chart does for any release not called "substrate", and which is what consuming substrate as a subchart produces, since a subchart inherits the parent's release name — renames more than the ServiceAccounts. The Service fronting ateapi becomes <release>-api and the ServiceAccount the CLI mints its token from becomes <release>-ate-client, and both were constants: kubectl-ate port-forwarded to a Service that does not exist, and once that was fixed it minted a token for the wrong audience and ateapi rejected it as an invalid bearer token. The client reads ATE_API_SERVICE_NAME and ATE_CLIENT_SERVICE_ACCOUNT alongside ATE_NAMESPACE; it runs outside the cluster and cannot discover any of this. hack/install-ate.sh reads the same ATE_API_SERVICE_NAME, since the audience it writes has to name the Service the client dials. The e2e harness addressed ate-controller, ate-api-server, atenet-router and the egress CA pool Secret by their canonical names, so every suite failed in preflight before running a case. E2E_RESOURCE_PREFIX carries what the install under test prepends. All of them default to the canonical names. The egress MITM CA pool Secret is the exception: hack/install-ate.sh creates it under a fixed name and the reconciler looks it up the same way, so it never carries a prefix and the harness reads it unprefixed. ATE_API_SERVICE_NAME joins ATE_NAMESPACE in the guard on the script's manifest-applying subcommands. The authentication config bakes the name into the accepted token audience, but the checked-in manifests create the Service literally named "api", so the combination would mint tokens no server accepts. A renamed Service comes from a Helm install; the script serves only the bootstrap steps there.

The identities this series makes configurable were each once a compiled-in constant, and every one of them failed closed in a relocated install without pointing at the naming. While the series was in review, the ateletauth and ateletdial extractions re-added two such constants, so the hardcoding is a recurring class, not a one-time cleanup. A test in installdefaults — the package that owns the canonical values — walks the tree and fails on new occurrences: Go string literals under cmd/ and internal/ may not name the canonical namespace or a SPIFFE identity unless they are a flag or env default carried by the explicit allowlist. Comments and kubebuilder markers (which must name a literal namespace) are not scanned. Tooling that targets the canonical layout by design — ate-setup, the benchmarking harness — is excluded as a tree.

Benjamin Elder (BenTheElder) · 2026-09-10T17:53:24Z

We're in the community meeting now so I didn't manage to read all the code yet. The concept sounds reasonable.

EDIT: And thanks, no need to form a new PR. Just trying to get ahead of future PRs. This one doesn't have much prior discussion yet anyhow :-)

Eitan Yarmush (EItanya) reviewed Jun 29, 2026

View reviewed changes

Comment thread cmd/ateapi/main.go Outdated

Comment thread cmd/atenet/internal/dns/dns.go Outdated

Comment thread cmd/atenet/internal/router/router.go Outdated

Jonathan Jamroga (jjamroga) marked this pull request as draft June 30, 2026 14:01

Jonathan Jamroga (jjamroga) marked this pull request as ready for review June 30, 2026 15:50

Jonathan Jamroga (jjamroga) requested a review from Eitan Yarmush (EItanya) June 30, 2026 16:03

Jonathan Jamroga (jjamroga) force-pushed the jjamroga/atenet-configurable-resource-names branch 2 times, most recently from 105c140 to 97b83db Compare September 3, 2026 16:46

Jonathan Jamroga (jjamroga) changed the title ~~atenet, ateapi: make system namespace and component Service names configurable~~ Make substrate's namespace, Service names and ServiceAccount names configurable Sep 3, 2026

Jonathan Jamroga (jjamroga) mentioned this pull request Sep 3, 2026

Make substrate's namespace, Service names and ServiceAccount names configurable kagent-dev/substrate#31

Open

Jonathan Jamroga (jjamroga) added 2 commits September 10, 2026 12:35

Jonathan Jamroga (jjamroga) force-pushed the jjamroga/atenet-configurable-resource-names branch from 97b83db to f4bb2f4 Compare September 10, 2026 16:46

Jonathan Jamroga (jjamroga) added 4 commits September 10, 2026 13:38

Jonathan Jamroga (jjamroga) force-pushed the jjamroga/atenet-configurable-resource-names branch from f4bb2f4 to 24916ce Compare September 10, 2026 17:41

Benjamin Elder (BenTheElder) assigned Taahir Ahmed (ahmedtd) Sep 10, 2026

Navigation Menu

Uh oh!

Make substrate's namespace, Service names and ServiceAccount names configurable - #350

Make substrate's namespace, Service names and ServiceAccount names configurable#350
Jonathan Jamroga (jjamroga) wants to merge 6 commits into
agent-substrate:mainfrom
jjamroga:jjamroga/atenet-configurable-resource-names

Jonathan Jamroga (jjamroga) commented Jun 29, 2026 •

edited

Loading

Uh oh!

Eitan Yarmush (EItanya) left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Benjamin Elder (BenTheElder) commented Sep 10, 2026

Uh oh!

Jonathan Jamroga (jjamroga) commented Sep 10, 2026 •

edited

Loading

Uh oh!

Benjamin Elder (BenTheElder) commented Sep 10, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

Jonathan Jamroga (jjamroga) commented Jun 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Design notes

Rebase notes (2026-09-10)

Testing

Additional Notes

Uh oh!

Eitan Yarmush (EItanya) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Benjamin Elder (BenTheElder) commented Sep 10, 2026

Uh oh!

Jonathan Jamroga (jjamroga) commented Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Benjamin Elder (BenTheElder) commented Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Jonathan Jamroga (jjamroga) commented Jun 29, 2026 •

edited

Loading

Jonathan Jamroga (jjamroga) commented Sep 10, 2026 •

edited

Loading

Benjamin Elder (BenTheElder) commented Sep 10, 2026 •

edited

Loading