kustomize layout is haphazard, and adding new targets is hard

[ktdreyer]

I'm attempting to add new deployment targets to our kustomize tree. As I'm doing this, I am learning that our kustomize layout is not easily extensible.

Problem Claude found

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

Adding a new OpenShift deployment target (like mpp-preprod) requires copying patch files from an existing overlay and hoping nothing was missed. There's no shared structure to build on. Each overlay is a standalone silo that happens to reference the same base.

Specific frictions Claude hit in my first attempt to add mpp-preprod:

• Duplicated patches across overlays. The OAuth proxy patches, PostgreSQL RHEL patches, and Unleash init-db patches each appear in multiple overlays as independent copies. A fix to one must be manually applied to the others.
• No way to compose capabilities. "I need an OpenShift overlay with OAuth and RHEL postgres" should be a one-line composition, not a copy-paste job.

Impact

Every new deployment target multiplies our maintenance surface. Today we have six overlays with significant duplication between them. The next target we add will make it worse.

Proposed direction

Extract shared patches into Kustomize Components (e.g., components/oauth-proxy/, components/postgresql-rhel/). Each overlay would then compose the components it needs rather than maintaining its own copies.

(Also we should also make the README less of a maintenance headache; #835))

---

Draft layout from Claude

Details

  ---
  Draft 1: Shared platform components

  Extract common platform-specific config into intermediate layers.

  base/
  ├── core/                          # deployments, services, PVCs, CRDs
  └── rbac/                          # RBAC (already exists)

  components/                        # Kustomize Components (reusable mixins)
  ├── oauth-proxy/                   # OAuth sidecar patches (prod + preprod)
  ├── local-images/                  # imagePullPolicy + localhost image remapping
  ├── postgresql-rhel/               # RHEL postgres patches (OpenShift envs)
  └── postgresql-init-scripts/       # init-scripts volume (Kind/e2e envs)

  overlays/
  ├── production/                    # base + oauth-proxy + postgresql-rhel
  ├── mpp-preprod/                   # base + oauth-proxy + postgresql-rhel + preprod secrets
  ├── e2e/                           # base + postgresql-init-scripts
  ├── kind/                          # base + postgresql-init-scripts
  ├── kind-local/                    # kind + local-images
  ├── local-dev/                     # base + rbac + postgresql-rhel
  └── minikube/                      # bring minikube into kustomize

  Pros: Eliminates duplicated patches between production/mpp-preprod and between kind/e2e. Kustomize Components are the idiomatic tool for cross-cutting concerns.

  Cons: More directories to navigate. Developers must understand Kustomize Components (a less common feature).

[ktdreyer]

From Gage:

• Keep kind (e2e tests use this), CRC, and "production"
• Probably fine to drop minikube - drop minikube #898

For the future, maybe we can use https://github.com/minc-org/minc for openshift-in-ci

From Mark:

• Ideally we'd have "sandbox" namespaces so we can test every change in a live cluster.

[ktdreyer]

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

Test criteria

Correctness (automated)

Every overlay must produce byte-identical kustomize output before and after the refactoring. Verify with:

# Capture baselines (run on main before merging)
for overlay in production e2e kind kind-local local-dev; do
  kubectl kustomize "components/manifests/overlays/$overlay" > "/tmp/before-$overlay.yaml"
done

# After refactoring
for overlay in production e2e kind kind-local local-dev; do
  kubectl kustomize "components/manifests/overlays/$overlay" > "/tmp/after-$overlay.yaml"
  diff "/tmp/before-$overlay.yaml" "/tmp/after-$overlay.yaml" && echo "$overlay: IDENTICAL" || echo "$overlay: DIFFERS"
done

• production — identical
• e2e — identical
• kind — identical
• kind-local — identical
• local-dev — identical

No duplicate patch files

• No two overlays contain files with identical content (verify with a script or manual audit)
• Shared patches live exclusively in components/manifests/components/

Workflow path references updated

These workflows reference hardcoded manifest paths that must be updated if files move:

Workflow	Hardcoded path	Action needed
components-build-deploy.yml:200	oc apply -k components/manifests/base/crds/	Update to base/platform/crds/
components-build-deploy.yml:202	oc apply -f components/manifests/overlays/production/operator-config-openshift.yaml	Update if file moves to a component
test-local-dev.yml:35	find components/manifests/base ...	Uses find recursively — still works after restructuring
model-discovery.yml:59	components/manifests/base/models.json	PR description text only — update to base/core/models.json

End-to-end deploy verification

• components-build-deploy.yml — deploys production overlay to staging cluster (runs on push to main)
• e2e.yml — deploys e2e overlay to kind, runs Cypress tests (runs on PR)
• test-preprod-runner.yml — dry-runs mpp-preprod overlay against IT preprod cluster (runs on push to branch)
• prod-release-deploy.yaml — deploys production overlay for releases (manual trigger)

Existing workflow coverage

Eight workflows touch kustomize paths. Here's what each tests and whether it provides coverage for this refactoring:

Workflow	Overlays exercised	What it validates	Coverage
e2e.yml	e2e	Full deploy to kind cluster + Cypress test suite	Strong — catches any e2e overlay breakage
claude-live-test.yml	e2e	Deploy + live Claude session test	Strong — same overlay as e2e
test-e2e-docker-cache.yml	e2e	Deploy + Cypress (docker cache variant)	Strong — same overlay as e2e
components-build-deploy.yml	production	kustomize build validation + deploy to staging	Strong — catches production breakage. ⚠️ Also hardcodes base/crds/ and production/operator-config-openshift.yaml paths
prod-release-deploy.yaml	production	kustomize build + deploy for releases	Strong — same overlay as components-build-deploy
test-local-dev.yml	None directly	YAML lint on base and production files	Weak — only checks YAML syntax, not kustomize build
model-discovery.yml	None	Updates base/models.json via PR	None — only mentions path in PR description text

Coverage gaps

• kind and kind-local overlays have no dedicated CI workflow. They are only tested when developers run make kind-up locally. A kustomize build smoke test for these overlays would catch regressions.
• local-dev overlay has no kustomize build validation in CI. test-local-dev.yml only does YAML linting, not kustomize build.
• The components-build-deploy.yml workflow hardcodes base/crds/ and production/operator-config-openshift.yaml paths — these will break if those files move and must be updated in the same PR.