Uh oh!
There was an error while loading. Please reload this page.
Support explicit remote namespaces via spec.destination - #2344
Support explicit remote namespaces via spec.destination#2344kahirokunn wants to merge 2 commits into
Conversation
There was a problem hiding this comment.
@kahirokunn: 0 warnings.
Details
In response to this:
Proposed Changes
- Replace the top-level
spec.clusterProfileReffield with an explicitspec.destinationobject containing bothclusterProfileRefand the remote
installationnamespace.- Reconcile remote manifests, ingress resources, and the anchor ConfigMap into
spec.destination.namespaceinstead of coupling the remote installation to
the management CR namespace.- Make the complete destination immutable and validate the ClusterProfile name,
ClusterProfile namespace, and installation namespace as Kubernetes names.- Update generated CRDs, Helm CRDs, documentation, unit tests, and multicluster
E2E coverage for the new API shape.This is an intentional breaking replacement of an unreleased API with no
current consumers. No compatibility or migration path for the removed
top-levelspec.clusterProfileReffield is retained.Design
The existing multicluster reconciliation design remains unchanged: the
operator resolves the referenced Cluster Inventory APIClusterProfile, swaps
the manifest client at the start of reconciliation, and uses an anchor
ConfigMap for namespace-scoped garbage collection on the remote cluster.This PR makes the destination explicit:
spec: destination: clusterProfileRef: name: cluster1namespace: cluster-inventorynamespace: knative-servingThe management CR namespace and remote installation namespace are independent.
spec.namespace.labelsandspec.namespace.annotationsare merged into the
selected remote namespace while preserving unrelated existing metadata.The complete
spec.destinationis immutable. Moving an existing installation
between clusters or namespaces requires deleting and recreating the management
CR so that remote finalization runs before the new destination is installed.Example CR
KnativeServing on a remote cluster
apiVersion: operator.knative.dev/v1beta1kind: KnativeServingmetadata: name: knative-serving-apacnamespace: management-servingspec: destination: clusterProfileRef: name: apac-cluster-01namespace: fleet-systemnamespace: knative-servingingress: gateway-api: enabled: trueistio: enabled: falseKnativeEventing on a remote cluster
apiVersion: operator.knative.dev/v1beta1kind: KnativeEventingmetadata: name: knative-eventing-apacnamespace: management-eventingspec: destination: clusterProfileRef: name: apac-cluster-01namespace: fleet-systemnamespace: knative-eventingLocal cluster
apiVersion: operator.knative.dev/v1beta1kind: KnativeServingmetadata: name: knative-servingnamespace: knative-servingspec: {}# No destination: deploys to the local cluster as before.Verification Steps
Automated verification
go test ./... go test -run '^$' -tags='e2e multicluster' ./test/e2e ./hack/verify-codegen.shThe repository multicluster Kind suite also passed all 11 tests:
SPOKE_CLUSTER_NAME=codex-destination-spoke \ ./test/e2e-tests-multicluster.sh --kubeconfig=<hub-kubeconfig>E2E verification with Kind
The following end-to-end flow was executed against this PR head. It uses three
Kind clusters (hub,cluster1, andcluster2), real OCM-managedClusterProfileobjects, the official Cluster Inventory API secretreader
plugin, Envoy Gateway, andcloud-provider-kind.1. Cluster and OCM setup
The OCM and Cluster Inventory API environment was created following the
OCM ClusterProfile guide:kind create cluster --name hub kind create cluster --name cluster1 kind create cluster --name cluster2 # Install OCM on hub, join cluster1 and cluster2, and enable ClusterProfile.# Install sandbox-fleet, cluster-proxy, and managed-serviceaccount. kubectl --context kind-hub get managedclusters kubectl --context kind-hub -n cluster-inventory get clusterprofiles kubectl --context kind-hub get managedclusteraddons -AVerified state:
cluster1andcluster2:Joined=True,Available=True- generated ClusterProfiles:
ControlPlaneHealthy=True,Joined=Truecluster-proxyandmanaged-serviceaccount:Available=True2. Spoke credentials and ClusterProfile access
An OCM
ManagedServiceAccountnamedknative-operatorwas created for each
managed cluster. The generated ClusterProfiles expose the cluster-proxy access
provider:kubectl --context kind-hub -n cluster1 get managedserviceaccount knative-operator kubectl --context kind-hub -n cluster-inventory get clusterprofile cluster1 -o yamlThe operator uses the official secretreader plugin:
{ "providers": [ { "name": "open-cluster-management", "execConfig": { "apiVersion": "client.authentication.k8s.io/v1", "command": "/access-plugins/secretreader/bin/secretreader-plugin", "provideClusterInfo": true, "interactiveMode": "Never" } } ] }The plugin image was
registry.k8s.io/cluster-inventory-api/secretreader:v0.1.3.3. Operator deployment
PATH="$(go env GOPATH)/bin:$PATH" \ KO_DOCKER_REPO=kind.local \ KIND_CLUSTER_NAME=hub \ ko apply -f config/ kubectl --context kind-hub -n knative-operator \ rollout status deployment/knative-operator --timeout=180sThe operator was configured with:
--clusterprofile-provider-file=/etc/cluster-inventory/config.json --remote-deployments-poll-interval=2sThe running operator reported commit
418c364and successfully resolved the
OCM cluster-proxy endpoint forcluster-inventory/cluster1.4. Install Envoy Gateway on cluster1
Gateway API v1.4.1 experimental CRDs were installed on the spoke. Envoy Gateway
v1.7.1 was then installed using the same GatewayNamespace deployment model as
the original multicluster PR:cat <<'EOF' > /tmp/values-eg.yamlconfig: envoyGateway: provider: type: Kubernetes kubernetes: deploy: type: GatewayNamespaceEOF kubectl --context kind-cluster1 create namespace envoy-gateway-system helm template eg oci://docker.io/envoyproxy/gateway-helm \ --version v1.7.1 \ -n envoy-gateway-system \ -f /tmp/values-eg.yaml \ --include-crds | \ sed -n '/^---$/,$p'| \ kubectl --context kind-cluster1 apply --server-side --force-conflicts -f - sudo cloud-provider-kind5. Create external and internal Gateway resources
Two Envoy Gateway instances were created:
eg-external/eg-external: LoadBalancer, ports 80 and 443eg-internal/eg-internal: ClusterIP, port 80apiVersion: v1kind: Namespacemetadata: name: eg-external --- apiVersion: gateway.envoyproxy.io/v1alpha1kind: EnvoyProxymetadata: name: knative-external-confignamespace: envoy-gateway-systemspec: provider: type: Kuberneteskubernetes: envoyService: name: knative-external --- apiVersion: gateway.networking.k8s.io/v1kind: GatewayClassmetadata: name: eg-externalspec: controllerName: gateway.envoyproxy.io/gatewayclass-controllerparametersRef: group: gateway.envoyproxy.iokind: EnvoyProxyname: knative-external-confignamespace: envoy-gateway-system --- apiVersion: gateway.networking.k8s.io/v1kind: Gatewaymetadata: name: eg-externalnamespace: eg-externalspec: gatewayClassName: eg-externallisteners: - name: httpport: 80protocol: HTTPallowedRoutes: namespaces: from: All - name: tlsport: 443protocol: TLStls: mode: PassthroughallowedRoutes: namespaces: from: All --- apiVersion: v1kind: Namespacemetadata: name: eg-internal --- apiVersion: gateway.envoyproxy.io/v1alpha1kind: EnvoyProxymetadata: name: knative-internal-confignamespace: envoy-gateway-systemspec: provider: type: Kuberneteskubernetes: envoyService: type: ClusterIPname: knative-internal --- apiVersion: gateway.networking.k8s.io/v1kind: GatewayClassmetadata: name: eg-internalspec: controllerName: gateway.envoyproxy.io/gatewayclass-controllerparametersRef: group: gateway.envoyproxy.iokind: EnvoyProxyname: knative-internal-confignamespace: envoy-gateway-system --- apiVersion: gateway.networking.k8s.io/v1kind: Gatewaymetadata: name: eg-internalnamespace: eg-internalspec: gatewayClassName: eg-internallisteners: - name: httpport: 80protocol: HTTPallowedRoutes: namespaces: from: AllBoth GatewayClasses reached
Accepted=True; both Gateways reachedProgrammed=True. The external service received172.18.0.12fromcloud-provider-kind.6. Deploy KnativeServing with an explicit destination
kubectl --context kind-hub apply -f - <<'EOF'apiVersion: operator.knative.dev/v1beta1kind: KnativeServingmetadata: name: ocm-serving namespace: management-serving-hubspec: destination: clusterProfileRef: name: cluster1 namespace: cluster-inventory namespace: knative-serving-ocm ingress: gateway-api: enabled: true istio: enabled: false config: config-gateway: external-gateways: | - class: eg-external gateway: eg-external/eg-external service: eg-external/knative-external supported-features: - HTTPRouteRequestTimeout local-gateways: | - class: eg-internal gateway: eg-internal/eg-internal service: eg-internal/knative-internal supported-features: - HTTPRouteRequestTimeout network: ingress-class: gateway-api.ingress.networking.knative.dev domain: example.com: ""EOFThe hub CR reached
Ready=TrueandTargetClusterResolved=True. All seven
Knative Serving Deployments became Available incluster1/knative-serving-ocm, with no Serving Deployment in the hub management
namespace or on cluster2.7. Verify ownership and data-plane traffic
The remote anchor was present and protected:
kubectl --context kind-cluster1 -n knative-serving-ocm \ get configmap knativeserving-ocm-serving-root-owner -o yaml kubectl --context kind-cluster1 -n knative-serving-ocm \ get deployment activator -o jsonpath='{.metadata.ownerReferences}'| jq .The
activatorDeployment ownerReference matched the anchor ConfigMap UID.A Knative Service was then deployed on cluster1:
kubectl --context kind-cluster1 apply -f - <<'EOF'apiVersion: serving.knative.dev/v1kind: Servicemetadata: name: helloworld-go namespace: defaultspec: template: spec: containers: - image: gcr.io/knative-samples/helloworld-go env: - name: TARGET value: Destination Kind E2EEOF kubectl --context kind-cluster1 -n default \ wait --for=condition=Ready kservice/helloworld-go --timeout=300s LB_IP=$(kubectl --context kind-cluster1 -n eg-external \ get service knative-external \ -o jsonpath='{.status.loadBalancer.ingress[0].ip}') curl -H 'Host: helloworld-go.default.example.com'"http://${LB_IP}"Observed response:
Hello Destination Kind E2E!The external and cluster-local HTTPRoutes both reported
Accepted=True.8. Verify CR deletion and finalizer cleanup
kubectl --context kind-cluster1 -n default delete kservice helloworld-go kubectl --context kind-hub -n management-serving-hub \ delete knativeserving ocm-servingVerified after finalization:
- the hub KnativeServing CR was deleted
- the remote anchor ConfigMap was deleted
- all labeled Knative Serving Deployments were deleted
- all Knative Serving ClusterRoles were deleted
The dynamically created
autoscaler-bucket-00-of-01Service and leader-election
Leases do not carry the anchor ownerReference and can remain after uninstall.
This also occurs onfork/mainand is not introduced by the destination API
replacement. Those test-only remnants were removed before reinstalling into the
same namespace.9. Recreate and leave the environment healthy
The same KnativeServing CR was recreated after cleanup. It returned to
Ready=TrueandTargetClusterResolved=True, and all seven Serving Deployments
became Available again. The OCM hub, both managed clusters, Envoy Gateway,
Knative Serving, and Knative Eventing were left running for inspection.Destination validation
The real API server rejected all of the following:
- invalid destination namespace
- missing ClusterProfile name, ClusterProfile namespace, or target namespace
- legacy top-level
spec.clusterProfileRef- changes to
destination.namespace- changes to
destination.clusterProfileRef.name- changes to
destination.clusterProfileRef.namespace- removal of
spec.destinationIntentional breaking change
Existing manifests using the unreleased top-level field must be rewritten:
# Beforespec: clusterProfileRef: name: cluster1namespace: cluster-inventory# Afterspec: destination: clusterProfileRef: name: cluster1namespace: cluster-inventorynamespace: knative-servingNo automatic migration or compatibility field is provided because the old API
has no current consumers.Release Note
action required: Replace spec.clusterProfileRef with spec.destination.clusterProfileRef and set spec.destination.namespace for remote Knative installations.
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: kahirokunn The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@## main #2344 +/- ##
==========================================
+ Coverage 63.99% 64.54% +0.55%
==========================================
Files 55 55 Lines 2491 2502 +11 ==========================================
+ Hits 1594 1615 +21 + Misses 777 761 -16 - Partials 120 126 +6 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
kahirokunn
commented
Aug 23, 2026
/retest |
418c364 to
449028eCompare449028e to
7345ad9CompareSigned-off-by: kahirokunn <okinakahiro@gmail.com>
7345ad9 to
8447d27CompareSigned-off-by: kahirokunn <okinakahiro@gmail.com>
8447d27 to
a5ca744Comparedsimansk
commented
Aug 24, 2026
@kahirokunn is running multiple CRs of the same kind in parallel namespaces supported use case now? I.e. KnativeServing being on |
kahirokunn
commented
Aug 26, 2026
No, multiple CRs of the same kind targeting the same cluster are not intended to be supported. Therefore, allowing an arbitrary destination namespace may be the wrong API design. The management CR may live in any hub namespace, while the remote installation namespace should probably be derived from the kind (knative-serving or knative-eventing). We should also prevent multiple CRs of the same kind from targeting the same ClusterProfile. |
kahirokunn
commented
Aug 26, 2026
Move to #2349 |
Fixes#2345
Proposed Changes
spec.clusterProfileReffield with an explicitspec.destinationobject containing bothclusterProfileRefand the remote installationnamespace.Design
This PR makes the destination explicit:
The complete
spec.destinationis immutable. Moving an existing installation between clusters or namespaces requires deleting and recreating the management CR so that remote finalization runs before the new destination is installed.Example CR
KnativeServing on a remote cluster
KnativeEventing on a remote cluster
Local cluster
Verification Steps
E2E verification with Kind
Verified with three Kind clusters (
hub,cluster1, andcluster2), OCM-managedClusterProfileobjects and cluster-proxy endpoints, the managed-serviceaccountClusterProfileCredSyncer,cp-creds, Envoy Gateway, andcloud-provider-kind.1. Set up OCM and Cluster Inventory API
The environment follows the OCM ClusterProfile guide: three Kind clusters, both spokes registered with OCM, the
sandbox-fleetManagedClusterSet, cluster-proxy, managed-serviceaccount, and ClusterProfile support.Enable ClusterProfile credential synchronization in the managed-serviceaccount add-on:
Bind the operator namespace to the ManagedClusterSet and label both ManagedServiceAccounts for credential synchronization:
Both ClusterProfiles reported
ControlPlaneHealthy=TrueandJoined=True. The managed-serviceaccount add-on synchronized their credentials into theknative-operatornamespace.2. Configure the operator to use cp-creds
The operator mounted
quay.io/open-cluster-management/cp-creds:latestas an image volume at/access-plugins/cp-credsand used this provider configuration:{ "providers": [ { "name": "open-cluster-management", "execConfig": { "apiVersion": "client.authentication.k8s.io/v1", "command": "/access-plugins/cp-creds/cp-creds", "args": ["--managed-serviceaccount=knative-operator"], "provideClusterInfo": true, "interactiveMode": "Never" } } ] }3. Build and deploy this PR head
PATH="$(go env GOPATH)/bin:$PATH" \ KO_DOCKER_REPO=kind.local \ KIND_CLUSTER_NAME=hub \ ko apply -f config/ kubectl --context kind-hub -n knative-operator \ rollout status deployment/knative-operator --timeout=180sBoth destination ClusterProfiles resolved through the configured
cp-credsprovider.4. Configure the Serving data plane on cluster1
Gateway API v1.4.1 experimental CRDs and Envoy Gateway v1.7.1 were installed on cluster1 using the same GatewayNamespace deployment model as the original multicluster PR. Both GatewayClasses reported
Accepted=True, and both Gateways reportedProgrammed=True.5. Deploy Serving and Eventing to explicit remote namespaces
The management CR namespaces intentionally differ from the remote installation namespaces:
Both hub CRs reached
TargetClusterResolved=TrueandReady=True. All seven Serving Deployments became Available only incluster1/knative-serving; Knative Eventing reachedReady=Truewith its controller, webhook, and broker Deployments only incluster2/knative-eventing; no Knative Deployments appeared in either hub management namespace or on the wrong spoke. The remote namespace metadata was applied, and every managed Deployment was owned by its remote anchor ConfigMap.6. Verify Serving traffic
Observed response:
The external and cluster-local HTTPRoutes both reported
Accepted=TrueandResolvedRefs=True.7. Verify cleanup with a cold cache
To exercise finalization before the ClusterProfile cache was warm, the operator was restarted and both management CRs were deleted immediately. Both remote anchors and their owned Deployments were removed, and both CRs completed deletion.
After recreating the CRs in the same destination namespaces, both returned to
TargetClusterResolved=TrueandReady=True.Breaking change
Existing manifests using the unreleased top-level field must be rewritten:
Release Note