Add multi-cluster deployment support via Cluster Inventory API - #2267

Merged
knative-prow[bot] merged 8 commits into
knative:mainfrom
kahirokunn:multi-cluster-support
Apr 22, 2026
Merged

Add multi-cluster deployment support via Cluster Inventory API#2267
knative-prow[bot] merged 8 commits into
knative:mainfrom
kahirokunn:multi-cluster-support

Conversation

@kahirokunn

@kahirokunnkahirokunn commented Apr 6, 2026•
edited
Loading

Copy link
Copy Markdown
Member

Fixes#2264

Requires knative/infra#827 to be merged first so that the integration-tests-multicluster presubmit exists when this PR's CI runs.

Proposed Changes

  • Enable deploying Knative Serving and Eventing components to remote clusters by setting spec.clusterProfileRef on the CR, using the SIG-Multicluster Cluster Inventory API (KEP-5339, ClusterProfile) to discover target clusters without depending on a specific fleet manager.
  • When spec.clusterProfileRef is not set, behavior is completely unchanged — existing single-cluster deployments are unaffected.
  • On CR deletion, the operator finalizes resources on the remote cluster. If the remote cluster is unreachable, the finalizer retries; operators can remove it manually to force-delete.

Design

The core idea is to swap the manifestival manifest.Client at the start of the reconcile stage pipeline (ResolveTargetCluster), so that all subsequent stages — Apply, Delete, Get — transparently operate on the remote cluster.

For garbage collection on the remote cluster, an anchor ConfigMap pattern (inspired by k0smotron) is used:

  • An anchor ConfigMap ({kind}-{cr-name}-root-owner) is created on the remote cluster.
  • All namespace-scoped resources get an ownerReference pointing to this anchor, enabling Kubernetes-native GC.
  • Cluster-scoped resources (ClusterRole, etc.) have no ownerReference and are explicitly deleted by the finalizer.
  • Deleting the anchor ConfigMap cascades to all namespace-scoped resources via GC.

Example CR

KnativeServing on a remote cluster

apiVersion: operator.knative.dev/v1beta1kind: KnativeServingmetadata:
name: knative-serving-apacnamespace: knative-servingspec:
version: "1.21"clusterProfileRef:
name: apac-cluster-01namespace: fleet-systemconfig:
network:
ingress-class: "kourier.ingress.networking.knative.dev"ingress:
kourier:
enabled: true

KnativeEventing on a remote cluster

apiVersion: operator.knative.dev/v1beta1kind: KnativeEventingmetadata:
name: knative-eventing-apacnamespace: knative-eventingspec:
version: "1.21"clusterProfileRef:
name: apac-cluster-01namespace: fleet-system

Local cluster (unchanged behavior)

apiVersion: operator.knative.dev/v1beta1kind: KnativeServingmetadata:
name: knative-servingnamespace: knative-servingspec:
version: "1.21"# No clusterProfileRef → deploys to the local cluster as before

Verification Steps

E2E verification with Kind

Note: The credential plugin used below (kubeconfig-secretreader-plugin) is a community plugin.
An official plugin is being tracked at kubernetes-sigs/cluster-inventory-api#45 —
once released, switch to the official one.

1. Cluster setup

# Create Kind clusters
kind create cluster --name hub
kind create cluster --name spoke
# Install ClusterProfile CRD on hub
kubectl --context kind-hub apply -f https://raw.githubusercontent.com/kubernetes-sigs/cluster-inventory-api/main/config/crd/bases/multicluster.x-k8s.io_clusterprofiles.yaml
# Create namespaces on hub
kubectl --context kind-hub create namespace fleet-system
kubectl --context kind-hub create namespace knative-serving
kubectl --context kind-hub create namespace knative-operator

2. Spoke cluster credentials

# Create a ServiceAccount + token Secret on spoke for the operator to use
kubectl --context kind-spoke apply -f - <<EOFapiVersion: v1kind: ServiceAccountmetadata: name: knative-operator namespace: default---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRoleBindingmetadata: name: knative-operator-adminroleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cluster-adminsubjects:- kind: ServiceAccount name: knative-operator namespace: default---apiVersion: v1kind: Secretmetadata: name: knative-operator-token namespace: default annotations: kubernetes.io/service-account.name: knative-operatortype: kubernetes.io/service-account-tokenEOF# Wait for the token to be populatedforiin$(seq 1 30);do
TOKEN=$(kubectl --context kind-spoke get secret knative-operator-token -o go-template='{{.data.token | base64decode}}'2>/dev/null)&& [ -n"$TOKEN" ] &&break
sleep 1
done# Copy the token as a Secret on the hub cluster
kubectl --context kind-hub -n knative-operator create secret generic spoke-token \
--from-literal=token="${TOKEN}" --dry-run=client -o yaml | kubectl --context kind-hub apply -f -

3. ClusterProfile

# Get spoke cluster connection info (Docker-internal IP, not 127.0.0.1)
SPOKE_SERVER="https://$(docker inspect spoke-control-plane --format '{{ .NetworkSettings.Networks.kind.IPAddress }}'):6443"
SPOKE_CA=$(kubectl --context kind-spoke config view --minify --flatten -o jsonpath='{.clusters[0].cluster.certificate-authority-data}')# Create ClusterProfile (spec only — status is a subresource)
kubectl --context kind-hub apply -f - <<EOFapiVersion: multicluster.x-k8s.io/v1alpha1kind: ClusterProfilemetadata: name: spoke-cluster namespace: fleet-systemspec: clusterManager: name: kindEOF# Patch status with access provider info.# The extensions tell the credential plugin which Secret to read the token from.
STATUS_PATCH=$(cat <<EOF{ "status": { "accessProviders": [ { "name": "token-secretreader", "cluster": { "server": "${SPOKE_SERVER}", "certificate-authority-data": "${SPOKE_CA}", "extensions": [ { "name": "client.authentication.k8s.io/exec", "extension": { "secretName": "spoke-token", "secretNamespace": "knative-operator", "key": "token" } } ] } } ] }}EOF)
kubectl --context kind-hub patch clusterprofile spoke-cluster \
-n fleet-system --type merge --subresource=status -p "${STATUS_PATCH}"

4. Operator deployment

# Build and deploy the operator with ko.# KIND_CLUSTER_NAME ensures the image is loaded into the correct Kind cluster.
kubectl config use-context kind-hub
KO_DOCKER_REPO=kind.local KIND_CLUSTER_NAME=hub ko apply -f config/
kubectl --context kind-hub -n knative-operator \
wait --for=condition=Available deployment/knative-operator --timeout=120s
# Create clusterprofile-provider-file ConfigMap (provider name must match accessProviders[].name)
kubectl --context kind-hub -n knative-operator create configmap clusterprofile-provider-file \
--from-literal=config.json='{"providers":[{"name":"token-secretreader","execConfig":{"apiVersion":"client.authentication.k8s.io/v1","command":"/credential-plugin/kubeconfig-secretreader-plugin","provideClusterInfo":true}}]}'# Patch the operator deployment:# - Mount the credential plugin binary via image volume# - Mount the clusterprofile-provider-file config# - Add the --clusterprofile-provider-file flag
kubectl --context kind-hub -n knative-operator patch deployment knative-operator --type json -p '[ {"op":"add","path":"/spec/template/spec/containers/0/args","value":["--clusterprofile-provider-file=/etc/cluster-inventory/config.json"]}, {"op":"add","path":"/spec/template/spec/containers/0/volumeMounts","value":[ {"name":"cred-config","mountPath":"/etc/cluster-inventory","readOnly":true}, {"name":"credential-plugin","mountPath":"/credential-plugin","readOnly":true} ]}, {"op":"add","path":"/spec/template/spec/volumes","value":[ {"name":"cred-config","configMap":{"name":"clusterprofile-provider-file"}}, {"name":"credential-plugin","image":{"reference":"ghcr.io/labthrust/kubeconfig-secretreader-plugin:v0.0.1-linux-arm64"}} ]}]'
kubectl --context kind-hub -n knative-operator rollout status deployment/knative-operator --timeout=120s

5. Install Envoy Gateway on the spoke cluster

The net-gateway-api ingress provider requires Gateway API resources (GatewayClass, Gateway) to exist on the target cluster before deploying KnativeServing. Install Envoy Gateway and create the necessary resources on the spoke cluster.

Prerequisites
go install sigs.k8s.io/cloud-provider-kind@latest
Install Envoy Gateway

Since cloud-provider-kind installs Gateway API CRDs with server-side apply, helm install will fail with field ownership conflicts. Use helm template + kubectl apply --server-side --force-conflicts to take over ownership.

cat <<'EOF' > /tmp/values-eg.yamlconfig: envoyGateway: provider: type: Kubernetes kubernetes: deploy: type: GatewayNamespaceEOFexport ENVOY_GATEWAY_VERSION=v1.7.1
kubectl --context kind-spoke create namespace envoy-gateway-system
helm template eg oci://docker.io/envoyproxy/gateway-helm \
--version $ENVOY_GATEWAY_VERSION \
-n envoy-gateway-system \
-f /tmp/values-eg.yaml \
--include-crds | \
kubectl --context kind-spoke apply --server-side --force-conflicts -f -

After installing Envoy Gateway, start cloud-provider-kind in a separate terminal (provides LoadBalancer support for Kind):

sudo cloud-provider-kind
Create Gateway API resources (external)
kubectl --context kind-spoke apply -f - <<EOFapiVersion: v1kind: Namespacemetadata: name: eg-external---apiVersion: gateway.envoyproxy.io/v1alpha1kind: EnvoyProxymetadata: name: knative-external-config namespace: envoy-gateway-systemspec: provider: type: Kubernetes kubernetes: envoyService: name: knative-external---apiVersion: gateway.networking.k8s.io/v1kind: GatewayClassmetadata: name: eg-externalspec: controllerName: gateway.envoyproxy.io/gatewayclass-controller parametersRef: group: gateway.envoyproxy.io kind: EnvoyProxy name: knative-external-config namespace: envoy-gateway-system---apiVersion: gateway.networking.k8s.io/v1kind: Gatewaymetadata: name: eg-external namespace: eg-externalspec: gatewayClassName: eg-external listeners: - name: http port: 80 protocol: HTTP allowedRoutes: namespaces: from: All - name: tls port: 443 protocol: TLS tls: mode: Passthrough allowedRoutes: namespaces: from: AllEOF
Create Gateway API resources (internal)

The internal Gateway uses the ClusterIP service type, so it is not accessible from outside the cluster.

kubectl --context kind-spoke apply -f - <<EOFapiVersion: v1kind: Namespacemetadata: name: eg-internal---apiVersion: gateway.envoyproxy.io/v1alpha1kind: EnvoyProxymetadata: name: knative-internal-config namespace: envoy-gateway-systemspec: provider: type: Kubernetes kubernetes: envoyService: type: ClusterIP name: knative-internal---apiVersion: gateway.networking.k8s.io/v1kind: GatewayClassmetadata: name: eg-internalspec: controllerName: gateway.envoyproxy.io/gatewayclass-controller parametersRef: group: gateway.envoyproxy.io kind: EnvoyProxy name: knative-internal-config namespace: envoy-gateway-system---apiVersion: gateway.networking.k8s.io/v1kind: Gatewaymetadata: name: eg-internal namespace: eg-internalspec: gatewayClassName: eg-internal listeners: - name: http port: 80 protocol: HTTP allowedRoutes: namespaces: from: AllEOF

6. Deploy KnativeServing

kubectl --context kind-hub apply -f - <<EOFapiVersion: operator.knative.dev/v1beta1kind: KnativeServingmetadata: name: knative-serving namespace: knative-servingspec: clusterProfileRef: name: spoke-cluster namespace: fleet-system ingress: gateway-api: enabled: true config: config-gateway: external-gateways: | - class: eg-external gateway: eg-external/eg-external service: eg-external/knative-external supported-features: - HTTPRouteRequestTimeout local-gateways: | - class: eg-internal gateway: eg-internal/eg-internal service: eg-internal/knative-internal supported-features: - HTTPRouteRequestTimeout network: ingress.class: gateway-api.ingress.networking.knative.dev domain: example.com: ""EOF

7. Verify

# Verify resources deployed to spoke cluster
kubectl --context kind-spoke get deployments -n knative-serving
kubectl --context kind-spoke get configmaps -n knative-serving | grep root-owner
# Verify anchor ConfigMap exists with correct labels
kubectl --context kind-spoke get configmap knativeserving-knative-serving-root-owner \
-n knative-serving -o yaml
# Verify ownerReferences on namespace-scoped resources point to the anchor
kubectl --context kind-spoke get deployment activator -n knative-serving \
-o jsonpath='{.metadata.ownerReferences}'| jq .# Deploy a sample application to verify traffic routing
kubectl --context kind-spoke apply -f - <<EOFapiVersion: serving.knative.dev/v1kind: Servicemetadata: name: helloworld-go namespace: defaultspec: template: spec: containers: - image: gcr.io/knative-samples/helloworld-go env: - name: TARGET value: Go Sample v1EOF# Get the external gateway IP and testexport LB_IP=$(kubectl --context kind-spoke -n eg-external get svc knative-external \ -o jsonpath='{.status.loadBalancer.ingress[0].ip}')echo"$LB_IP helloworld-go.default.example.com"| sudo tee -a /etc/hosts
curl http://helloworld-go.default.example.com

8. Verify CR deletion and finalizer cleanup

# Delete the CR and wait for the finalizer to complete
kubectl --context kind-hub delete knativeserving knative-serving -n knative-serving
# Verify spoke cluster is clean
kubectl --context kind-spoke get all -n knative-serving
kubectl --context kind-spoke get clusterroles | grep knative

9. Cleanup

kind delete cluster --name hub
kind delete cluster --name spoke

Backward compatibility

# Deploy without --clusterprofile-provider-file and without clusterProfileRef# → must behave identically to previous releases
kubectl apply -f - <<EOFapiVersion: operator.knative.dev/v1beta1kind: KnativeServingmetadata: name: knative-serving namespace: knative-servingspec: version: "1.21"EOF
kubectl get knativeserving -n knative-serving -o jsonpath='{.status.conditions}'

Release Note

Add multi-cluster deployment support via the Cluster Inventory API (KEP-5339). When `spec.clusterProfileRef` is set on a KnativeServing or KnativeEventing CR, the operator deploys components to the remote cluster described by the referenced ClusterProfile. Requires the `--clusterprofile-provider-file` flag to be set on the operator deployment.

@knative-prowknative-prowBot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Apr 6, 2026
knative-prow[bot]
knative-prowBot reviewed Apr 6, 2026
View reviewed changes

@knative-prowknative-prowBot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kahirokunn: 0 warnings.

Details

In response to this:

Fixes #

Proposed Changes

Release Note

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@codecov

codecovBot commented Apr 6, 2026•
edited
Loading

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 60.40000% with 198 lines in your changes missing coverage. Please review.
✅ Project coverage is 63.84%. Comparing base (9c66d7b) to head (72fc85c).
⚠️ Report is 1 commits behind head on main.

Files with missing linesPatch %Lines
pkg/reconciler/common/multicluster.go62.18%121 Missing and 14 partials ⚠️
pkg/reconciler/knativeeventing/controller.go0.00%25 Missing ⚠️
pkg/reconciler/knativeserving/controller.go0.00%25 Missing ⚠️
pkg/reconciler/common/clusterprofile_informer.go86.53%3 Missing and 4 partials ⚠️
pkg/apis/operator/base/common.go0.00%2 Missing ⚠️
pkg/reconciler/common/access_provider_flag.go66.66%2 Missing ⚠️
pkg/reconciler/knativeeventing/eventing_tls.go0.00%2 Missing ⚠️
Additional details and impacted files
@@ Coverage Diff @@## main #2267 +/- ##
==========================================
- Coverage 64.58% 63.84% -0.75% 
==========================================
Files 51 55 +4 Lines 1999 2478 +479 ==========================================
+ Hits 1291 1582 +291 - Misses 606 777 +171 - Partials 102 119 +17

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@kahirokunnkahirokunn changed the title Add multi-cluster deployment support via Cluster Inventory API[WIP] Add multi-cluster deployment support via Cluster Inventory APIApr 6, 2026
@knative-prowknative-prowBot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 6, 2026
@kahirokunn
kahirokunnforce-pushed the multi-cluster-support branch 5 times, most recently from 3e585a9 to 141c875CompareApril 7, 2026 06:40
@knative-prow-robotknative-prow-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 7, 2026
@kahirokunn
kahirokunnforce-pushed the multi-cluster-support branch 2 times, most recently from d308bf0 to 3e742cbCompareApril 7, 2026 12:53
@kahirokunnkahirokunn changed the title [WIP] Add multi-cluster deployment support via Cluster Inventory APIAdd multi-cluster deployment support via Cluster Inventory APIApr 7, 2026
@knative-prowknative-prowBot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 7, 2026
@kahirokunn
kahirokunnforce-pushed the multi-cluster-support branch from 3e742cb to 43919c6CompareApril 7, 2026 14:11
@knative-prow-robotknative-prow-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 7, 2026
@kahirokunn
kahirokunnforce-pushed the multi-cluster-support branch from 43919c6 to 28a1e02CompareApril 7, 2026 14:22
@knative-prow-robotknative-prow-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 7, 2026
@kahirokunn
kahirokunnforce-pushed the multi-cluster-support branch from 28a1e02 to 88410efCompareApril 7, 2026 14:46
@knative-prow

knative-prowBot commented Apr 7, 2026

Copy link
Copy Markdown

There are empty aliases in OWNER_ALIASES, cleanup is advised.

@knative-prow-robotknative-prow-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 7, 2026
@kahirokunnkahirokunn changed the title Add multi-cluster deployment support via Cluster Inventory API[WIP] Add multi-cluster deployment support via Cluster Inventory APIApr 7, 2026
@knative-prowknative-prowBot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 7, 2026
@knative-prow

knative-prowBot commented Apr 7, 2026

Copy link
Copy Markdown

There are empty aliases in OWNER_ALIASES, cleanup is advised.

@kahirokunn
kahirokunnforce-pushed the multi-cluster-support branch 3 times, most recently from 29a9317 to 6217b27CompareApril 8, 2026 04:05
@knative-prow-robotknative-prow-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 16, 2026
@kahirokunn
kahirokunnforce-pushed the multi-cluster-support branch 2 times, most recently from 9068c06 to cecc1b8CompareApril 16, 2026 05:54
@knative-prow-robotknative-prow-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 16, 2026
@knative-prow

knative-prowBot commented Apr 16, 2026

Copy link
Copy Markdown

There are empty aliases in OWNER_ALIASES, cleanup is advised.

@kahirokunnkahirokunn changed the title [WIP] Add multi-cluster deployment support via Cluster Inventory APIAdd multi-cluster deployment support via Cluster Inventory APIApr 16, 2026
@knative-prowknative-prowBot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 16, 2026
@kahirokunn
kahirokunnforce-pushed the multi-cluster-support branch from cecc1b8 to 725b1beCompareApril 16, 2026 08:34
@kahirokunn

kahirokunn commented Apr 17, 2026

Copy link
Copy Markdown
MemberAuthor

I didn't have the necessary permissions.
integration-tests-multicluster_operator_main will keep failing until the following is merged, so I'd like to ignore it using an override until then #2267

@kahirokunn

kahirokunn commented Apr 17, 2026

Copy link
Copy Markdown
MemberAuthor

/retest

houshengbo
houshengbo reviewed Apr 17, 2026
View reviewed changes

@houshengbohoushengbo left a comment •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review

The PR description is excellent — the verification steps, design explanation, example CRs, and backward compatibility section are thorough and well-written.

2 blocking items and 7 non-blocking items are noted in inline comments below.

Comment threadpkg/apis/operator/v1beta1/knativeserving_lifecycle.go Outdated
Comment threadpkg/reconciler/common/clusterprofile_informer.go
Comment threadpkg/reconciler/common/multicluster.go
Comment threadpkg/reconciler/common/multicluster.go
Comment threadpkg/reconciler/common/multicluster.go
Comment threadpkg/reconciler/common/transformers.go
Comment threadpkg/reconciler/common/stages.go Outdated
Comment threadtest/e2e/knativeserving_spoke_test.go
@kahirokunn

kahirokunn commented Apr 21, 2026

Copy link
Copy Markdown
MemberAuthor

@houshengbo Thank you for your feedback! I have addressed all the points you raised.

@kahirokunn

kahirokunn commented Apr 21, 2026

Copy link
Copy Markdown
MemberAuthor

I will check it

@kahirokunn
add multi-cluster deployment support via Cluster Inventory API
4845dee
Signed-off-by: kahirokunn <okinakahiro@gmail.com>
@kahirokunn
change MarkTargetClusterNotResolved to mark only TargetClusterResolved
d1e2351
Signed-off-by: kahirokunn <okinakahiro@gmail.com>
@kahirokunn
add doRefresh error path tests for ClusterProvider
c3c500f
Signed-off-by: kahirokunn <okinakahiro@gmail.com>
@kahirokunn
mark anchor ConfigMap as protected to surface deletion risk
6b7c4e0
Signed-off-by: kahirokunn <okinakahiro@gmail.com>
@kahirokunn
stabilise ClusterProvider shutdown and propagate namespace config
4c2c9b4
Signed-off-by: kahirokunn <okinakahiro@gmail.com>
@kahirokunn
add --remote-deployments-poll-interval CLI flag
5093973
Signed-off-by: kahirokunn <okinakahiro@gmail.com>
@kahirokunn
add multi-cluster e2e test for KnativeEventing
b901f31
Signed-off-by: kahirokunn <okinakahiro@gmail.com>
@kahirokunn
./hack/update-codegen.sh
72fc85c
@kahirokunn

kahirokunn commented Apr 21, 2026

Copy link
Copy Markdown
MemberAuthor

All CI passed! 🙌

@houshengbo

houshengbo commented Apr 22, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

/lgtm
/approve

@knative-prow

knative-prowBot commented Apr 22, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: houshengbo, kahirokunn

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kahirokunn

kahirokunn commented Apr 22, 2026

Copy link
Copy Markdown
MemberAuthor

Thank you😆

@kahirokunnkahirokunn mentioned this pull request Apr 23, 2026
Merged
@kahirokunnkahirokunn mentioned this pull request May 26, 2026
Closed
This was referenced Aug 22, 2026
Closed
Closed
Closed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@knative-prowknative-prow[bot]knative-prow[bot] left review comments
@matzewmatzewAwaiting requested review from matzew
@houshengbohoushengboAwaiting requested review from houshengbo

Assignees

@houshengbohoushengbo

Labels

approvedIndicates a PR has been approved by an approver from all required OWNERS files.lgtmIndicates that a PR is ready to be merged.size/XXLDenotes a PR that changes 1000+ lines, ignoring generated files.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Support multi-cluster deployment via Cluster Inventory API

3 participants

@kahirokunn@houshengbo@knative-prow-robot
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
Add multi-cluster deployment support via Cluster Inventory API - #2267
Merged
knative-prow[bot] merged 8 commits into
knative:mainfrom 
kahirokunn:multi-cluster-support
Apr 22, 2026
Merged
Add multi-cluster deployment support via Cluster Inventory API#2267
knative-prow[bot] merged 8 commits into
knative:mainfrom 
kahirokunn:multi-cluster-support
Conversation
@kahirokunn
@kahirokunnkahirokunn
commented
Apr 6, 2026•

edited

Loading

Copy link


Copy Markdown

Member
Fixes#2264
Requires knative/infra#827 to be merged first so that the integration-tests-multicluster presubmit exists when this PR's CI runs.
Proposed Changes
  • Enable deploying Knative Serving and Eventing components to remote clusters by setting spec.clusterProfileRef on the CR, using the SIG-Multicluster Cluster Inventory API (KEP-5339, ClusterProfile) to discover target clusters without depending on a specific fleet manager.
  • When spec.clusterProfileRef is not set, behavior is completely unchanged — existing single-cluster deployments are unaffected.
  • On CR deletion, the operator finalizes resources on the remote cluster. If the remote cluster is unreachable, the finalizer retries; operators can remove it manually to force-delete.
Design
The core idea is to swap the manifestival manifest.Client at the start of the reconcile stage pipeline (ResolveTargetCluster), so that all subsequent stages — Apply, Delete, Get — transparently operate on the remote cluster.
For garbage collection on the remote cluster, an anchor ConfigMap pattern (inspired by k0smotron) is used:
  • An anchor ConfigMap ({kind}-{cr-name}-root-owner) is created on the remote cluster.
  • All namespace-scoped resources get an ownerReference pointing to this anchor, enabling Kubernetes-native GC.
  • Cluster-scoped resources (ClusterRole, etc.) have no ownerReference and are explicitly deleted by the finalizer.
  • Deleting the anchor ConfigMap cascades to all namespace-scoped resources via GC.
Example CR
KnativeServing on a remote cluster
apiVersion: operator.knative.dev/v1beta1kind: KnativeServingmetadata:
name: knative-serving-apacnamespace: knative-servingspec:
version: "1.21"clusterProfileRef:
name: apac-cluster-01namespace: fleet-systemconfig:
network:
ingress-class: "kourier.ingress.networking.knative.dev"ingress:
kourier:
enabled: true
KnativeEventing on a remote cluster
apiVersion: operator.knative.dev/v1beta1kind: KnativeEventingmetadata:
name: knative-eventing-apacnamespace: knative-eventingspec:
version: "1.21"clusterProfileRef:
name: apac-cluster-01namespace: fleet-system
Local cluster (unchanged behavior)
apiVersion: operator.knative.dev/v1beta1kind: KnativeServingmetadata:
name: knative-servingnamespace: knative-servingspec:
version: "1.21"# No clusterProfileRef → deploys to the local cluster as before
Verification Steps
E2E verification with Kind
Note: The credential plugin used below (kubeconfig-secretreader-plugin) is a community plugin.

An official plugin is being tracked at kubernetes-sigs/cluster-inventory-api#45 —

once released, switch to the official one.
1. Cluster setup
# Create Kind clusters
kind create cluster --name hub
kind create cluster --name spoke
# Install ClusterProfile CRD on hub
kubectl --context kind-hub apply -f https://raw.githubusercontent.com/kubernetes-sigs/cluster-inventory-api/main/config/crd/bases/multicluster.x-k8s.io_clusterprofiles.yaml
# Create namespaces on hub
kubectl --context kind-hub create namespace fleet-system
kubectl --context kind-hub create namespace knative-serving
kubectl --context kind-hub create namespace knative-operator
2. Spoke cluster credentials
# Create a ServiceAccount + token Secret on spoke for the operator to use
kubectl --context kind-spoke apply -f - <<EOFapiVersion: v1kind: ServiceAccountmetadata: name: knative-operator namespace: default---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRoleBindingmetadata: name: knative-operator-adminroleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cluster-adminsubjects:- kind: ServiceAccount name: knative-operator namespace: default---apiVersion: v1kind: Secretmetadata: name: knative-operator-token namespace: default annotations: kubernetes.io/service-account.name: knative-operatortype: kubernetes.io/service-account-tokenEOF# Wait for the token to be populatedforiin$(seq 1 30);do
TOKEN=$(kubectl --context kind-spoke get secret knative-operator-token -o go-template='{{.data.token | base64decode}}'2>/dev/null)&& [ -n"$TOKEN" ] &&break
sleep 1
done# Copy the token as a Secret on the hub cluster
kubectl --context kind-hub -n knative-operator create secret generic spoke-token \
--from-literal=token="${TOKEN}" --dry-run=client -o yaml | kubectl --context kind-hub apply -f -
3. ClusterProfile
# Get spoke cluster connection info (Docker-internal IP, not 127.0.0.1)
SPOKE_SERVER="https://$(docker inspect spoke-control-plane --format '{{ .NetworkSettings.Networks.kind.IPAddress }}'):6443"
SPOKE_CA=$(kubectl --context kind-spoke config view --minify --flatten -o jsonpath='{.clusters[0].cluster.certificate-authority-data}')# Create ClusterProfile (spec only — status is a subresource)
kubectl --context kind-hub apply -f - <<EOFapiVersion: multicluster.x-k8s.io/v1alpha1kind: ClusterProfilemetadata: name: spoke-cluster namespace: fleet-systemspec: clusterManager: name: kindEOF# Patch status with access provider info.# The extensions tell the credential plugin which Secret to read the token from.
STATUS_PATCH=$(cat <<EOF{ "status": { "accessProviders": [ { "name": "token-secretreader", "cluster": { "server": "${SPOKE_SERVER}", "certificate-authority-data": "${SPOKE_CA}", "extensions": [ { "name": "client.authentication.k8s.io/exec", "extension": { "secretName": "spoke-token", "secretNamespace": "knative-operator", "key": "token" } } ] } } ] }}EOF)
kubectl --context kind-hub patch clusterprofile spoke-cluster \
-n fleet-system --type merge --subresource=status -p "${STATUS_PATCH}"
4. Operator deployment
# Build and deploy the operator with ko.# KIND_CLUSTER_NAME ensures the image is loaded into the correct Kind cluster.
kubectl config use-context kind-hub
KO_DOCKER_REPO=kind.local KIND_CLUSTER_NAME=hub ko apply -f config/
kubectl --context kind-hub -n knative-operator \
wait --for=condition=Available deployment/knative-operator --timeout=120s
# Create clusterprofile-provider-file ConfigMap (provider name must match accessProviders[].name)
kubectl --context kind-hub -n knative-operator create configmap clusterprofile-provider-file \
--from-literal=config.json='{"providers":[{"name":"token-secretreader","execConfig":{"apiVersion":"client.authentication.k8s.io/v1","command":"/credential-plugin/kubeconfig-secretreader-plugin","provideClusterInfo":true}}]}'# Patch the operator deployment:# - Mount the credential plugin binary via image volume# - Mount the clusterprofile-provider-file config# - Add the --clusterprofile-provider-file flag
kubectl --context kind-hub -n knative-operator patch deployment knative-operator --type json -p '[ {"op":"add","path":"/spec/template/spec/containers/0/args","value":["--clusterprofile-provider-file=/etc/cluster-inventory/config.json"]}, {"op":"add","path":"/spec/template/spec/containers/0/volumeMounts","value":[ {"name":"cred-config","mountPath":"/etc/cluster-inventory","readOnly":true}, {"name":"credential-plugin","mountPath":"/credential-plugin","readOnly":true} ]}, {"op":"add","path":"/spec/template/spec/volumes","value":[ {"name":"cred-config","configMap":{"name":"clusterprofile-provider-file"}}, {"name":"credential-plugin","image":{"reference":"ghcr.io/labthrust/kubeconfig-secretreader-plugin:v0.0.1-linux-arm64"}} ]}]'
kubectl --context kind-hub -n knative-operator rollout status deployment/knative-operator --timeout=120s
5. Install Envoy Gateway on the spoke cluster
The net-gateway-api ingress provider requires Gateway API resources (GatewayClass, Gateway) to exist on the target cluster before deploying KnativeServing. Install Envoy Gateway and create the necessary resources on the spoke cluster.
Prerequisites
go install sigs.k8s.io/cloud-provider-kind@latest
Install Envoy Gateway
Since cloud-provider-kind installs Gateway API CRDs with server-side apply, helm install will fail with field ownership conflicts. Use helm template + kubectl apply --server-side --force-conflicts to take over ownership.
cat <<'EOF' > /tmp/values-eg.yamlconfig: envoyGateway: provider: type: Kubernetes kubernetes: deploy: type: GatewayNamespaceEOFexport ENVOY_GATEWAY_VERSION=v1.7.1
kubectl --context kind-spoke create namespace envoy-gateway-system
helm template eg oci://docker.io/envoyproxy/gateway-helm \
--version $ENVOY_GATEWAY_VERSION \
-n envoy-gateway-system \
-f /tmp/values-eg.yaml \
--include-crds | \
kubectl --context kind-spoke apply --server-side --force-conflicts -f -
After installing Envoy Gateway, start cloud-provider-kind in a separate terminal (provides LoadBalancer support for Kind):
sudo cloud-provider-kind
Create Gateway API resources (external)
kubectl --context kind-spoke apply -f - <<EOFapiVersion: v1kind: Namespacemetadata: name: eg-external---apiVersion: gateway.envoyproxy.io/v1alpha1kind: EnvoyProxymetadata: name: knative-external-config namespace: envoy-gateway-systemspec: provider: type: Kubernetes kubernetes: envoyService: name: knative-external---apiVersion: gateway.networking.k8s.io/v1kind: GatewayClassmetadata: name: eg-externalspec: controllerName: gateway.envoyproxy.io/gatewayclass-controller parametersRef: group: gateway.envoyproxy.io kind: EnvoyProxy name: knative-external-config namespace: envoy-gateway-system---apiVersion: gateway.networking.k8s.io/v1kind: Gatewaymetadata: name: eg-external namespace: eg-externalspec: gatewayClassName: eg-external listeners: - name: http port: 80 protocol: HTTP allowedRoutes: namespaces: from: All - name: tls port: 443 protocol: TLS tls: mode: Passthrough allowedRoutes: namespaces: from: AllEOF
Create Gateway API resources (internal)
The internal Gateway uses the ClusterIP service type, so it is not accessible from outside the cluster.
kubectl --context kind-spoke apply -f - <<EOFapiVersion: v1kind: Namespacemetadata: name: eg-internal---apiVersion: gateway.envoyproxy.io/v1alpha1kind: EnvoyProxymetadata: name: knative-internal-config namespace: envoy-gateway-systemspec: provider: type: Kubernetes kubernetes: envoyService: type: ClusterIP name: knative-internal---apiVersion: gateway.networking.k8s.io/v1kind: GatewayClassmetadata: name: eg-internalspec: controllerName: gateway.envoyproxy.io/gatewayclass-controller parametersRef: group: gateway.envoyproxy.io kind: EnvoyProxy name: knative-internal-config namespace: envoy-gateway-system---apiVersion: gateway.networking.k8s.io/v1kind: Gatewaymetadata: name: eg-internal namespace: eg-internalspec: gatewayClassName: eg-internal listeners: - name: http port: 80 protocol: HTTP allowedRoutes: namespaces: from: AllEOF
6. Deploy KnativeServing
kubectl --context kind-hub apply -f - <<EOFapiVersion: operator.knative.dev/v1beta1kind: KnativeServingmetadata: name: knative-serving namespace: knative-servingspec: clusterProfileRef: name: spoke-cluster namespace: fleet-system ingress: gateway-api: enabled: true config: config-gateway: external-gateways: | - class: eg-external gateway: eg-external/eg-external service: eg-external/knative-external supported-features: - HTTPRouteRequestTimeout local-gateways: | - class: eg-internal gateway: eg-internal/eg-internal service: eg-internal/knative-internal supported-features: - HTTPRouteRequestTimeout network: ingress.class: gateway-api.ingress.networking.knative.dev domain: example.com: ""EOF
7. Verify
# Verify resources deployed to spoke cluster
kubectl --context kind-spoke get deployments -n knative-serving
kubectl --context kind-spoke get configmaps -n knative-serving | grep root-owner
# Verify anchor ConfigMap exists with correct labels
kubectl --context kind-spoke get configmap knativeserving-knative-serving-root-owner \
-n knative-serving -o yaml
# Verify ownerReferences on namespace-scoped resources point to the anchor
kubectl --context kind-spoke get deployment activator -n knative-serving \
-o jsonpath='{.metadata.ownerReferences}'| jq .# Deploy a sample application to verify traffic routing
kubectl --context kind-spoke apply -f - <<EOFapiVersion: serving.knative.dev/v1kind: Servicemetadata: name: helloworld-go namespace: defaultspec: template: spec: containers: - image: gcr.io/knative-samples/helloworld-go env: - name: TARGET value: Go Sample v1EOF# Get the external gateway IP and testexport LB_IP=$(kubectl --context kind-spoke -n eg-external get svc knative-external \ -o jsonpath='{.status.loadBalancer.ingress[0].ip}')echo"$LB_IP helloworld-go.default.example.com"| sudo tee -a /etc/hosts
curl http://helloworld-go.default.example.com

8. Verify CR deletion and finalizer cleanup
# Delete the CR and wait for the finalizer to complete
kubectl --context kind-hub delete knativeserving knative-serving -n knative-serving
# Verify spoke cluster is clean
kubectl --context kind-spoke get all -n knative-serving
kubectl --context kind-spoke get clusterroles | grep knative
9. Cleanup
kind delete cluster --name hub
kind delete cluster --name spoke
Backward compatibility
# Deploy without --clusterprofile-provider-file and without clusterProfileRef# → must behave identically to previous releases
kubectl apply -f - <<EOFapiVersion: operator.knative.dev/v1beta1kind: KnativeServingmetadata: name: knative-serving namespace: knative-servingspec: version: "1.21"EOF
kubectl get knativeserving -n knative-serving -o jsonpath='{.status.conditions}'
Release Note
Add multi-cluster deployment support via the Cluster Inventory API (KEP-5339). When `spec.clusterProfileRef` is set on a KnativeServing or KnativeEventing CR, the operator deploys components to the remote cluster described by the referenced ClusterProfile. Requires the `--clusterprofile-provider-file` flag to be set on the operator deployment.

@knative-prowknative-prowBot
added
the 
size/XXL
Denotes a PR that changes 1000+ lines, ignoring generated files.
label
Apr 6, 2026
knative-prow[bot]
knative-prowBot
reviewed
Apr 6, 2026
View reviewed changes
@knative-prowknative-prowBot
left a comment


Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kahirokunn: 0 warnings.
Details
In response to this:
Fixes #
Proposed Changes
Release Note
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.
@codecov
codecovBot
commented
Apr 6, 2026•

edited

Loading

Copy link


Copy Markdown

Codecov Report
❌ Patch coverage is 60.40000% with 198 lines in your changes missing coverage. Please review.

✅ Project coverage is 63.84%. Comparing base (9c66d7b) to head (72fc85c).
⚠️ Report is 1 commits behind head on main.
Files with missing linesPatch %Lines
pkg/reconciler/common/multicluster.go62.18%121 Missing and 14 partials ⚠️
pkg/reconciler/knativeeventing/controller.go0.00%25 Missing ⚠️
pkg/reconciler/knativeserving/controller.go0.00%25 Missing ⚠️
pkg/reconciler/common/clusterprofile_informer.go86.53%3 Missing and 4 partials ⚠️
pkg/apis/operator/base/common.go0.00%2 Missing ⚠️
pkg/reconciler/common/access_provider_flag.go66.66%2 Missing ⚠️
pkg/reconciler/knativeeventing/eventing_tls.go0.00%2 Missing ⚠️
Additional details and impacted files
@@ Coverage Diff @@## main #2267 +/- ##
==========================================
- Coverage 64.58% 63.84% -0.75% 
==========================================
Files 51 55 +4 Lines 1999 2478 +479 ==========================================
+ Hits 1291 1582 +291 - Misses 606 777 +171 - Partials 102 119 +17 
☔ View full report in Codecov by Sentry.

📢 Have feedback on the report? Share it here.
 🚀 New features to boost your workflow: 
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
@kahirokunnkahirokunn
changed the title
Add multi-cluster deployment support via Cluster Inventory API[WIP] Add multi-cluster deployment support via Cluster Inventory APIApr 6, 2026
@knative-prowknative-prowBot
added
the 
do-not-merge/work-in-progress
Indicates that a PR should not merge because it is a work in progress.
label
Apr 6, 2026
@kahirokunn
kahirokunnforce-pushed
the
multi-cluster-support
branch
5 times, most recently
from
3e585a9 to
141c875CompareApril 7, 2026 06:40
@knative-prow-robotknative-prow-robot
added
the 
needs-rebase
Indicates a PR cannot be merged because it has merge conflicts with HEAD.
label
Apr 7, 2026
@kahirokunn
kahirokunnforce-pushed
the
multi-cluster-support
branch
2 times, most recently
from
d308bf0 to
3e742cbCompareApril 7, 2026 12:53
@kahirokunnkahirokunn
changed the title
[WIP] Add multi-cluster deployment support via Cluster Inventory APIAdd multi-cluster deployment support via Cluster Inventory APIApr 7, 2026
@knative-prowknative-prowBot
removed
the 
do-not-merge/work-in-progress
Indicates that a PR should not merge because it is a work in progress.
label
Apr 7, 2026
@kahirokunn
kahirokunnforce-pushed
the
multi-cluster-support
branch
from
3e742cb to
43919c6CompareApril 7, 2026 14:11
@knative-prow-robotknative-prow-robot
removed
the 
needs-rebase
Indicates a PR cannot be merged because it has merge conflicts with HEAD.
label
Apr 7, 2026
@kahirokunn
kahirokunnforce-pushed
the
multi-cluster-support
branch
from
43919c6 to
28a1e02CompareApril 7, 2026 14:22
@knative-prow-robotknative-prow-robot
added
the 
needs-rebase
Indicates a PR cannot be merged because it has merge conflicts with HEAD.
label
Apr 7, 2026
@kahirokunn
kahirokunnforce-pushed
the
multi-cluster-support
branch
from
28a1e02 to
88410efCompareApril 7, 2026 14:46
@knative-prow
knative-prowBot
commented
Apr 7, 2026

Copy link


Copy Markdown

There are empty aliases in OWNER_ALIASES, cleanup is advised.
@knative-prow-robotknative-prow-robot
removed
the 
needs-rebase
Indicates a PR cannot be merged because it has merge conflicts with HEAD.
label
Apr 7, 2026
@kahirokunnkahirokunn
changed the title
Add multi-cluster deployment support via Cluster Inventory API[WIP] Add multi-cluster deployment support via Cluster Inventory APIApr 7, 2026
@knative-prowknative-prowBot
added
the 
do-not-merge/work-in-progress
Indicates that a PR should not merge because it is a work in progress.
label
Apr 7, 2026
@knative-prow
knative-prowBot
commented
Apr 7, 2026

Copy link


Copy Markdown

There are empty aliases in OWNER_ALIASES, cleanup is advised.
@kahirokunn
kahirokunnforce-pushed
the
multi-cluster-support
branch
3 times, most recently
from
29a9317 to
6217b27CompareApril 8, 2026 04:05
@knative-prow-robotknative-prow-robot
added
the 
needs-rebase
Indicates a PR cannot be merged because it has merge conflicts with HEAD.
label
Apr 16, 2026
@kahirokunn
kahirokunnforce-pushed
the
multi-cluster-support
branch
2 times, most recently
from
9068c06 to
cecc1b8CompareApril 16, 2026 05:54
@knative-prow-robotknative-prow-robot
removed
the 
needs-rebase
Indicates a PR cannot be merged because it has merge conflicts with HEAD.
label
Apr 16, 2026
@knative-prow
knative-prowBot
commented
Apr 16, 2026

Copy link


Copy Markdown

There are empty aliases in OWNER_ALIASES, cleanup is advised.
@kahirokunnkahirokunn
changed the title
[WIP] Add multi-cluster deployment support via Cluster Inventory APIAdd multi-cluster deployment support via Cluster Inventory APIApr 16, 2026
@knative-prowknative-prowBot
removed
the 
do-not-merge/work-in-progress
Indicates that a PR should not merge because it is a work in progress.
label
Apr 16, 2026
@kahirokunn
kahirokunnforce-pushed
the
multi-cluster-support
branch
from
cecc1b8 to
725b1beCompareApril 16, 2026 08:34
@kahirokunn
kahirokunn
commented
Apr 17, 2026

Copy link


Copy Markdown

MemberAuthor
I didn't have the necessary permissions.
integration-tests-multicluster_operator_main will keep failing until the following is merged, so I'd like to ignore it using an override until then #2267
@kahirokunn
kahirokunn
commented
Apr 17, 2026

Copy link


Copy Markdown

MemberAuthor
/retest
houshengbo
houshengbo
reviewed
Apr 17, 2026
View reviewed changes
@houshengbohoushengbo
left a comment
•

edited

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review
The PR description is excellent — the verification steps, design explanation, example CRs, and backward compatibility section are thorough and well-written.
2 blocking items and 7 non-blocking items are noted in inline comments below.
Comment threadpkg/apis/operator/v1beta1/knativeserving_lifecycle.go
Outdated

Comment threadpkg/reconciler/common/clusterprofile_informer.go
Comment threadpkg/reconciler/common/multicluster.go
Comment threadpkg/reconciler/common/multicluster.go
Comment threadpkg/reconciler/common/multicluster.go
Comment threadpkg/reconciler/common/transformers.go
Comment threadpkg/reconciler/common/stages.go
Outdated

Comment threadtest/e2e/knativeserving_spoke_test.go
@kahirokunn
kahirokunn
commented
Apr 21, 2026

Copy link


Copy Markdown

MemberAuthor
@houshengbo Thank you for your feedback! I have addressed all the points you raised.
@kahirokunn
kahirokunn
commented
Apr 21, 2026

Copy link


Copy Markdown

MemberAuthor
I will check it
@kahirokunn
add multi-cluster deployment support via Cluster Inventory API
4845dee
Signed-off-by: kahirokunn <okinakahiro@gmail.com>
@kahirokunn
change MarkTargetClusterNotResolved to mark only TargetClusterResolved
d1e2351
Signed-off-by: kahirokunn <okinakahiro@gmail.com>
@kahirokunn
add doRefresh error path tests for ClusterProvider
c3c500f
Signed-off-by: kahirokunn <okinakahiro@gmail.com>
@kahirokunn
mark anchor ConfigMap as protected to surface deletion risk
6b7c4e0
Signed-off-by: kahirokunn <okinakahiro@gmail.com>
@kahirokunn
stabilise ClusterProvider shutdown and propagate namespace config
4c2c9b4
Signed-off-by: kahirokunn <okinakahiro@gmail.com>
@kahirokunn
add --remote-deployments-poll-interval CLI flag
5093973
Signed-off-by: kahirokunn <okinakahiro@gmail.com>
@kahirokunn
add multi-cluster e2e test for KnativeEventing
b901f31
Signed-off-by: kahirokunn <okinakahiro@gmail.com>
@kahirokunn
./hack/update-codegen.sh
72fc85c
@kahirokunn
kahirokunn
commented
Apr 21, 2026

Copy link


Copy Markdown

MemberAuthor
All CI passed! 🙌
@houshengbo
houshengbo
commented
Apr 22, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
/lgtm

/approve
@knative-prow
knative-prowBot
commented
Apr 22, 2026

Copy link


Copy Markdown

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: houshengbo, kahirokunn
The full list of commands accepted by this bot can be found here.
The pull request process is described here
Details
Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment

Approvers can cancel approval by writing /approve cancel in a comment
@kahirokunn
kahirokunn
commented
Apr 22, 2026

Copy link


Copy Markdown

MemberAuthor
Thank you😆
@kahirokunnkahirokunn
mentioned this pull request
Apr 23, 2026
 Merged

@kahirokunnkahirokunn
mentioned this pull request
May 26, 2026

Closed


This was referenced Aug 22, 2026
 Closed

 Closed


Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@knative-prowknative-prow[bot]knative-prow[bot] left review comments
@matzewmatzewAwaiting requested review from matzew
@houshengbohoushengboAwaiting requested review from houshengbo

Assignees

@houshengbohoushengbo

Labels

approvedIndicates a PR has been approved by an approver from all required OWNERS files.lgtmIndicates that a PR is ready to be merged.size/XXLDenotes a PR that changes 1000+ lines, ignoring generated files.

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

Support multi-cluster deployment via Cluster Inventory API


3 participants

@kahirokunn@houshengbo@knative-prow-robot

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Add multi-cluster deployment support via Cluster Inventory API - #2267
Merged
knative-prow[bot] merged 8 commits into
knative:mainfrom 
kahirokunn:multi-cluster-support
Apr 22, 2026
Merged
Add multi-cluster deployment support via Cluster Inventory API#2267
knative-prow[bot] merged 8 commits into
knative:mainfrom 
kahirokunn:multi-cluster-support
Conversation
@kahirokunn
@kahirokunnkahirokunn
commented
Apr 6, 2026•

edited

Loading

Copy link


Copy Markdown

Member
Fixes#2264
Requires knative/infra#827 to be merged first so that the integration-tests-multicluster presubmit exists when this PR's CI runs.
Proposed Changes
  • Enable deploying Knative Serving and Eventing components to remote clusters by setting spec.clusterProfileRef on the CR, using the SIG-Multicluster Cluster Inventory API (KEP-5339, ClusterProfile) to discover target clusters without depending on a specific fleet manager.
  • When spec.clusterProfileRef is not set, behavior is completely unchanged — existing single-cluster deployments are unaffected.
  • On CR deletion, the operator finalizes resources on the remote cluster. If the remote cluster is unreachable, the finalizer retries; operators can remove it manually to force-delete.
Design
The core idea is to swap the manifestival manifest.Client at the start of the reconcile stage pipeline (ResolveTargetCluster), so that all subsequent stages — Apply, Delete, Get — transparently operate on the remote cluster.
For garbage collection on the remote cluster, an anchor ConfigMap pattern (inspired by k0smotron) is used:
  • An anchor ConfigMap ({kind}-{cr-name}-root-owner) is created on the remote cluster.
  • All namespace-scoped resources get an ownerReference pointing to this anchor, enabling Kubernetes-native GC.
  • Cluster-scoped resources (ClusterRole, etc.) have no ownerReference and are explicitly deleted by the finalizer.
  • Deleting the anchor ConfigMap cascades to all namespace-scoped resources via GC.
Example CR
KnativeServing on a remote cluster
apiVersion: operator.knative.dev/v1beta1kind: KnativeServingmetadata:
name: knative-serving-apacnamespace: knative-servingspec:
version: "1.21"clusterProfileRef:
name: apac-cluster-01namespace: fleet-systemconfig:
network:
ingress-class: "kourier.ingress.networking.knative.dev"ingress:
kourier:
enabled: true
KnativeEventing on a remote cluster
apiVersion: operator.knative.dev/v1beta1kind: KnativeEventingmetadata:
name: knative-eventing-apacnamespace: knative-eventingspec:
version: "1.21"clusterProfileRef:
name: apac-cluster-01namespace: fleet-system
Local cluster (unchanged behavior)
apiVersion: operator.knative.dev/v1beta1kind: KnativeServingmetadata:
name: knative-servingnamespace: knative-servingspec:
version: "1.21"# No clusterProfileRef → deploys to the local cluster as before
Verification Steps
E2E verification with Kind
Note: The credential plugin used below (kubeconfig-secretreader-plugin) is a community plugin.

An official plugin is being tracked at kubernetes-sigs/cluster-inventory-api#45 —

once released, switch to the official one.
1. Cluster setup
# Create Kind clusters
kind create cluster --name hub
kind create cluster --name spoke
# Install ClusterProfile CRD on hub
kubectl --context kind-hub apply -f https://raw.githubusercontent.com/kubernetes-sigs/cluster-inventory-api/main/config/crd/bases/multicluster.x-k8s.io_clusterprofiles.yaml
# Create namespaces on hub
kubectl --context kind-hub create namespace fleet-system
kubectl --context kind-hub create namespace knative-serving
kubectl --context kind-hub create namespace knative-operator
2. Spoke cluster credentials
# Create a ServiceAccount + token Secret on spoke for the operator to use
kubectl --context kind-spoke apply -f - <<EOFapiVersion: v1kind: ServiceAccountmetadata: name: knative-operator namespace: default---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRoleBindingmetadata: name: knative-operator-adminroleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cluster-adminsubjects:- kind: ServiceAccount name: knative-operator namespace: default---apiVersion: v1kind: Secretmetadata: name: knative-operator-token namespace: default annotations: kubernetes.io/service-account.name: knative-operatortype: kubernetes.io/service-account-tokenEOF# Wait for the token to be populatedforiin$(seq 1 30);do
TOKEN=$(kubectl --context kind-spoke get secret knative-operator-token -o go-template='{{.data.token | base64decode}}'2>/dev/null)&& [ -n"$TOKEN" ] &&break
sleep 1
done# Copy the token as a Secret on the hub cluster
kubectl --context kind-hub -n knative-operator create secret generic spoke-token \
--from-literal=token="${TOKEN}" --dry-run=client -o yaml | kubectl --context kind-hub apply -f -
3. ClusterProfile
# Get spoke cluster connection info (Docker-internal IP, not 127.0.0.1)
SPOKE_SERVER="https://$(docker inspect spoke-control-plane --format '{{ .NetworkSettings.Networks.kind.IPAddress }}'):6443"
SPOKE_CA=$(kubectl --context kind-spoke config view --minify --flatten -o jsonpath='{.clusters[0].cluster.certificate-authority-data}')# Create ClusterProfile (spec only — status is a subresource)
kubectl --context kind-hub apply -f - <<EOFapiVersion: multicluster.x-k8s.io/v1alpha1kind: ClusterProfilemetadata: name: spoke-cluster namespace: fleet-systemspec: clusterManager: name: kindEOF# Patch status with access provider info.# The extensions tell the credential plugin which Secret to read the token from.
STATUS_PATCH=$(cat <<EOF{ "status": { "accessProviders": [ { "name": "token-secretreader", "cluster": { "server": "${SPOKE_SERVER}", "certificate-authority-data": "${SPOKE_CA}", "extensions": [ { "name": "client.authentication.k8s.io/exec", "extension": { "secretName": "spoke-token", "secretNamespace": "knative-operator", "key": "token" } } ] } } ] }}EOF)
kubectl --context kind-hub patch clusterprofile spoke-cluster \
-n fleet-system --type merge --subresource=status -p "${STATUS_PATCH}"
4. Operator deployment
# Build and deploy the operator with ko.# KIND_CLUSTER_NAME ensures the image is loaded into the correct Kind cluster.
kubectl config use-context kind-hub
KO_DOCKER_REPO=kind.local KIND_CLUSTER_NAME=hub ko apply -f config/
kubectl --context kind-hub -n knative-operator \
wait --for=condition=Available deployment/knative-operator --timeout=120s
# Create clusterprofile-provider-file ConfigMap (provider name must match accessProviders[].name)
kubectl --context kind-hub -n knative-operator create configmap clusterprofile-provider-file \
--from-literal=config.json='{"providers":[{"name":"token-secretreader","execConfig":{"apiVersion":"client.authentication.k8s.io/v1","command":"/credential-plugin/kubeconfig-secretreader-plugin","provideClusterInfo":true}}]}'# Patch the operator deployment:# - Mount the credential plugin binary via image volume# - Mount the clusterprofile-provider-file config# - Add the --clusterprofile-provider-file flag
kubectl --context kind-hub -n knative-operator patch deployment knative-operator --type json -p '[ {"op":"add","path":"/spec/template/spec/containers/0/args","value":["--clusterprofile-provider-file=/etc/cluster-inventory/config.json"]}, {"op":"add","path":"/spec/template/spec/containers/0/volumeMounts","value":[ {"name":"cred-config","mountPath":"/etc/cluster-inventory","readOnly":true}, {"name":"credential-plugin","mountPath":"/credential-plugin","readOnly":true} ]}, {"op":"add","path":"/spec/template/spec/volumes","value":[ {"name":"cred-config","configMap":{"name":"clusterprofile-provider-file"}}, {"name":"credential-plugin","image":{"reference":"ghcr.io/labthrust/kubeconfig-secretreader-plugin:v0.0.1-linux-arm64"}} ]}]'
kubectl --context kind-hub -n knative-operator rollout status deployment/knative-operator --timeout=120s
5. Install Envoy Gateway on the spoke cluster
The net-gateway-api ingress provider requires Gateway API resources (GatewayClass, Gateway) to exist on the target cluster before deploying KnativeServing. Install Envoy Gateway and create the necessary resources on the spoke cluster.
Prerequisites
go install sigs.k8s.io/cloud-provider-kind@latest
Install Envoy Gateway
Since cloud-provider-kind installs Gateway API CRDs with server-side apply, helm install will fail with field ownership conflicts. Use helm template + kubectl apply --server-side --force-conflicts to take over ownership.
cat <<'EOF' > /tmp/values-eg.yamlconfig: envoyGateway: provider: type: Kubernetes kubernetes: deploy: type: GatewayNamespaceEOFexport ENVOY_GATEWAY_VERSION=v1.7.1
kubectl --context kind-spoke create namespace envoy-gateway-system
helm template eg oci://docker.io/envoyproxy/gateway-helm \
--version $ENVOY_GATEWAY_VERSION \
-n envoy-gateway-system \
-f /tmp/values-eg.yaml \
--include-crds | \
kubectl --context kind-spoke apply --server-side --force-conflicts -f -
After installing Envoy Gateway, start cloud-provider-kind in a separate terminal (provides LoadBalancer support for Kind):
sudo cloud-provider-kind
Create Gateway API resources (external)
kubectl --context kind-spoke apply -f - <<EOFapiVersion: v1kind: Namespacemetadata: name: eg-external---apiVersion: gateway.envoyproxy.io/v1alpha1kind: EnvoyProxymetadata: name: knative-external-config namespace: envoy-gateway-systemspec: provider: type: Kubernetes kubernetes: envoyService: name: knative-external---apiVersion: gateway.networking.k8s.io/v1kind: GatewayClassmetadata: name: eg-externalspec: controllerName: gateway.envoyproxy.io/gatewayclass-controller parametersRef: group: gateway.envoyproxy.io kind: EnvoyProxy name: knative-external-config namespace: envoy-gateway-system---apiVersion: gateway.networking.k8s.io/v1kind: Gatewaymetadata: name: eg-external namespace: eg-externalspec: gatewayClassName: eg-external listeners: - name: http port: 80 protocol: HTTP allowedRoutes: namespaces: from: All - name: tls port: 443 protocol: TLS tls: mode: Passthrough allowedRoutes: namespaces: from: AllEOF
Create Gateway API resources (internal)
The internal Gateway uses the ClusterIP service type, so it is not accessible from outside the cluster.
kubectl --context kind-spoke apply -f - <<EOFapiVersion: v1kind: Namespacemetadata: name: eg-internal---apiVersion: gateway.envoyproxy.io/v1alpha1kind: EnvoyProxymetadata: name: knative-internal-config namespace: envoy-gateway-systemspec: provider: type: Kubernetes kubernetes: envoyService: type: ClusterIP name: knative-internal---apiVersion: gateway.networking.k8s.io/v1kind: GatewayClassmetadata: name: eg-internalspec: controllerName: gateway.envoyproxy.io/gatewayclass-controller parametersRef: group: gateway.envoyproxy.io kind: EnvoyProxy name: knative-internal-config namespace: envoy-gateway-system---apiVersion: gateway.networking.k8s.io/v1kind: Gatewaymetadata: name: eg-internal namespace: eg-internalspec: gatewayClassName: eg-internal listeners: - name: http port: 80 protocol: HTTP allowedRoutes: namespaces: from: AllEOF
6. Deploy KnativeServing
kubectl --context kind-hub apply -f - <<EOFapiVersion: operator.knative.dev/v1beta1kind: KnativeServingmetadata: name: knative-serving namespace: knative-servingspec: clusterProfileRef: name: spoke-cluster namespace: fleet-system ingress: gateway-api: enabled: true config: config-gateway: external-gateways: | - class: eg-external gateway: eg-external/eg-external service: eg-external/knative-external supported-features: - HTTPRouteRequestTimeout local-gateways: | - class: eg-internal gateway: eg-internal/eg-internal service: eg-internal/knative-internal supported-features: - HTTPRouteRequestTimeout network: ingress.class: gateway-api.ingress.networking.knative.dev domain: example.com: ""EOF
7. Verify
# Verify resources deployed to spoke cluster
kubectl --context kind-spoke get deployments -n knative-serving
kubectl --context kind-spoke get configmaps -n knative-serving | grep root-owner
# Verify anchor ConfigMap exists with correct labels
kubectl --context kind-spoke get configmap knativeserving-knative-serving-root-owner \
-n knative-serving -o yaml
# Verify ownerReferences on namespace-scoped resources point to the anchor
kubectl --context kind-spoke get deployment activator -n knative-serving \
-o jsonpath='{.metadata.ownerReferences}'| jq .# Deploy a sample application to verify traffic routing
kubectl --context kind-spoke apply -f - <<EOFapiVersion: serving.knative.dev/v1kind: Servicemetadata: name: helloworld-go namespace: defaultspec: template: spec: containers: - image: gcr.io/knative-samples/helloworld-go env: - name: TARGET value: Go Sample v1EOF# Get the external gateway IP and testexport LB_IP=$(kubectl --context kind-spoke -n eg-external get svc knative-external \ -o jsonpath='{.status.loadBalancer.ingress[0].ip}')echo"$LB_IP helloworld-go.default.example.com"| sudo tee -a /etc/hosts
curl http://helloworld-go.default.example.com

8. Verify CR deletion and finalizer cleanup
# Delete the CR and wait for the finalizer to complete
kubectl --context kind-hub delete knativeserving knative-serving -n knative-serving
# Verify spoke cluster is clean
kubectl --context kind-spoke get all -n knative-serving
kubectl --context kind-spoke get clusterroles | grep knative
9. Cleanup
kind delete cluster --name hub
kind delete cluster --name spoke
Backward compatibility
# Deploy without --clusterprofile-provider-file and without clusterProfileRef# → must behave identically to previous releases
kubectl apply -f - <<EOFapiVersion: operator.knative.dev/v1beta1kind: KnativeServingmetadata: name: knative-serving namespace: knative-servingspec: version: "1.21"EOF
kubectl get knativeserving -n knative-serving -o jsonpath='{.status.conditions}'
Release Note
Add multi-cluster deployment support via the Cluster Inventory API (KEP-5339). When `spec.clusterProfileRef` is set on a KnativeServing or KnativeEventing CR, the operator deploys components to the remote cluster described by the referenced ClusterProfile. Requires the `--clusterprofile-provider-file` flag to be set on the operator deployment.

@knative-prowknative-prowBot
added
the 
size/XXL
Denotes a PR that changes 1000+ lines, ignoring generated files.
label
Apr 6, 2026
knative-prow[bot]
knative-prowBot
reviewed
Apr 6, 2026
View reviewed changes
@knative-prowknative-prowBot
left a comment


Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kahirokunn: 0 warnings.
Details
In response to this:
Fixes #
Proposed Changes
Release Note
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.
@codecov
codecovBot
commented
Apr 6, 2026•

edited

Loading

Copy link


Copy Markdown

Codecov Report
❌ Patch coverage is 60.40000% with 198 lines in your changes missing coverage. Please review.

✅ Project coverage is 63.84%. Comparing base (9c66d7b) to head (72fc85c).
⚠️ Report is 1 commits behind head on main.
Files with missing linesPatch %Lines
pkg/reconciler/common/multicluster.go62.18%121 Missing and 14 partials ⚠️
pkg/reconciler/knativeeventing/controller.go0.00%25 Missing ⚠️
pkg/reconciler/knativeserving/controller.go0.00%25 Missing ⚠️
pkg/reconciler/common/clusterprofile_informer.go86.53%3 Missing and 4 partials ⚠️
pkg/apis/operator/base/common.go0.00%2 Missing ⚠️
pkg/reconciler/common/access_provider_flag.go66.66%2 Missing ⚠️
pkg/reconciler/knativeeventing/eventing_tls.go0.00%2 Missing ⚠️
Additional details and impacted files
@@ Coverage Diff @@## main #2267 +/- ##
==========================================
- Coverage 64.58% 63.84% -0.75% 
==========================================
Files 51 55 +4 Lines 1999 2478 +479 ==========================================
+ Hits 1291 1582 +291 - Misses 606 777 +171 - Partials 102 119 +17 
☔ View full report in Codecov by Sentry.

📢 Have feedback on the report? Share it here.
 🚀 New features to boost your workflow: 
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
@kahirokunnkahirokunn
changed the title
Add multi-cluster deployment support via Cluster Inventory API[WIP] Add multi-cluster deployment support via Cluster Inventory APIApr 6, 2026
@knative-prowknative-prowBot
added
the 
do-not-merge/work-in-progress
Indicates that a PR should not merge because it is a work in progress.
label
Apr 6, 2026
@kahirokunn
kahirokunnforce-pushed
the
multi-cluster-support
branch
5 times, most recently
from
3e585a9 to
141c875CompareApril 7, 2026 06:40
@knative-prow-robotknative-prow-robot
added
the 
needs-rebase
Indicates a PR cannot be merged because it has merge conflicts with HEAD.
label
Apr 7, 2026
@kahirokunn
kahirokunnforce-pushed
the
multi-cluster-support
branch
2 times, most recently
from
d308bf0 to
3e742cbCompareApril 7, 2026 12:53
@kahirokunnkahirokunn
changed the title
[WIP] Add multi-cluster deployment support via Cluster Inventory APIAdd multi-cluster deployment support via Cluster Inventory APIApr 7, 2026
@knative-prowknative-prowBot
removed
the 
do-not-merge/work-in-progress
Indicates that a PR should not merge because it is a work in progress.
label
Apr 7, 2026
@kahirokunn
kahirokunnforce-pushed
the
multi-cluster-support
branch
from
3e742cb to
43919c6CompareApril 7, 2026 14:11
@knative-prow-robotknative-prow-robot
removed
the 
needs-rebase
Indicates a PR cannot be merged because it has merge conflicts with HEAD.
label
Apr 7, 2026
@kahirokunn
kahirokunnforce-pushed
the
multi-cluster-support
branch
from
43919c6 to
28a1e02CompareApril 7, 2026 14:22
@knative-prow-robotknative-prow-robot
added
the 
needs-rebase
Indicates a PR cannot be merged because it has merge conflicts with HEAD.
label
Apr 7, 2026
@kahirokunn
kahirokunnforce-pushed
the
multi-cluster-support
branch
from
28a1e02 to
88410efCompareApril 7, 2026 14:46
@knative-prow
knative-prowBot
commented
Apr 7, 2026

Copy link


Copy Markdown

There are empty aliases in OWNER_ALIASES, cleanup is advised.
@knative-prow-robotknative-prow-robot
removed
the 
needs-rebase
Indicates a PR cannot be merged because it has merge conflicts with HEAD.
label
Apr 7, 2026
@kahirokunnkahirokunn
changed the title
Add multi-cluster deployment support via Cluster Inventory API[WIP] Add multi-cluster deployment support via Cluster Inventory APIApr 7, 2026
@knative-prowknative-prowBot
added
the 
do-not-merge/work-in-progress
Indicates that a PR should not merge because it is a work in progress.
label
Apr 7, 2026
@knative-prow
knative-prowBot
commented
Apr 7, 2026

Copy link


Copy Markdown

There are empty aliases in OWNER_ALIASES, cleanup is advised.
@kahirokunn
kahirokunnforce-pushed
the
multi-cluster-support
branch
3 times, most recently
from
29a9317 to
6217b27CompareApril 8, 2026 04:05
@knative-prow-robotknative-prow-robot
added
the 
needs-rebase
Indicates a PR cannot be merged because it has merge conflicts with HEAD.
label
Apr 16, 2026
@kahirokunn
kahirokunnforce-pushed
the
multi-cluster-support
branch
2 times, most recently
from
9068c06 to
cecc1b8CompareApril 16, 2026 05:54
@knative-prow-robotknative-prow-robot
removed
the 
needs-rebase
Indicates a PR cannot be merged because it has merge conflicts with HEAD.
label
Apr 16, 2026
@knative-prow
knative-prowBot
commented
Apr 16, 2026

Copy link


Copy Markdown

There are empty aliases in OWNER_ALIASES, cleanup is advised.
@kahirokunnkahirokunn
changed the title
[WIP] Add multi-cluster deployment support via Cluster Inventory APIAdd multi-cluster deployment support via Cluster Inventory APIApr 16, 2026
@knative-prowknative-prowBot
removed
the 
do-not-merge/work-in-progress
Indicates that a PR should not merge because it is a work in progress.
label
Apr 16, 2026
@kahirokunn
kahirokunnforce-pushed
the
multi-cluster-support
branch
from
cecc1b8 to
725b1beCompareApril 16, 2026 08:34
@kahirokunn
kahirokunn
commented
Apr 17, 2026

Copy link


Copy Markdown

MemberAuthor
I didn't have the necessary permissions.
integration-tests-multicluster_operator_main will keep failing until the following is merged, so I'd like to ignore it using an override until then #2267
@kahirokunn
kahirokunn
commented
Apr 17, 2026

Copy link


Copy Markdown

MemberAuthor
/retest
houshengbo
houshengbo
reviewed
Apr 17, 2026
View reviewed changes
@houshengbohoushengbo
left a comment
•

edited

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review
The PR description is excellent — the verification steps, design explanation, example CRs, and backward compatibility section are thorough and well-written.
2 blocking items and 7 non-blocking items are noted in inline comments below.
Comment threadpkg/apis/operator/v1beta1/knativeserving_lifecycle.go
Outdated

Comment threadpkg/reconciler/common/clusterprofile_informer.go
Comment threadpkg/reconciler/common/multicluster.go
Comment threadpkg/reconciler/common/multicluster.go
Comment threadpkg/reconciler/common/multicluster.go
Comment threadpkg/reconciler/common/transformers.go
Comment threadpkg/reconciler/common/stages.go
Outdated

Comment threadtest/e2e/knativeserving_spoke_test.go
@kahirokunn
kahirokunn
commented
Apr 21, 2026

Copy link


Copy Markdown

MemberAuthor
@houshengbo Thank you for your feedback! I have addressed all the points you raised.
@kahirokunn
kahirokunn
commented
Apr 21, 2026

Copy link


Copy Markdown

MemberAuthor
I will check it
@kahirokunn
add multi-cluster deployment support via Cluster Inventory API
4845dee
Signed-off-by: kahirokunn <okinakahiro@gmail.com>
@kahirokunn
change MarkTargetClusterNotResolved to mark only TargetClusterResolved
d1e2351
Signed-off-by: kahirokunn <okinakahiro@gmail.com>
@kahirokunn
add doRefresh error path tests for ClusterProvider
c3c500f
Signed-off-by: kahirokunn <okinakahiro@gmail.com>
@kahirokunn
mark anchor ConfigMap as protected to surface deletion risk
6b7c4e0
Signed-off-by: kahirokunn <okinakahiro@gmail.com>
@kahirokunn
stabilise ClusterProvider shutdown and propagate namespace config
4c2c9b4
Signed-off-by: kahirokunn <okinakahiro@gmail.com>
@kahirokunn
add --remote-deployments-poll-interval CLI flag
5093973
Signed-off-by: kahirokunn <okinakahiro@gmail.com>
@kahirokunn
add multi-cluster e2e test for KnativeEventing
b901f31
Signed-off-by: kahirokunn <okinakahiro@gmail.com>
@kahirokunn
./hack/update-codegen.sh
72fc85c
@kahirokunn
kahirokunn
commented
Apr 21, 2026

Copy link


Copy Markdown

MemberAuthor
All CI passed! 🙌
@houshengbo
houshengbo
commented
Apr 22, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
/lgtm

/approve
@knative-prow
knative-prowBot
commented
Apr 22, 2026

Copy link


Copy Markdown

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: houshengbo, kahirokunn
The full list of commands accepted by this bot can be found here.
The pull request process is described here
Details
Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment

Approvers can cancel approval by writing /approve cancel in a comment
@kahirokunn
kahirokunn
commented
Apr 22, 2026

Copy link


Copy Markdown

MemberAuthor
Thank you😆
@kahirokunnkahirokunn
mentioned this pull request
Apr 23, 2026
 Merged

@kahirokunnkahirokunn
mentioned this pull request
May 26, 2026

Closed


This was referenced Aug 22, 2026
 Closed

 Closed


Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@knative-prowknative-prow[bot]knative-prow[bot] left review comments
@matzewmatzewAwaiting requested review from matzew
@houshengbohoushengboAwaiting requested review from houshengbo

Assignees

@houshengbohoushengbo

Labels

approvedIndicates a PR has been approved by an approver from all required OWNERS files.lgtmIndicates that a PR is ready to be merged.size/XXLDenotes a PR that changes 1000+ lines, ignoring generated files.

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

Support multi-cluster deployment via Cluster Inventory API


3 participants

@kahirokunn@houshengbo@knative-prow-robot

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Add multi-cluster deployment support via Cluster Inventory API - #2267
Merged
knative-prow[bot] merged 8 commits into
knative:mainfrom 
kahirokunn:multi-cluster-support
Apr 22, 2026
Merged
Add multi-cluster deployment support via Cluster Inventory API#2267
knative-prow[bot] merged 8 commits into
knative:mainfrom 
kahirokunn:multi-cluster-support
Conversation
@kahirokunn
@kahirokunnkahirokunn
commented
Apr 6, 2026•

edited

Loading

Copy link


Copy Markdown

Member
Fixes#2264
Requires knative/infra#827 to be merged first so that the integration-tests-multicluster presubmit exists when this PR's CI runs.
Proposed Changes
  • Enable deploying Knative Serving and Eventing components to remote clusters by setting spec.clusterProfileRef on the CR, using the SIG-Multicluster Cluster Inventory API (KEP-5339, ClusterProfile) to discover target clusters without depending on a specific fleet manager.
  • When spec.clusterProfileRef is not set, behavior is completely unchanged — existing single-cluster deployments are unaffected.
  • On CR deletion, the operator finalizes resources on the remote cluster. If the remote cluster is unreachable, the finalizer retries; operators can remove it manually to force-delete.
Design
The core idea is to swap the manifestival manifest.Client at the start of the reconcile stage pipeline (ResolveTargetCluster), so that all subsequent stages — Apply, Delete, Get — transparently operate on the remote cluster.
For garbage collection on the remote cluster, an anchor ConfigMap pattern (inspired by k0smotron) is used:
  • An anchor ConfigMap ({kind}-{cr-name}-root-owner) is created on the remote cluster.
  • All namespace-scoped resources get an ownerReference pointing to this anchor, enabling Kubernetes-native GC.
  • Cluster-scoped resources (ClusterRole, etc.) have no ownerReference and are explicitly deleted by the finalizer.
  • Deleting the anchor ConfigMap cascades to all namespace-scoped resources via GC.
Example CR
KnativeServing on a remote cluster
apiVersion: operator.knative.dev/v1beta1kind: KnativeServingmetadata:
name: knative-serving-apacnamespace: knative-servingspec:
version: "1.21"clusterProfileRef:
name: apac-cluster-01namespace: fleet-systemconfig:
network:
ingress-class: "kourier.ingress.networking.knative.dev"ingress:
kourier:
enabled: true
KnativeEventing on a remote cluster
apiVersion: operator.knative.dev/v1beta1kind: KnativeEventingmetadata:
name: knative-eventing-apacnamespace: knative-eventingspec:
version: "1.21"clusterProfileRef:
name: apac-cluster-01namespace: fleet-system
Local cluster (unchanged behavior)
apiVersion: operator.knative.dev/v1beta1kind: KnativeServingmetadata:
name: knative-servingnamespace: knative-servingspec:
version: "1.21"# No clusterProfileRef → deploys to the local cluster as before
Verification Steps
E2E verification with Kind
Note: The credential plugin used below (kubeconfig-secretreader-plugin) is a community plugin.

An official plugin is being tracked at kubernetes-sigs/cluster-inventory-api#45 —

once released, switch to the official one.
1. Cluster setup
# Create Kind clusters
kind create cluster --name hub
kind create cluster --name spoke
# Install ClusterProfile CRD on hub
kubectl --context kind-hub apply -f https://raw.githubusercontent.com/kubernetes-sigs/cluster-inventory-api/main/config/crd/bases/multicluster.x-k8s.io_clusterprofiles.yaml
# Create namespaces on hub
kubectl --context kind-hub create namespace fleet-system
kubectl --context kind-hub create namespace knative-serving
kubectl --context kind-hub create namespace knative-operator
2. Spoke cluster credentials
# Create a ServiceAccount + token Secret on spoke for the operator to use
kubectl --context kind-spoke apply -f - <<EOFapiVersion: v1kind: ServiceAccountmetadata: name: knative-operator namespace: default---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRoleBindingmetadata: name: knative-operator-adminroleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cluster-adminsubjects:- kind: ServiceAccount name: knative-operator namespace: default---apiVersion: v1kind: Secretmetadata: name: knative-operator-token namespace: default annotations: kubernetes.io/service-account.name: knative-operatortype: kubernetes.io/service-account-tokenEOF# Wait for the token to be populatedforiin$(seq 1 30);do
TOKEN=$(kubectl --context kind-spoke get secret knative-operator-token -o go-template='{{.data.token | base64decode}}'2>/dev/null)&& [ -n"$TOKEN" ] &&break
sleep 1
done# Copy the token as a Secret on the hub cluster
kubectl --context kind-hub -n knative-operator create secret generic spoke-token \
--from-literal=token="${TOKEN}" --dry-run=client -o yaml | kubectl --context kind-hub apply -f -
3. ClusterProfile
# Get spoke cluster connection info (Docker-internal IP, not 127.0.0.1)
SPOKE_SERVER="https://$(docker inspect spoke-control-plane --format '{{ .NetworkSettings.Networks.kind.IPAddress }}'):6443"
SPOKE_CA=$(kubectl --context kind-spoke config view --minify --flatten -o jsonpath='{.clusters[0].cluster.certificate-authority-data}')# Create ClusterProfile (spec only — status is a subresource)
kubectl --context kind-hub apply -f - <<EOFapiVersion: multicluster.x-k8s.io/v1alpha1kind: ClusterProfilemetadata: name: spoke-cluster namespace: fleet-systemspec: clusterManager: name: kindEOF# Patch status with access provider info.# The extensions tell the credential plugin which Secret to read the token from.
STATUS_PATCH=$(cat <<EOF{ "status": { "accessProviders": [ { "name": "token-secretreader", "cluster": { "server": "${SPOKE_SERVER}", "certificate-authority-data": "${SPOKE_CA}", "extensions": [ { "name": "client.authentication.k8s.io/exec", "extension": { "secretName": "spoke-token", "secretNamespace": "knative-operator", "key": "token" } } ] } } ] }}EOF)
kubectl --context kind-hub patch clusterprofile spoke-cluster \
-n fleet-system --type merge --subresource=status -p "${STATUS_PATCH}"
4. Operator deployment
# Build and deploy the operator with ko.# KIND_CLUSTER_NAME ensures the image is loaded into the correct Kind cluster.
kubectl config use-context kind-hub
KO_DOCKER_REPO=kind.local KIND_CLUSTER_NAME=hub ko apply -f config/
kubectl --context kind-hub -n knative-operator \
wait --for=condition=Available deployment/knative-operator --timeout=120s
# Create clusterprofile-provider-file ConfigMap (provider name must match accessProviders[].name)
kubectl --context kind-hub -n knative-operator create configmap clusterprofile-provider-file \
--from-literal=config.json='{"providers":[{"name":"token-secretreader","execConfig":{"apiVersion":"client.authentication.k8s.io/v1","command":"/credential-plugin/kubeconfig-secretreader-plugin","provideClusterInfo":true}}]}'# Patch the operator deployment:# - Mount the credential plugin binary via image volume# - Mount the clusterprofile-provider-file config# - Add the --clusterprofile-provider-file flag
kubectl --context kind-hub -n knative-operator patch deployment knative-operator --type json -p '[ {"op":"add","path":"/spec/template/spec/containers/0/args","value":["--clusterprofile-provider-file=/etc/cluster-inventory/config.json"]}, {"op":"add","path":"/spec/template/spec/containers/0/volumeMounts","value":[ {"name":"cred-config","mountPath":"/etc/cluster-inventory","readOnly":true}, {"name":"credential-plugin","mountPath":"/credential-plugin","readOnly":true} ]}, {"op":"add","path":"/spec/template/spec/volumes","value":[ {"name":"cred-config","configMap":{"name":"clusterprofile-provider-file"}}, {"name":"credential-plugin","image":{"reference":"ghcr.io/labthrust/kubeconfig-secretreader-plugin:v0.0.1-linux-arm64"}} ]}]'
kubectl --context kind-hub -n knative-operator rollout status deployment/knative-operator --timeout=120s
5. Install Envoy Gateway on the spoke cluster
The net-gateway-api ingress provider requires Gateway API resources (GatewayClass, Gateway) to exist on the target cluster before deploying KnativeServing. Install Envoy Gateway and create the necessary resources on the spoke cluster.
Prerequisites
go install sigs.k8s.io/cloud-provider-kind@latest
Install Envoy Gateway
Since cloud-provider-kind installs Gateway API CRDs with server-side apply, helm install will fail with field ownership conflicts. Use helm template + kubectl apply --server-side --force-conflicts to take over ownership.
cat <<'EOF' > /tmp/values-eg.yamlconfig: envoyGateway: provider: type: Kubernetes kubernetes: deploy: type: GatewayNamespaceEOFexport ENVOY_GATEWAY_VERSION=v1.7.1
kubectl --context kind-spoke create namespace envoy-gateway-system
helm template eg oci://docker.io/envoyproxy/gateway-helm \
--version $ENVOY_GATEWAY_VERSION \
-n envoy-gateway-system \
-f /tmp/values-eg.yaml \
--include-crds | \
kubectl --context kind-spoke apply --server-side --force-conflicts -f -
After installing Envoy Gateway, start cloud-provider-kind in a separate terminal (provides LoadBalancer support for Kind):
sudo cloud-provider-kind
Create Gateway API resources (external)
kubectl --context kind-spoke apply -f - <<EOFapiVersion: v1kind: Namespacemetadata: name: eg-external---apiVersion: gateway.envoyproxy.io/v1alpha1kind: EnvoyProxymetadata: name: knative-external-config namespace: envoy-gateway-systemspec: provider: type: Kubernetes kubernetes: envoyService: name: knative-external---apiVersion: gateway.networking.k8s.io/v1kind: GatewayClassmetadata: name: eg-externalspec: controllerName: gateway.envoyproxy.io/gatewayclass-controller parametersRef: group: gateway.envoyproxy.io kind: EnvoyProxy name: knative-external-config namespace: envoy-gateway-system---apiVersion: gateway.networking.k8s.io/v1kind: Gatewaymetadata: name: eg-external namespace: eg-externalspec: gatewayClassName: eg-external listeners: - name: http port: 80 protocol: HTTP allowedRoutes: namespaces: from: All - name: tls port: 443 protocol: TLS tls: mode: Passthrough allowedRoutes: namespaces: from: AllEOF
Create Gateway API resources (internal)
The internal Gateway uses the ClusterIP service type, so it is not accessible from outside the cluster.
kubectl --context kind-spoke apply -f - <<EOFapiVersion: v1kind: Namespacemetadata: name: eg-internal---apiVersion: gateway.envoyproxy.io/v1alpha1kind: EnvoyProxymetadata: name: knative-internal-config namespace: envoy-gateway-systemspec: provider: type: Kubernetes kubernetes: envoyService: type: ClusterIP name: knative-internal---apiVersion: gateway.networking.k8s.io/v1kind: GatewayClassmetadata: name: eg-internalspec: controllerName: gateway.envoyproxy.io/gatewayclass-controller parametersRef: group: gateway.envoyproxy.io kind: EnvoyProxy name: knative-internal-config namespace: envoy-gateway-system---apiVersion: gateway.networking.k8s.io/v1kind: Gatewaymetadata: name: eg-internal namespace: eg-internalspec: gatewayClassName: eg-internal listeners: - name: http port: 80 protocol: HTTP allowedRoutes: namespaces: from: AllEOF
6. Deploy KnativeServing
kubectl --context kind-hub apply -f - <<EOFapiVersion: operator.knative.dev/v1beta1kind: KnativeServingmetadata: name: knative-serving namespace: knative-servingspec: clusterProfileRef: name: spoke-cluster namespace: fleet-system ingress: gateway-api: enabled: true config: config-gateway: external-gateways: | - class: eg-external gateway: eg-external/eg-external service: eg-external/knative-external supported-features: - HTTPRouteRequestTimeout local-gateways: | - class: eg-internal gateway: eg-internal/eg-internal service: eg-internal/knative-internal supported-features: - HTTPRouteRequestTimeout network: ingress.class: gateway-api.ingress.networking.knative.dev domain: example.com: ""EOF
7. Verify
# Verify resources deployed to spoke cluster
kubectl --context kind-spoke get deployments -n knative-serving
kubectl --context kind-spoke get configmaps -n knative-serving | grep root-owner
# Verify anchor ConfigMap exists with correct labels
kubectl --context kind-spoke get configmap knativeserving-knative-serving-root-owner \
-n knative-serving -o yaml
# Verify ownerReferences on namespace-scoped resources point to the anchor
kubectl --context kind-spoke get deployment activator -n knative-serving \
-o jsonpath='{.metadata.ownerReferences}'| jq .# Deploy a sample application to verify traffic routing
kubectl --context kind-spoke apply -f - <<EOFapiVersion: serving.knative.dev/v1kind: Servicemetadata: name: helloworld-go namespace: defaultspec: template: spec: containers: - image: gcr.io/knative-samples/helloworld-go env: - name: TARGET value: Go Sample v1EOF# Get the external gateway IP and testexport LB_IP=$(kubectl --context kind-spoke -n eg-external get svc knative-external \ -o jsonpath='{.status.loadBalancer.ingress[0].ip}')echo"$LB_IP helloworld-go.default.example.com"| sudo tee -a /etc/hosts
curl http://helloworld-go.default.example.com

8. Verify CR deletion and finalizer cleanup
# Delete the CR and wait for the finalizer to complete
kubectl --context kind-hub delete knativeserving knative-serving -n knative-serving
# Verify spoke cluster is clean
kubectl --context kind-spoke get all -n knative-serving
kubectl --context kind-spoke get clusterroles | grep knative
9. Cleanup
kind delete cluster --name hub
kind delete cluster --name spoke
Backward compatibility
# Deploy without --clusterprofile-provider-file and without clusterProfileRef# → must behave identically to previous releases
kubectl apply -f - <<EOFapiVersion: operator.knative.dev/v1beta1kind: KnativeServingmetadata: name: knative-serving namespace: knative-servingspec: version: "1.21"EOF
kubectl get knativeserving -n knative-serving -o jsonpath='{.status.conditions}'
Release Note
Add multi-cluster deployment support via the Cluster Inventory API (KEP-5339). When `spec.clusterProfileRef` is set on a KnativeServing or KnativeEventing CR, the operator deploys components to the remote cluster described by the referenced ClusterProfile. Requires the `--clusterprofile-provider-file` flag to be set on the operator deployment.

@knative-prowknative-prowBot
added
the 
size/XXL
Denotes a PR that changes 1000+ lines, ignoring generated files.
label
Apr 6, 2026
knative-prow[bot]
knative-prowBot
reviewed
Apr 6, 2026
View reviewed changes
@knative-prowknative-prowBot
left a comment


Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kahirokunn: 0 warnings.
Details
In response to this:
Fixes #
Proposed Changes
Release Note
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.
@codecov
codecovBot
commented
Apr 6, 2026•

edited

Loading

Copy link


Copy Markdown

Codecov Report
❌ Patch coverage is 60.40000% with 198 lines in your changes missing coverage. Please review.

✅ Project coverage is 63.84%. Comparing base (9c66d7b) to head (72fc85c).
⚠️ Report is 1 commits behind head on main.
Files with missing linesPatch %Lines
pkg/reconciler/common/multicluster.go62.18%121 Missing and 14 partials ⚠️
pkg/reconciler/knativeeventing/controller.go0.00%25 Missing ⚠️
pkg/reconciler/knativeserving/controller.go0.00%25 Missing ⚠️
pkg/reconciler/common/clusterprofile_informer.go86.53%3 Missing and 4 partials ⚠️
pkg/apis/operator/base/common.go0.00%2 Missing ⚠️
pkg/reconciler/common/access_provider_flag.go66.66%2 Missing ⚠️
pkg/reconciler/knativeeventing/eventing_tls.go0.00%2 Missing ⚠️
Additional details and impacted files
@@ Coverage Diff @@## main #2267 +/- ##
==========================================
- Coverage 64.58% 63.84% -0.75% 
==========================================
Files 51 55 +4 Lines 1999 2478 +479 ==========================================
+ Hits 1291 1582 +291 - Misses 606 777 +171 - Partials 102 119 +17 
☔ View full report in Codecov by Sentry.

📢 Have feedback on the report? Share it here.
 🚀 New features to boost your workflow: 
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
@kahirokunnkahirokunn
changed the title
Add multi-cluster deployment support via Cluster Inventory API[WIP] Add multi-cluster deployment support via Cluster Inventory APIApr 6, 2026
@knative-prowknative-prowBot
added
the 
do-not-merge/work-in-progress
Indicates that a PR should not merge because it is a work in progress.
label
Apr 6, 2026
@kahirokunn
kahirokunnforce-pushed
the
multi-cluster-support
branch
5 times, most recently
from
3e585a9 to
141c875CompareApril 7, 2026 06:40
@knative-prow-robotknative-prow-robot
added
the 
needs-rebase
Indicates a PR cannot be merged because it has merge conflicts with HEAD.
label
Apr 7, 2026
@kahirokunn
kahirokunnforce-pushed
the
multi-cluster-support
branch
2 times, most recently
from
d308bf0 to
3e742cbCompareApril 7, 2026 12:53
@kahirokunnkahirokunn
changed the title
[WIP] Add multi-cluster deployment support via Cluster Inventory APIAdd multi-cluster deployment support via Cluster Inventory APIApr 7, 2026
@knative-prowknative-prowBot
removed
the 
do-not-merge/work-in-progress
Indicates that a PR should not merge because it is a work in progress.
label
Apr 7, 2026
@kahirokunn
kahirokunnforce-pushed
the
multi-cluster-support
branch
from
3e742cb to
43919c6CompareApril 7, 2026 14:11
@knative-prow-robotknative-prow-robot
removed
the 
needs-rebase
Indicates a PR cannot be merged because it has merge conflicts with HEAD.
label
Apr 7, 2026
@kahirokunn
kahirokunnforce-pushed
the
multi-cluster-support
branch
from
43919c6 to
28a1e02CompareApril 7, 2026 14:22
@knative-prow-robotknative-prow-robot
added
the 
needs-rebase
Indicates a PR cannot be merged because it has merge conflicts with HEAD.
label
Apr 7, 2026
@kahirokunn
kahirokunnforce-pushed
the
multi-cluster-support
branch
from
28a1e02 to
88410efCompareApril 7, 2026 14:46
@knative-prow
knative-prowBot
commented
Apr 7, 2026

Copy link


Copy Markdown

There are empty aliases in OWNER_ALIASES, cleanup is advised.
@knative-prow-robotknative-prow-robot
removed
the 
needs-rebase
Indicates a PR cannot be merged because it has merge conflicts with HEAD.
label
Apr 7, 2026
@kahirokunnkahirokunn
changed the title
Add multi-cluster deployment support via Cluster Inventory API[WIP] Add multi-cluster deployment support via Cluster Inventory APIApr 7, 2026
@knative-prowknative-prowBot
added
the 
do-not-merge/work-in-progress
Indicates that a PR should not merge because it is a work in progress.
label
Apr 7, 2026
@knative-prow
knative-prowBot
commented
Apr 7, 2026

Copy link


Copy Markdown

There are empty aliases in OWNER_ALIASES, cleanup is advised.
@kahirokunn
kahirokunnforce-pushed
the
multi-cluster-support
branch
3 times, most recently
from
29a9317 to
6217b27CompareApril 8, 2026 04:05
@knative-prow-robotknative-prow-robot
added
the 
needs-rebase
Indicates a PR cannot be merged because it has merge conflicts with HEAD.
label
Apr 16, 2026
@kahirokunn
kahirokunnforce-pushed
the
multi-cluster-support
branch
2 times, most recently
from
9068c06 to
cecc1b8CompareApril 16, 2026 05:54
@knative-prow-robotknative-prow-robot
removed
the 
needs-rebase
Indicates a PR cannot be merged because it has merge conflicts with HEAD.
label
Apr 16, 2026
@knative-prow
knative-prowBot
commented
Apr 16, 2026

Copy link


Copy Markdown

There are empty aliases in OWNER_ALIASES, cleanup is advised.
@kahirokunnkahirokunn
changed the title
[WIP] Add multi-cluster deployment support via Cluster Inventory APIAdd multi-cluster deployment support via Cluster Inventory APIApr 16, 2026
@knative-prowknative-prowBot
removed
the 
do-not-merge/work-in-progress
Indicates that a PR should not merge because it is a work in progress.
label
Apr 16, 2026
@kahirokunn
kahirokunnforce-pushed
the
multi-cluster-support
branch
from
cecc1b8 to
725b1beCompareApril 16, 2026 08:34
@kahirokunn
kahirokunn
commented
Apr 17, 2026

Copy link


Copy Markdown

MemberAuthor
I didn't have the necessary permissions.
integration-tests-multicluster_operator_main will keep failing until the following is merged, so I'd like to ignore it using an override until then #2267
@kahirokunn
kahirokunn
commented
Apr 17, 2026

Copy link


Copy Markdown

MemberAuthor
/retest
houshengbo
houshengbo
reviewed
Apr 17, 2026
View reviewed changes
@houshengbohoushengbo
left a comment
•

edited

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review
The PR description is excellent — the verification steps, design explanation, example CRs, and backward compatibility section are thorough and well-written.
2 blocking items and 7 non-blocking items are noted in inline comments below.
Comment threadpkg/apis/operator/v1beta1/knativeserving_lifecycle.go
Outdated

Comment threadpkg/reconciler/common/clusterprofile_informer.go
Comment threadpkg/reconciler/common/multicluster.go
Comment threadpkg/reconciler/common/multicluster.go
Comment threadpkg/reconciler/common/multicluster.go
Comment threadpkg/reconciler/common/transformers.go
Comment threadpkg/reconciler/common/stages.go
Outdated

Comment threadtest/e2e/knativeserving_spoke_test.go
@kahirokunn
kahirokunn
commented
Apr 21, 2026

Copy link


Copy Markdown

MemberAuthor
@houshengbo Thank you for your feedback! I have addressed all the points you raised.
@kahirokunn
kahirokunn
commented
Apr 21, 2026

Copy link


Copy Markdown

MemberAuthor
I will check it
@kahirokunn
add multi-cluster deployment support via Cluster Inventory API
4845dee
Signed-off-by: kahirokunn <okinakahiro@gmail.com>
@kahirokunn
change MarkTargetClusterNotResolved to mark only TargetClusterResolved
d1e2351
Signed-off-by: kahirokunn <okinakahiro@gmail.com>
@kahirokunn
add doRefresh error path tests for ClusterProvider
c3c500f
Signed-off-by: kahirokunn <okinakahiro@gmail.com>
@kahirokunn
mark anchor ConfigMap as protected to surface deletion risk
6b7c4e0
Signed-off-by: kahirokunn <okinakahiro@gmail.com>
@kahirokunn
stabilise ClusterProvider shutdown and propagate namespace config
4c2c9b4
Signed-off-by: kahirokunn <okinakahiro@gmail.com>
@kahirokunn
add --remote-deployments-poll-interval CLI flag
5093973
Signed-off-by: kahirokunn <okinakahiro@gmail.com>
@kahirokunn
add multi-cluster e2e test for KnativeEventing
b901f31
Signed-off-by: kahirokunn <okinakahiro@gmail.com>
@kahirokunn
./hack/update-codegen.sh
72fc85c
@kahirokunn
kahirokunn
commented
Apr 21, 2026

Copy link


Copy Markdown

MemberAuthor
All CI passed! 🙌
@houshengbo
houshengbo
commented
Apr 22, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
/lgtm

/approve
@knative-prow
knative-prowBot
commented
Apr 22, 2026

Copy link


Copy Markdown

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: houshengbo, kahirokunn
The full list of commands accepted by this bot can be found here.
The pull request process is described here
Details
Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment

Approvers can cancel approval by writing /approve cancel in a comment
@kahirokunn
kahirokunn
commented
Apr 22, 2026

Copy link


Copy Markdown

MemberAuthor
Thank you😆
@kahirokunnkahirokunn
mentioned this pull request
Apr 23, 2026
 Merged

@kahirokunnkahirokunn
mentioned this pull request
May 26, 2026

Closed


This was referenced Aug 22, 2026
 Closed

 Closed


Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@knative-prowknative-prow[bot]knative-prow[bot] left review comments
@matzewmatzewAwaiting requested review from matzew
@houshengbohoushengboAwaiting requested review from houshengbo

Assignees

@houshengbohoushengbo

Labels

approvedIndicates a PR has been approved by an approver from all required OWNERS files.lgtmIndicates that a PR is ready to be merged.size/XXLDenotes a PR that changes 1000+ lines, ignoring generated files.

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

Support multi-cluster deployment via Cluster Inventory API


3 participants

@kahirokunn@houshengbo@knative-prow-robot

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
Add multi-cluster deployment support via Cluster Inventory API - #2267
Merged
knative-prow[bot] merged 8 commits into
knative:mainfrom 
kahirokunn:multi-cluster-support
Apr 22, 2026
Merged
Add multi-cluster deployment support via Cluster Inventory API#2267
knative-prow[bot] merged 8 commits into
knative:mainfrom 
kahirokunn:multi-cluster-support
Conversation
@kahirokunn
@kahirokunnkahirokunn
commented
Apr 6, 2026•

edited

Loading

Copy link


Copy Markdown

Member
Fixes#2264
Requires knative/infra#827 to be merged first so that the integration-tests-multicluster presubmit exists when this PR's CI runs.
Proposed Changes
  • Enable deploying Knative Serving and Eventing components to remote clusters by setting spec.clusterProfileRef on the CR, using the SIG-Multicluster Cluster Inventory API (KEP-5339, ClusterProfile) to discover target clusters without depending on a specific fleet manager.
  • When spec.clusterProfileRef is not set, behavior is completely unchanged — existing single-cluster deployments are unaffected.
  • On CR deletion, the operator finalizes resources on the remote cluster. If the remote cluster is unreachable, the finalizer retries; operators can remove it manually to force-delete.
Design
The core idea is to swap the manifestival manifest.Client at the start of the reconcile stage pipeline (ResolveTargetCluster), so that all subsequent stages — Apply, Delete, Get — transparently operate on the remote cluster.
For garbage collection on the remote cluster, an anchor ConfigMap pattern (inspired by k0smotron) is used:
  • An anchor ConfigMap ({kind}-{cr-name}-root-owner) is created on the remote cluster.
  • All namespace-scoped resources get an ownerReference pointing to this anchor, enabling Kubernetes-native GC.
  • Cluster-scoped resources (ClusterRole, etc.) have no ownerReference and are explicitly deleted by the finalizer.
  • Deleting the anchor ConfigMap cascades to all namespace-scoped resources via GC.
Example CR
KnativeServing on a remote cluster
apiVersion: operator.knative.dev/v1beta1kind: KnativeServingmetadata:
name: knative-serving-apacnamespace: knative-servingspec:
version: "1.21"clusterProfileRef:
name: apac-cluster-01namespace: fleet-systemconfig:
network:
ingress-class: "kourier.ingress.networking.knative.dev"ingress:
kourier:
enabled: true
KnativeEventing on a remote cluster
apiVersion: operator.knative.dev/v1beta1kind: KnativeEventingmetadata:
name: knative-eventing-apacnamespace: knative-eventingspec:
version: "1.21"clusterProfileRef:
name: apac-cluster-01namespace: fleet-system
Local cluster (unchanged behavior)
apiVersion: operator.knative.dev/v1beta1kind: KnativeServingmetadata:
name: knative-servingnamespace: knative-servingspec:
version: "1.21"# No clusterProfileRef → deploys to the local cluster as before
Verification Steps
E2E verification with Kind
Note: The credential plugin used below (kubeconfig-secretreader-plugin) is a community plugin.

An official plugin is being tracked at kubernetes-sigs/cluster-inventory-api#45 —

once released, switch to the official one.
1. Cluster setup
# Create Kind clusters
kind create cluster --name hub
kind create cluster --name spoke
# Install ClusterProfile CRD on hub
kubectl --context kind-hub apply -f https://raw.githubusercontent.com/kubernetes-sigs/cluster-inventory-api/main/config/crd/bases/multicluster.x-k8s.io_clusterprofiles.yaml
# Create namespaces on hub
kubectl --context kind-hub create namespace fleet-system
kubectl --context kind-hub create namespace knative-serving
kubectl --context kind-hub create namespace knative-operator
2. Spoke cluster credentials
# Create a ServiceAccount + token Secret on spoke for the operator to use
kubectl --context kind-spoke apply -f - <<EOFapiVersion: v1kind: ServiceAccountmetadata: name: knative-operator namespace: default---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRoleBindingmetadata: name: knative-operator-adminroleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cluster-adminsubjects:- kind: ServiceAccount name: knative-operator namespace: default---apiVersion: v1kind: Secretmetadata: name: knative-operator-token namespace: default annotations: kubernetes.io/service-account.name: knative-operatortype: kubernetes.io/service-account-tokenEOF# Wait for the token to be populatedforiin$(seq 1 30);do
TOKEN=$(kubectl --context kind-spoke get secret knative-operator-token -o go-template='{{.data.token | base64decode}}'2>/dev/null)&& [ -n"$TOKEN" ] &&break
sleep 1
done# Copy the token as a Secret on the hub cluster
kubectl --context kind-hub -n knative-operator create secret generic spoke-token \
--from-literal=token="${TOKEN}" --dry-run=client -o yaml | kubectl --context kind-hub apply -f -
3. ClusterProfile
# Get spoke cluster connection info (Docker-internal IP, not 127.0.0.1)
SPOKE_SERVER="https://$(docker inspect spoke-control-plane --format '{{ .NetworkSettings.Networks.kind.IPAddress }}'):6443"
SPOKE_CA=$(kubectl --context kind-spoke config view --minify --flatten -o jsonpath='{.clusters[0].cluster.certificate-authority-data}')# Create ClusterProfile (spec only — status is a subresource)
kubectl --context kind-hub apply -f - <<EOFapiVersion: multicluster.x-k8s.io/v1alpha1kind: ClusterProfilemetadata: name: spoke-cluster namespace: fleet-systemspec: clusterManager: name: kindEOF# Patch status with access provider info.# The extensions tell the credential plugin which Secret to read the token from.
STATUS_PATCH=$(cat <<EOF{ "status": { "accessProviders": [ { "name": "token-secretreader", "cluster": { "server": "${SPOKE_SERVER}", "certificate-authority-data": "${SPOKE_CA}", "extensions": [ { "name": "client.authentication.k8s.io/exec", "extension": { "secretName": "spoke-token", "secretNamespace": "knative-operator", "key": "token" } } ] } } ] }}EOF)
kubectl --context kind-hub patch clusterprofile spoke-cluster \
-n fleet-system --type merge --subresource=status -p "${STATUS_PATCH}"
4. Operator deployment
# Build and deploy the operator with ko.# KIND_CLUSTER_NAME ensures the image is loaded into the correct Kind cluster.
kubectl config use-context kind-hub
KO_DOCKER_REPO=kind.local KIND_CLUSTER_NAME=hub ko apply -f config/
kubectl --context kind-hub -n knative-operator \
wait --for=condition=Available deployment/knative-operator --timeout=120s
# Create clusterprofile-provider-file ConfigMap (provider name must match accessProviders[].name)
kubectl --context kind-hub -n knative-operator create configmap clusterprofile-provider-file \
--from-literal=config.json='{"providers":[{"name":"token-secretreader","execConfig":{"apiVersion":"client.authentication.k8s.io/v1","command":"/credential-plugin/kubeconfig-secretreader-plugin","provideClusterInfo":true}}]}'# Patch the operator deployment:# - Mount the credential plugin binary via image volume# - Mount the clusterprofile-provider-file config# - Add the --clusterprofile-provider-file flag
kubectl --context kind-hub -n knative-operator patch deployment knative-operator --type json -p '[ {"op":"add","path":"/spec/template/spec/containers/0/args","value":["--clusterprofile-provider-file=/etc/cluster-inventory/config.json"]}, {"op":"add","path":"/spec/template/spec/containers/0/volumeMounts","value":[ {"name":"cred-config","mountPath":"/etc/cluster-inventory","readOnly":true}, {"name":"credential-plugin","mountPath":"/credential-plugin","readOnly":true} ]}, {"op":"add","path":"/spec/template/spec/volumes","value":[ {"name":"cred-config","configMap":{"name":"clusterprofile-provider-file"}}, {"name":"credential-plugin","image":{"reference":"ghcr.io/labthrust/kubeconfig-secretreader-plugin:v0.0.1-linux-arm64"}} ]}]'
kubectl --context kind-hub -n knative-operator rollout status deployment/knative-operator --timeout=120s
5. Install Envoy Gateway on the spoke cluster
The net-gateway-api ingress provider requires Gateway API resources (GatewayClass, Gateway) to exist on the target cluster before deploying KnativeServing. Install Envoy Gateway and create the necessary resources on the spoke cluster.
Prerequisites
go install sigs.k8s.io/cloud-provider-kind@latest
Install Envoy Gateway
Since cloud-provider-kind installs Gateway API CRDs with server-side apply, helm install will fail with field ownership conflicts. Use helm template + kubectl apply --server-side --force-conflicts to take over ownership.
cat <<'EOF' > /tmp/values-eg.yamlconfig: envoyGateway: provider: type: Kubernetes kubernetes: deploy: type: GatewayNamespaceEOFexport ENVOY_GATEWAY_VERSION=v1.7.1
kubectl --context kind-spoke create namespace envoy-gateway-system
helm template eg oci://docker.io/envoyproxy/gateway-helm \
--version $ENVOY_GATEWAY_VERSION \
-n envoy-gateway-system \
-f /tmp/values-eg.yaml \
--include-crds | \
kubectl --context kind-spoke apply --server-side --force-conflicts -f -
After installing Envoy Gateway, start cloud-provider-kind in a separate terminal (provides LoadBalancer support for Kind):
sudo cloud-provider-kind
Create Gateway API resources (external)
kubectl --context kind-spoke apply -f - <<EOFapiVersion: v1kind: Namespacemetadata: name: eg-external---apiVersion: gateway.envoyproxy.io/v1alpha1kind: EnvoyProxymetadata: name: knative-external-config namespace: envoy-gateway-systemspec: provider: type: Kubernetes kubernetes: envoyService: name: knative-external---apiVersion: gateway.networking.k8s.io/v1kind: GatewayClassmetadata: name: eg-externalspec: controllerName: gateway.envoyproxy.io/gatewayclass-controller parametersRef: group: gateway.envoyproxy.io kind: EnvoyProxy name: knative-external-config namespace: envoy-gateway-system---apiVersion: gateway.networking.k8s.io/v1kind: Gatewaymetadata: name: eg-external namespace: eg-externalspec: gatewayClassName: eg-external listeners: - name: http port: 80 protocol: HTTP allowedRoutes: namespaces: from: All - name: tls port: 443 protocol: TLS tls: mode: Passthrough allowedRoutes: namespaces: from: AllEOF
Create Gateway API resources (internal)
The internal Gateway uses the ClusterIP service type, so it is not accessible from outside the cluster.
kubectl --context kind-spoke apply -f - <<EOFapiVersion: v1kind: Namespacemetadata: name: eg-internal---apiVersion: gateway.envoyproxy.io/v1alpha1kind: EnvoyProxymetadata: name: knative-internal-config namespace: envoy-gateway-systemspec: provider: type: Kubernetes kubernetes: envoyService: type: ClusterIP name: knative-internal---apiVersion: gateway.networking.k8s.io/v1kind: GatewayClassmetadata: name: eg-internalspec: controllerName: gateway.envoyproxy.io/gatewayclass-controller parametersRef: group: gateway.envoyproxy.io kind: EnvoyProxy name: knative-internal-config namespace: envoy-gateway-system---apiVersion: gateway.networking.k8s.io/v1kind: Gatewaymetadata: name: eg-internal namespace: eg-internalspec: gatewayClassName: eg-internal listeners: - name: http port: 80 protocol: HTTP allowedRoutes: namespaces: from: AllEOF
6. Deploy KnativeServing
kubectl --context kind-hub apply -f - <<EOFapiVersion: operator.knative.dev/v1beta1kind: KnativeServingmetadata: name: knative-serving namespace: knative-servingspec: clusterProfileRef: name: spoke-cluster namespace: fleet-system ingress: gateway-api: enabled: true config: config-gateway: external-gateways: | - class: eg-external gateway: eg-external/eg-external service: eg-external/knative-external supported-features: - HTTPRouteRequestTimeout local-gateways: | - class: eg-internal gateway: eg-internal/eg-internal service: eg-internal/knative-internal supported-features: - HTTPRouteRequestTimeout network: ingress.class: gateway-api.ingress.networking.knative.dev domain: example.com: ""EOF
7. Verify
# Verify resources deployed to spoke cluster
kubectl --context kind-spoke get deployments -n knative-serving
kubectl --context kind-spoke get configmaps -n knative-serving | grep root-owner
# Verify anchor ConfigMap exists with correct labels
kubectl --context kind-spoke get configmap knativeserving-knative-serving-root-owner \
-n knative-serving -o yaml
# Verify ownerReferences on namespace-scoped resources point to the anchor
kubectl --context kind-spoke get deployment activator -n knative-serving \
-o jsonpath='{.metadata.ownerReferences}'| jq .# Deploy a sample application to verify traffic routing
kubectl --context kind-spoke apply -f - <<EOFapiVersion: serving.knative.dev/v1kind: Servicemetadata: name: helloworld-go namespace: defaultspec: template: spec: containers: - image: gcr.io/knative-samples/helloworld-go env: - name: TARGET value: Go Sample v1EOF# Get the external gateway IP and testexport LB_IP=$(kubectl --context kind-spoke -n eg-external get svc knative-external \ -o jsonpath='{.status.loadBalancer.ingress[0].ip}')echo"$LB_IP helloworld-go.default.example.com"| sudo tee -a /etc/hosts
curl http://helloworld-go.default.example.com

8. Verify CR deletion and finalizer cleanup
# Delete the CR and wait for the finalizer to complete
kubectl --context kind-hub delete knativeserving knative-serving -n knative-serving
# Verify spoke cluster is clean
kubectl --context kind-spoke get all -n knative-serving
kubectl --context kind-spoke get clusterroles | grep knative
9. Cleanup
kind delete cluster --name hub
kind delete cluster --name spoke
Backward compatibility
# Deploy without --clusterprofile-provider-file and without clusterProfileRef# → must behave identically to previous releases
kubectl apply -f - <<EOFapiVersion: operator.knative.dev/v1beta1kind: KnativeServingmetadata: name: knative-serving namespace: knative-servingspec: version: "1.21"EOF
kubectl get knativeserving -n knative-serving -o jsonpath='{.status.conditions}'
Release Note
Add multi-cluster deployment support via the Cluster Inventory API (KEP-5339). When `spec.clusterProfileRef` is set on a KnativeServing or KnativeEventing CR, the operator deploys components to the remote cluster described by the referenced ClusterProfile. Requires the `--clusterprofile-provider-file` flag to be set on the operator deployment.

@knative-prowknative-prowBot
added
the 
size/XXL
Denotes a PR that changes 1000+ lines, ignoring generated files.
label
Apr 6, 2026
knative-prow[bot]
knative-prowBot
reviewed
Apr 6, 2026
View reviewed changes
@knative-prowknative-prowBot
left a comment


Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kahirokunn: 0 warnings.
Details
In response to this:
Fixes #
Proposed Changes
Release Note
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.
@codecov
codecovBot
commented
Apr 6, 2026•

edited

Loading

Copy link


Copy Markdown

Codecov Report
❌ Patch coverage is 60.40000% with 198 lines in your changes missing coverage. Please review.

✅ Project coverage is 63.84%. Comparing base (9c66d7b) to head (72fc85c).
⚠️ Report is 1 commits behind head on main.
Files with missing linesPatch %Lines
pkg/reconciler/common/multicluster.go62.18%121 Missing and 14 partials ⚠️
pkg/reconciler/knativeeventing/controller.go0.00%25 Missing ⚠️
pkg/reconciler/knativeserving/controller.go0.00%25 Missing ⚠️
pkg/reconciler/common/clusterprofile_informer.go86.53%3 Missing and 4 partials ⚠️
pkg/apis/operator/base/common.go0.00%2 Missing ⚠️
pkg/reconciler/common/access_provider_flag.go66.66%2 Missing ⚠️
pkg/reconciler/knativeeventing/eventing_tls.go0.00%2 Missing ⚠️
Additional details and impacted files
@@ Coverage Diff @@## main #2267 +/- ##
==========================================
- Coverage 64.58% 63.84% -0.75% 
==========================================
Files 51 55 +4 Lines 1999 2478 +479 ==========================================
+ Hits 1291 1582 +291 - Misses 606 777 +171 - Partials 102 119 +17 
☔ View full report in Codecov by Sentry.

📢 Have feedback on the report? Share it here.
 🚀 New features to boost your workflow: 
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
@kahirokunnkahirokunn
changed the title
Add multi-cluster deployment support via Cluster Inventory API[WIP] Add multi-cluster deployment support via Cluster Inventory APIApr 6, 2026
@knative-prowknative-prowBot
added
the 
do-not-merge/work-in-progress
Indicates that a PR should not merge because it is a work in progress.
label
Apr 6, 2026
@kahirokunn
kahirokunnforce-pushed
the
multi-cluster-support
branch
5 times, most recently
from
3e585a9 to
141c875CompareApril 7, 2026 06:40
@knative-prow-robotknative-prow-robot
added
the 
needs-rebase
Indicates a PR cannot be merged because it has merge conflicts with HEAD.
label
Apr 7, 2026
@kahirokunn
kahirokunnforce-pushed
the
multi-cluster-support
branch
2 times, most recently
from
d308bf0 to
3e742cbCompareApril 7, 2026 12:53
@kahirokunnkahirokunn
changed the title
[WIP] Add multi-cluster deployment support via Cluster Inventory APIAdd multi-cluster deployment support via Cluster Inventory APIApr 7, 2026
@knative-prowknative-prowBot
removed
the 
do-not-merge/work-in-progress
Indicates that a PR should not merge because it is a work in progress.
label
Apr 7, 2026
@kahirokunn
kahirokunnforce-pushed
the
multi-cluster-support
branch
from
3e742cb to
43919c6CompareApril 7, 2026 14:11
@knative-prow-robotknative-prow-robot
removed
the 
needs-rebase
Indicates a PR cannot be merged because it has merge conflicts with HEAD.
label
Apr 7, 2026
@kahirokunn
kahirokunnforce-pushed
the
multi-cluster-support
branch
from
43919c6 to
28a1e02CompareApril 7, 2026 14:22
@knative-prow-robotknative-prow-robot
added
the 
needs-rebase
Indicates a PR cannot be merged because it has merge conflicts with HEAD.
label
Apr 7, 2026
@kahirokunn
kahirokunnforce-pushed
the
multi-cluster-support
branch
from
28a1e02 to
88410efCompareApril 7, 2026 14:46
@knative-prow
knative-prowBot
commented
Apr 7, 2026

Copy link


Copy Markdown

There are empty aliases in OWNER_ALIASES, cleanup is advised.
@knative-prow-robotknative-prow-robot
removed
the 
needs-rebase
Indicates a PR cannot be merged because it has merge conflicts with HEAD.
label
Apr 7, 2026
@kahirokunnkahirokunn
changed the title
Add multi-cluster deployment support via Cluster Inventory API[WIP] Add multi-cluster deployment support via Cluster Inventory APIApr 7, 2026
@knative-prowknative-prowBot
added
the 
do-not-merge/work-in-progress
Indicates that a PR should not merge because it is a work in progress.
label
Apr 7, 2026
@knative-prow
knative-prowBot
commented
Apr 7, 2026

Copy link


Copy Markdown

There are empty aliases in OWNER_ALIASES, cleanup is advised.
@kahirokunn
kahirokunnforce-pushed
the
multi-cluster-support
branch
3 times, most recently
from
29a9317 to
6217b27CompareApril 8, 2026 04:05
@knative-prow-robotknative-prow-robot
added
the 
needs-rebase
Indicates a PR cannot be merged because it has merge conflicts with HEAD.
label
Apr 16, 2026
@kahirokunn
kahirokunnforce-pushed
the
multi-cluster-support
branch
2 times, most recently
from
9068c06 to
cecc1b8CompareApril 16, 2026 05:54
@knative-prow-robotknative-prow-robot
removed
the 
needs-rebase
Indicates a PR cannot be merged because it has merge conflicts with HEAD.
label
Apr 16, 2026
@knative-prow
knative-prowBot
commented
Apr 16, 2026

Copy link


Copy Markdown

There are empty aliases in OWNER_ALIASES, cleanup is advised.
@kahirokunnkahirokunn
changed the title
[WIP] Add multi-cluster deployment support via Cluster Inventory APIAdd multi-cluster deployment support via Cluster Inventory APIApr 16, 2026
@knative-prowknative-prowBot
removed
the 
do-not-merge/work-in-progress
Indicates that a PR should not merge because it is a work in progress.
label
Apr 16, 2026
@kahirokunn
kahirokunnforce-pushed
the
multi-cluster-support
branch
from
cecc1b8 to
725b1beCompareApril 16, 2026 08:34
@kahirokunn
kahirokunn
commented
Apr 17, 2026

Copy link


Copy Markdown

MemberAuthor
I didn't have the necessary permissions.
integration-tests-multicluster_operator_main will keep failing until the following is merged, so I'd like to ignore it using an override until then #2267
@kahirokunn
kahirokunn
commented
Apr 17, 2026

Copy link


Copy Markdown

MemberAuthor
/retest
houshengbo
houshengbo
reviewed
Apr 17, 2026
View reviewed changes
@houshengbohoushengbo
left a comment
•

edited

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review
The PR description is excellent — the verification steps, design explanation, example CRs, and backward compatibility section are thorough and well-written.
2 blocking items and 7 non-blocking items are noted in inline comments below.
Comment threadpkg/apis/operator/v1beta1/knativeserving_lifecycle.go
Outdated

Comment threadpkg/reconciler/common/clusterprofile_informer.go
Comment threadpkg/reconciler/common/multicluster.go
Comment threadpkg/reconciler/common/multicluster.go
Comment threadpkg/reconciler/common/multicluster.go
Comment threadpkg/reconciler/common/transformers.go
Comment threadpkg/reconciler/common/stages.go
Outdated

Comment threadtest/e2e/knativeserving_spoke_test.go
@kahirokunn
kahirokunn
commented
Apr 21, 2026

Copy link


Copy Markdown

MemberAuthor
@houshengbo Thank you for your feedback! I have addressed all the points you raised.
@kahirokunn
kahirokunn
commented
Apr 21, 2026

Copy link


Copy Markdown

MemberAuthor
I will check it
@kahirokunn
add multi-cluster deployment support via Cluster Inventory API
4845dee
Signed-off-by: kahirokunn <okinakahiro@gmail.com>
@kahirokunn
change MarkTargetClusterNotResolved to mark only TargetClusterResolved
d1e2351
Signed-off-by: kahirokunn <okinakahiro@gmail.com>
@kahirokunn
add doRefresh error path tests for ClusterProvider
c3c500f
Signed-off-by: kahirokunn <okinakahiro@gmail.com>
@kahirokunn
mark anchor ConfigMap as protected to surface deletion risk
6b7c4e0
Signed-off-by: kahirokunn <okinakahiro@gmail.com>
@kahirokunn
stabilise ClusterProvider shutdown and propagate namespace config
4c2c9b4
Signed-off-by: kahirokunn <okinakahiro@gmail.com>
@kahirokunn
add --remote-deployments-poll-interval CLI flag
5093973
Signed-off-by: kahirokunn <okinakahiro@gmail.com>
@kahirokunn
add multi-cluster e2e test for KnativeEventing
b901f31
Signed-off-by: kahirokunn <okinakahiro@gmail.com>
@kahirokunn
./hack/update-codegen.sh
72fc85c
@kahirokunn
kahirokunn
commented
Apr 21, 2026

Copy link


Copy Markdown

MemberAuthor
All CI passed! 🙌
@houshengbo
houshengbo
commented
Apr 22, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
/lgtm

/approve
@knative-prow
knative-prowBot
commented
Apr 22, 2026

Copy link


Copy Markdown

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: houshengbo, kahirokunn
The full list of commands accepted by this bot can be found here.
The pull request process is described here
Details
Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment

Approvers can cancel approval by writing /approve cancel in a comment
@kahirokunn
kahirokunn
commented
Apr 22, 2026

Copy link


Copy Markdown

MemberAuthor
Thank you😆
@kahirokunnkahirokunn
mentioned this pull request
Apr 23, 2026
 Merged

@kahirokunnkahirokunn
mentioned this pull request
May 26, 2026

Closed


This was referenced Aug 22, 2026
 Closed

 Closed


Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@knative-prowknative-prow[bot]knative-prow[bot] left review comments
@matzewmatzewAwaiting requested review from matzew
@houshengbohoushengboAwaiting requested review from houshengbo

Assignees

@houshengbohoushengbo

Labels

approvedIndicates a PR has been approved by an approver from all required OWNERS files.lgtmIndicates that a PR is ready to be merged.size/XXLDenotes a PR that changes 1000+ lines, ignoring generated files.

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

Support multi-cluster deployment via Cluster Inventory API


3 participants

@kahirokunn@houshengbo@knative-prow-robot

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Add multi-cluster deployment support via Cluster Inventory API - #2267
Merged
knative-prow[bot] merged 8 commits into
knative:mainfrom 
kahirokunn:multi-cluster-support
Apr 22, 2026
Merged
Add multi-cluster deployment support via Cluster Inventory API#2267
knative-prow[bot] merged 8 commits into
knative:mainfrom 
kahirokunn:multi-cluster-support
Conversation
@kahirokunn
@kahirokunnkahirokunn
commented
Apr 6, 2026•

edited

Loading

Copy link


Copy Markdown

Member
Fixes#2264
Requires knative/infra#827 to be merged first so that the integration-tests-multicluster presubmit exists when this PR's CI runs.
Proposed Changes
  • Enable deploying Knative Serving and Eventing components to remote clusters by setting spec.clusterProfileRef on the CR, using the SIG-Multicluster Cluster Inventory API (KEP-5339, ClusterProfile) to discover target clusters without depending on a specific fleet manager.
  • When spec.clusterProfileRef is not set, behavior is completely unchanged — existing single-cluster deployments are unaffected.
  • On CR deletion, the operator finalizes resources on the remote cluster. If the remote cluster is unreachable, the finalizer retries; operators can remove it manually to force-delete.
Design
The core idea is to swap the manifestival manifest.Client at the start of the reconcile stage pipeline (ResolveTargetCluster), so that all subsequent stages — Apply, Delete, Get — transparently operate on the remote cluster.
For garbage collection on the remote cluster, an anchor ConfigMap pattern (inspired by k0smotron) is used:
  • An anchor ConfigMap ({kind}-{cr-name}-root-owner) is created on the remote cluster.
  • All namespace-scoped resources get an ownerReference pointing to this anchor, enabling Kubernetes-native GC.
  • Cluster-scoped resources (ClusterRole, etc.) have no ownerReference and are explicitly deleted by the finalizer.
  • Deleting the anchor ConfigMap cascades to all namespace-scoped resources via GC.
Example CR
KnativeServing on a remote cluster
apiVersion: operator.knative.dev/v1beta1kind: KnativeServingmetadata:
name: knative-serving-apacnamespace: knative-servingspec:
version: "1.21"clusterProfileRef:
name: apac-cluster-01namespace: fleet-systemconfig:
network:
ingress-class: "kourier.ingress.networking.knative.dev"ingress:
kourier:
enabled: true
KnativeEventing on a remote cluster
apiVersion: operator.knative.dev/v1beta1kind: KnativeEventingmetadata:
name: knative-eventing-apacnamespace: knative-eventingspec:
version: "1.21"clusterProfileRef:
name: apac-cluster-01namespace: fleet-system
Local cluster (unchanged behavior)
apiVersion: operator.knative.dev/v1beta1kind: KnativeServingmetadata:
name: knative-servingnamespace: knative-servingspec:
version: "1.21"# No clusterProfileRef → deploys to the local cluster as before
Verification Steps
E2E verification with Kind
Note: The credential plugin used below (kubeconfig-secretreader-plugin) is a community plugin.

An official plugin is being tracked at kubernetes-sigs/cluster-inventory-api#45 —

once released, switch to the official one.
1. Cluster setup
# Create Kind clusters
kind create cluster --name hub
kind create cluster --name spoke
# Install ClusterProfile CRD on hub
kubectl --context kind-hub apply -f https://raw.githubusercontent.com/kubernetes-sigs/cluster-inventory-api/main/config/crd/bases/multicluster.x-k8s.io_clusterprofiles.yaml
# Create namespaces on hub
kubectl --context kind-hub create namespace fleet-system
kubectl --context kind-hub create namespace knative-serving
kubectl --context kind-hub create namespace knative-operator
2. Spoke cluster credentials
# Create a ServiceAccount + token Secret on spoke for the operator to use
kubectl --context kind-spoke apply -f - <<EOFapiVersion: v1kind: ServiceAccountmetadata: name: knative-operator namespace: default---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRoleBindingmetadata: name: knative-operator-adminroleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cluster-adminsubjects:- kind: ServiceAccount name: knative-operator namespace: default---apiVersion: v1kind: Secretmetadata: name: knative-operator-token namespace: default annotations: kubernetes.io/service-account.name: knative-operatortype: kubernetes.io/service-account-tokenEOF# Wait for the token to be populatedforiin$(seq 1 30);do
TOKEN=$(kubectl --context kind-spoke get secret knative-operator-token -o go-template='{{.data.token | base64decode}}'2>/dev/null)&& [ -n"$TOKEN" ] &&break
sleep 1
done# Copy the token as a Secret on the hub cluster
kubectl --context kind-hub -n knative-operator create secret generic spoke-token \
--from-literal=token="${TOKEN}" --dry-run=client -o yaml | kubectl --context kind-hub apply -f -
3. ClusterProfile
# Get spoke cluster connection info (Docker-internal IP, not 127.0.0.1)
SPOKE_SERVER="https://$(docker inspect spoke-control-plane --format '{{ .NetworkSettings.Networks.kind.IPAddress }}'):6443"
SPOKE_CA=$(kubectl --context kind-spoke config view --minify --flatten -o jsonpath='{.clusters[0].cluster.certificate-authority-data}')# Create ClusterProfile (spec only — status is a subresource)
kubectl --context kind-hub apply -f - <<EOFapiVersion: multicluster.x-k8s.io/v1alpha1kind: ClusterProfilemetadata: name: spoke-cluster namespace: fleet-systemspec: clusterManager: name: kindEOF# Patch status with access provider info.# The extensions tell the credential plugin which Secret to read the token from.
STATUS_PATCH=$(cat <<EOF{ "status": { "accessProviders": [ { "name": "token-secretreader", "cluster": { "server": "${SPOKE_SERVER}", "certificate-authority-data": "${SPOKE_CA}", "extensions": [ { "name": "client.authentication.k8s.io/exec", "extension": { "secretName": "spoke-token", "secretNamespace": "knative-operator", "key": "token" } } ] } } ] }}EOF)
kubectl --context kind-hub patch clusterprofile spoke-cluster \
-n fleet-system --type merge --subresource=status -p "${STATUS_PATCH}"
4. Operator deployment
# Build and deploy the operator with ko.# KIND_CLUSTER_NAME ensures the image is loaded into the correct Kind cluster.
kubectl config use-context kind-hub
KO_DOCKER_REPO=kind.local KIND_CLUSTER_NAME=hub ko apply -f config/
kubectl --context kind-hub -n knative-operator \
wait --for=condition=Available deployment/knative-operator --timeout=120s
# Create clusterprofile-provider-file ConfigMap (provider name must match accessProviders[].name)
kubectl --context kind-hub -n knative-operator create configmap clusterprofile-provider-file \
--from-literal=config.json='{"providers":[{"name":"token-secretreader","execConfig":{"apiVersion":"client.authentication.k8s.io/v1","command":"/credential-plugin/kubeconfig-secretreader-plugin","provideClusterInfo":true}}]}'# Patch the operator deployment:# - Mount the credential plugin binary via image volume# - Mount the clusterprofile-provider-file config# - Add the --clusterprofile-provider-file flag
kubectl --context kind-hub -n knative-operator patch deployment knative-operator --type json -p '[ {"op":"add","path":"/spec/template/spec/containers/0/args","value":["--clusterprofile-provider-file=/etc/cluster-inventory/config.json"]}, {"op":"add","path":"/spec/template/spec/containers/0/volumeMounts","value":[ {"name":"cred-config","mountPath":"/etc/cluster-inventory","readOnly":true}, {"name":"credential-plugin","mountPath":"/credential-plugin","readOnly":true} ]}, {"op":"add","path":"/spec/template/spec/volumes","value":[ {"name":"cred-config","configMap":{"name":"clusterprofile-provider-file"}}, {"name":"credential-plugin","image":{"reference":"ghcr.io/labthrust/kubeconfig-secretreader-plugin:v0.0.1-linux-arm64"}} ]}]'
kubectl --context kind-hub -n knative-operator rollout status deployment/knative-operator --timeout=120s
5. Install Envoy Gateway on the spoke cluster
The net-gateway-api ingress provider requires Gateway API resources (GatewayClass, Gateway) to exist on the target cluster before deploying KnativeServing. Install Envoy Gateway and create the necessary resources on the spoke cluster.
Prerequisites
go install sigs.k8s.io/cloud-provider-kind@latest
Install Envoy Gateway
Since cloud-provider-kind installs Gateway API CRDs with server-side apply, helm install will fail with field ownership conflicts. Use helm template + kubectl apply --server-side --force-conflicts to take over ownership.
cat <<'EOF' > /tmp/values-eg.yamlconfig: envoyGateway: provider: type: Kubernetes kubernetes: deploy: type: GatewayNamespaceEOFexport ENVOY_GATEWAY_VERSION=v1.7.1
kubectl --context kind-spoke create namespace envoy-gateway-system
helm template eg oci://docker.io/envoyproxy/gateway-helm \
--version $ENVOY_GATEWAY_VERSION \
-n envoy-gateway-system \
-f /tmp/values-eg.yaml \
--include-crds | \
kubectl --context kind-spoke apply --server-side --force-conflicts -f -
After installing Envoy Gateway, start cloud-provider-kind in a separate terminal (provides LoadBalancer support for Kind):
sudo cloud-provider-kind
Create Gateway API resources (external)
kubectl --context kind-spoke apply -f - <<EOFapiVersion: v1kind: Namespacemetadata: name: eg-external---apiVersion: gateway.envoyproxy.io/v1alpha1kind: EnvoyProxymetadata: name: knative-external-config namespace: envoy-gateway-systemspec: provider: type: Kubernetes kubernetes: envoyService: name: knative-external---apiVersion: gateway.networking.k8s.io/v1kind: GatewayClassmetadata: name: eg-externalspec: controllerName: gateway.envoyproxy.io/gatewayclass-controller parametersRef: group: gateway.envoyproxy.io kind: EnvoyProxy name: knative-external-config namespace: envoy-gateway-system---apiVersion: gateway.networking.k8s.io/v1kind: Gatewaymetadata: name: eg-external namespace: eg-externalspec: gatewayClassName: eg-external listeners: - name: http port: 80 protocol: HTTP allowedRoutes: namespaces: from: All - name: tls port: 443 protocol: TLS tls: mode: Passthrough allowedRoutes: namespaces: from: AllEOF
Create Gateway API resources (internal)
The internal Gateway uses the ClusterIP service type, so it is not accessible from outside the cluster.
kubectl --context kind-spoke apply -f - <<EOFapiVersion: v1kind: Namespacemetadata: name: eg-internal---apiVersion: gateway.envoyproxy.io/v1alpha1kind: EnvoyProxymetadata: name: knative-internal-config namespace: envoy-gateway-systemspec: provider: type: Kubernetes kubernetes: envoyService: type: ClusterIP name: knative-internal---apiVersion: gateway.networking.k8s.io/v1kind: GatewayClassmetadata: name: eg-internalspec: controllerName: gateway.envoyproxy.io/gatewayclass-controller parametersRef: group: gateway.envoyproxy.io kind: EnvoyProxy name: knative-internal-config namespace: envoy-gateway-system---apiVersion: gateway.networking.k8s.io/v1kind: Gatewaymetadata: name: eg-internal namespace: eg-internalspec: gatewayClassName: eg-internal listeners: - name: http port: 80 protocol: HTTP allowedRoutes: namespaces: from: AllEOF
6. Deploy KnativeServing
kubectl --context kind-hub apply -f - <<EOFapiVersion: operator.knative.dev/v1beta1kind: KnativeServingmetadata: name: knative-serving namespace: knative-servingspec: clusterProfileRef: name: spoke-cluster namespace: fleet-system ingress: gateway-api: enabled: true config: config-gateway: external-gateways: | - class: eg-external gateway: eg-external/eg-external service: eg-external/knative-external supported-features: - HTTPRouteRequestTimeout local-gateways: | - class: eg-internal gateway: eg-internal/eg-internal service: eg-internal/knative-internal supported-features: - HTTPRouteRequestTimeout network: ingress.class: gateway-api.ingress.networking.knative.dev domain: example.com: ""EOF
7. Verify
# Verify resources deployed to spoke cluster
kubectl --context kind-spoke get deployments -n knative-serving
kubectl --context kind-spoke get configmaps -n knative-serving | grep root-owner
# Verify anchor ConfigMap exists with correct labels
kubectl --context kind-spoke get configmap knativeserving-knative-serving-root-owner \
-n knative-serving -o yaml
# Verify ownerReferences on namespace-scoped resources point to the anchor
kubectl --context kind-spoke get deployment activator -n knative-serving \
-o jsonpath='{.metadata.ownerReferences}'| jq .# Deploy a sample application to verify traffic routing
kubectl --context kind-spoke apply -f - <<EOFapiVersion: serving.knative.dev/v1kind: Servicemetadata: name: helloworld-go namespace: defaultspec: template: spec: containers: - image: gcr.io/knative-samples/helloworld-go env: - name: TARGET value: Go Sample v1EOF# Get the external gateway IP and testexport LB_IP=$(kubectl --context kind-spoke -n eg-external get svc knative-external \ -o jsonpath='{.status.loadBalancer.ingress[0].ip}')echo"$LB_IP helloworld-go.default.example.com"| sudo tee -a /etc/hosts
curl http://helloworld-go.default.example.com

8. Verify CR deletion and finalizer cleanup
# Delete the CR and wait for the finalizer to complete
kubectl --context kind-hub delete knativeserving knative-serving -n knative-serving
# Verify spoke cluster is clean
kubectl --context kind-spoke get all -n knative-serving
kubectl --context kind-spoke get clusterroles | grep knative
9. Cleanup
kind delete cluster --name hub
kind delete cluster --name spoke
Backward compatibility
# Deploy without --clusterprofile-provider-file and without clusterProfileRef# → must behave identically to previous releases
kubectl apply -f - <<EOFapiVersion: operator.knative.dev/v1beta1kind: KnativeServingmetadata: name: knative-serving namespace: knative-servingspec: version: "1.21"EOF
kubectl get knativeserving -n knative-serving -o jsonpath='{.status.conditions}'
Release Note
Add multi-cluster deployment support via the Cluster Inventory API (KEP-5339). When `spec.clusterProfileRef` is set on a KnativeServing or KnativeEventing CR, the operator deploys components to the remote cluster described by the referenced ClusterProfile. Requires the `--clusterprofile-provider-file` flag to be set on the operator deployment.

@knative-prowknative-prowBot
added
the 
size/XXL
Denotes a PR that changes 1000+ lines, ignoring generated files.
label
Apr 6, 2026
knative-prow[bot]
knative-prowBot
reviewed
Apr 6, 2026
View reviewed changes
@knative-prowknative-prowBot
left a comment


Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kahirokunn: 0 warnings.
Details
In response to this:
Fixes #
Proposed Changes
Release Note
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.
@codecov
codecovBot
commented
Apr 6, 2026•

edited

Loading

Copy link


Copy Markdown

Codecov Report
❌ Patch coverage is 60.40000% with 198 lines in your changes missing coverage. Please review.

✅ Project coverage is 63.84%. Comparing base (9c66d7b) to head (72fc85c).
⚠️ Report is 1 commits behind head on main.
Files with missing linesPatch %Lines
pkg/reconciler/common/multicluster.go62.18%121 Missing and 14 partials ⚠️
pkg/reconciler/knativeeventing/controller.go0.00%25 Missing ⚠️
pkg/reconciler/knativeserving/controller.go0.00%25 Missing ⚠️
pkg/reconciler/common/clusterprofile_informer.go86.53%3 Missing and 4 partials ⚠️
pkg/apis/operator/base/common.go0.00%2 Missing ⚠️
pkg/reconciler/common/access_provider_flag.go66.66%2 Missing ⚠️
pkg/reconciler/knativeeventing/eventing_tls.go0.00%2 Missing ⚠️
Additional details and impacted files
@@ Coverage Diff @@## main #2267 +/- ##
==========================================
- Coverage 64.58% 63.84% -0.75% 
==========================================
Files 51 55 +4 Lines 1999 2478 +479 ==========================================
+ Hits 1291 1582 +291 - Misses 606 777 +171 - Partials 102 119 +17 
☔ View full report in Codecov by Sentry.

📢 Have feedback on the report? Share it here.
 🚀 New features to boost your workflow: 
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
@kahirokunnkahirokunn
changed the title
Add multi-cluster deployment support via Cluster Inventory API[WIP] Add multi-cluster deployment support via Cluster Inventory APIApr 6, 2026
@knative-prowknative-prowBot
added
the 
do-not-merge/work-in-progress
Indicates that a PR should not merge because it is a work in progress.
label
Apr 6, 2026
@kahirokunn
kahirokunnforce-pushed
the
multi-cluster-support
branch
5 times, most recently
from
3e585a9 to
141c875CompareApril 7, 2026 06:40
@knative-prow-robotknative-prow-robot
added
the 
needs-rebase
Indicates a PR cannot be merged because it has merge conflicts with HEAD.
label
Apr 7, 2026
@kahirokunn
kahirokunnforce-pushed
the
multi-cluster-support
branch
2 times, most recently
from
d308bf0 to
3e742cbCompareApril 7, 2026 12:53
@kahirokunnkahirokunn
changed the title
[WIP] Add multi-cluster deployment support via Cluster Inventory APIAdd multi-cluster deployment support via Cluster Inventory APIApr 7, 2026
@knative-prowknative-prowBot
removed
the 
do-not-merge/work-in-progress
Indicates that a PR should not merge because it is a work in progress.
label
Apr 7, 2026
@kahirokunn
kahirokunnforce-pushed
the
multi-cluster-support
branch
from
3e742cb to
43919c6CompareApril 7, 2026 14:11
@knative-prow-robotknative-prow-robot
removed
the 
needs-rebase
Indicates a PR cannot be merged because it has merge conflicts with HEAD.
label
Apr 7, 2026
@kahirokunn
kahirokunnforce-pushed
the
multi-cluster-support
branch
from
43919c6 to
28a1e02CompareApril 7, 2026 14:22
@knative-prow-robotknative-prow-robot
added
the 
needs-rebase
Indicates a PR cannot be merged because it has merge conflicts with HEAD.
label
Apr 7, 2026
@kahirokunn
kahirokunnforce-pushed
the
multi-cluster-support
branch
from
28a1e02 to
88410efCompareApril 7, 2026 14:46
@knative-prow
knative-prowBot
commented
Apr 7, 2026

Copy link


Copy Markdown

There are empty aliases in OWNER_ALIASES, cleanup is advised.
@knative-prow-robotknative-prow-robot
removed
the 
needs-rebase
Indicates a PR cannot be merged because it has merge conflicts with HEAD.
label
Apr 7, 2026
@kahirokunnkahirokunn
changed the title
Add multi-cluster deployment support via Cluster Inventory API[WIP] Add multi-cluster deployment support via Cluster Inventory APIApr 7, 2026
@knative-prowknative-prowBot
added
the 
do-not-merge/work-in-progress
Indicates that a PR should not merge because it is a work in progress.
label
Apr 7, 2026
@knative-prow
knative-prowBot
commented
Apr 7, 2026

Copy link


Copy Markdown

There are empty aliases in OWNER_ALIASES, cleanup is advised.
@kahirokunn
kahirokunnforce-pushed
the
multi-cluster-support
branch
3 times, most recently
from
29a9317 to
6217b27CompareApril 8, 2026 04:05
@knative-prow-robotknative-prow-robot
added
the 
needs-rebase
Indicates a PR cannot be merged because it has merge conflicts with HEAD.
label
Apr 16, 2026
@kahirokunn
kahirokunnforce-pushed
the
multi-cluster-support
branch
2 times, most recently
from
9068c06 to
cecc1b8CompareApril 16, 2026 05:54
@knative-prow-robotknative-prow-robot
removed
the 
needs-rebase
Indicates a PR cannot be merged because it has merge conflicts with HEAD.
label
Apr 16, 2026
@knative-prow
knative-prowBot
commented
Apr 16, 2026

Copy link


Copy Markdown

There are empty aliases in OWNER_ALIASES, cleanup is advised.
@kahirokunnkahirokunn
changed the title
[WIP] Add multi-cluster deployment support via Cluster Inventory APIAdd multi-cluster deployment support via Cluster Inventory APIApr 16, 2026
@knative-prowknative-prowBot
removed
the 
do-not-merge/work-in-progress
Indicates that a PR should not merge because it is a work in progress.
label
Apr 16, 2026
@kahirokunn
kahirokunnforce-pushed
the
multi-cluster-support
branch
from
cecc1b8 to
725b1beCompareApril 16, 2026 08:34
@kahirokunn
kahirokunn
commented
Apr 17, 2026

Copy link


Copy Markdown

MemberAuthor
I didn't have the necessary permissions.
integration-tests-multicluster_operator_main will keep failing until the following is merged, so I'd like to ignore it using an override until then #2267
@kahirokunn
kahirokunn
commented
Apr 17, 2026

Copy link


Copy Markdown

MemberAuthor
/retest
houshengbo
houshengbo
reviewed
Apr 17, 2026
View reviewed changes
@houshengbohoushengbo
left a comment
•

edited

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review
The PR description is excellent — the verification steps, design explanation, example CRs, and backward compatibility section are thorough and well-written.
2 blocking items and 7 non-blocking items are noted in inline comments below.
Comment threadpkg/apis/operator/v1beta1/knativeserving_lifecycle.go
Outdated

Comment threadpkg/reconciler/common/clusterprofile_informer.go
Comment threadpkg/reconciler/common/multicluster.go
Comment threadpkg/reconciler/common/multicluster.go
Comment threadpkg/reconciler/common/multicluster.go
Comment threadpkg/reconciler/common/transformers.go
Comment threadpkg/reconciler/common/stages.go
Outdated

Comment threadtest/e2e/knativeserving_spoke_test.go
@kahirokunn
kahirokunn
commented
Apr 21, 2026

Copy link


Copy Markdown

MemberAuthor
@houshengbo Thank you for your feedback! I have addressed all the points you raised.
@kahirokunn
kahirokunn
commented
Apr 21, 2026

Copy link


Copy Markdown

MemberAuthor
I will check it
@kahirokunn
add multi-cluster deployment support via Cluster Inventory API
4845dee
Signed-off-by: kahirokunn <okinakahiro@gmail.com>
@kahirokunn
change MarkTargetClusterNotResolved to mark only TargetClusterResolved
d1e2351
Signed-off-by: kahirokunn <okinakahiro@gmail.com>
@kahirokunn
add doRefresh error path tests for ClusterProvider
c3c500f
Signed-off-by: kahirokunn <okinakahiro@gmail.com>
@kahirokunn
mark anchor ConfigMap as protected to surface deletion risk
6b7c4e0
Signed-off-by: kahirokunn <okinakahiro@gmail.com>
@kahirokunn
stabilise ClusterProvider shutdown and propagate namespace config
4c2c9b4
Signed-off-by: kahirokunn <okinakahiro@gmail.com>
@kahirokunn
add --remote-deployments-poll-interval CLI flag
5093973
Signed-off-by: kahirokunn <okinakahiro@gmail.com>
@kahirokunn
add multi-cluster e2e test for KnativeEventing
b901f31
Signed-off-by: kahirokunn <okinakahiro@gmail.com>
@kahirokunn
./hack/update-codegen.sh
72fc85c
@kahirokunn
kahirokunn
commented
Apr 21, 2026

Copy link


Copy Markdown

MemberAuthor
All CI passed! 🙌
@houshengbo
houshengbo
commented
Apr 22, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
/lgtm

/approve
@knative-prow
knative-prowBot
commented
Apr 22, 2026

Copy link


Copy Markdown

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: houshengbo, kahirokunn
The full list of commands accepted by this bot can be found here.
The pull request process is described here
Details
Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment

Approvers can cancel approval by writing /approve cancel in a comment
@kahirokunn
kahirokunn
commented
Apr 22, 2026

Copy link


Copy Markdown

MemberAuthor
Thank you😆
@kahirokunnkahirokunn
mentioned this pull request
Apr 23, 2026
 Merged

@kahirokunnkahirokunn
mentioned this pull request
May 26, 2026

Closed


This was referenced Aug 22, 2026
 Closed

 Closed


Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@knative-prowknative-prow[bot]knative-prow[bot] left review comments
@matzewmatzewAwaiting requested review from matzew
@houshengbohoushengboAwaiting requested review from houshengbo

Assignees

@houshengbohoushengbo

Labels

approvedIndicates a PR has been approved by an approver from all required OWNERS files.lgtmIndicates that a PR is ready to be merged.size/XXLDenotes a PR that changes 1000+ lines, ignoring generated files.

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

Support multi-cluster deployment via Cluster Inventory API


3 participants

@kahirokunn@houshengbo@knative-prow-robot

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Add multi-cluster deployment support via Cluster Inventory API - #2267
Merged
knative-prow[bot] merged 8 commits into
knative:mainfrom 
kahirokunn:multi-cluster-support
Apr 22, 2026
Merged
Add multi-cluster deployment support via Cluster Inventory API#2267
knative-prow[bot] merged 8 commits into
knative:mainfrom 
kahirokunn:multi-cluster-support
Conversation
@kahirokunn
@kahirokunnkahirokunn
commented
Apr 6, 2026•

edited

Loading

Copy link


Copy Markdown

Member
Fixes#2264
Requires knative/infra#827 to be merged first so that the integration-tests-multicluster presubmit exists when this PR's CI runs.
Proposed Changes
  • Enable deploying Knative Serving and Eventing components to remote clusters by setting spec.clusterProfileRef on the CR, using the SIG-Multicluster Cluster Inventory API (KEP-5339, ClusterProfile) to discover target clusters without depending on a specific fleet manager.
  • When spec.clusterProfileRef is not set, behavior is completely unchanged — existing single-cluster deployments are unaffected.
  • On CR deletion, the operator finalizes resources on the remote cluster. If the remote cluster is unreachable, the finalizer retries; operators can remove it manually to force-delete.
Design
The core idea is to swap the manifestival manifest.Client at the start of the reconcile stage pipeline (ResolveTargetCluster), so that all subsequent stages — Apply, Delete, Get — transparently operate on the remote cluster.
For garbage collection on the remote cluster, an anchor ConfigMap pattern (inspired by k0smotron) is used:
  • An anchor ConfigMap ({kind}-{cr-name}-root-owner) is created on the remote cluster.
  • All namespace-scoped resources get an ownerReference pointing to this anchor, enabling Kubernetes-native GC.
  • Cluster-scoped resources (ClusterRole, etc.) have no ownerReference and are explicitly deleted by the finalizer.
  • Deleting the anchor ConfigMap cascades to all namespace-scoped resources via GC.
Example CR
KnativeServing on a remote cluster
apiVersion: operator.knative.dev/v1beta1kind: KnativeServingmetadata:
name: knative-serving-apacnamespace: knative-servingspec:
version: "1.21"clusterProfileRef:
name: apac-cluster-01namespace: fleet-systemconfig:
network:
ingress-class: "kourier.ingress.networking.knative.dev"ingress:
kourier:
enabled: true
KnativeEventing on a remote cluster
apiVersion: operator.knative.dev/v1beta1kind: KnativeEventingmetadata:
name: knative-eventing-apacnamespace: knative-eventingspec:
version: "1.21"clusterProfileRef:
name: apac-cluster-01namespace: fleet-system
Local cluster (unchanged behavior)
apiVersion: operator.knative.dev/v1beta1kind: KnativeServingmetadata:
name: knative-servingnamespace: knative-servingspec:
version: "1.21"# No clusterProfileRef → deploys to the local cluster as before
Verification Steps
E2E verification with Kind
Note: The credential plugin used below (kubeconfig-secretreader-plugin) is a community plugin.

An official plugin is being tracked at kubernetes-sigs/cluster-inventory-api#45 —

once released, switch to the official one.
1. Cluster setup
# Create Kind clusters
kind create cluster --name hub
kind create cluster --name spoke
# Install ClusterProfile CRD on hub
kubectl --context kind-hub apply -f https://raw.githubusercontent.com/kubernetes-sigs/cluster-inventory-api/main/config/crd/bases/multicluster.x-k8s.io_clusterprofiles.yaml
# Create namespaces on hub
kubectl --context kind-hub create namespace fleet-system
kubectl --context kind-hub create namespace knative-serving
kubectl --context kind-hub create namespace knative-operator
2. Spoke cluster credentials
# Create a ServiceAccount + token Secret on spoke for the operator to use
kubectl --context kind-spoke apply -f - <<EOFapiVersion: v1kind: ServiceAccountmetadata: name: knative-operator namespace: default---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRoleBindingmetadata: name: knative-operator-adminroleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cluster-adminsubjects:- kind: ServiceAccount name: knative-operator namespace: default---apiVersion: v1kind: Secretmetadata: name: knative-operator-token namespace: default annotations: kubernetes.io/service-account.name: knative-operatortype: kubernetes.io/service-account-tokenEOF# Wait for the token to be populatedforiin$(seq 1 30);do
TOKEN=$(kubectl --context kind-spoke get secret knative-operator-token -o go-template='{{.data.token | base64decode}}'2>/dev/null)&& [ -n"$TOKEN" ] &&break
sleep 1
done# Copy the token as a Secret on the hub cluster
kubectl --context kind-hub -n knative-operator create secret generic spoke-token \
--from-literal=token="${TOKEN}" --dry-run=client -o yaml | kubectl --context kind-hub apply -f -
3. ClusterProfile
# Get spoke cluster connection info (Docker-internal IP, not 127.0.0.1)
SPOKE_SERVER="https://$(docker inspect spoke-control-plane --format '{{ .NetworkSettings.Networks.kind.IPAddress }}'):6443"
SPOKE_CA=$(kubectl --context kind-spoke config view --minify --flatten -o jsonpath='{.clusters[0].cluster.certificate-authority-data}')# Create ClusterProfile (spec only — status is a subresource)
kubectl --context kind-hub apply -f - <<EOFapiVersion: multicluster.x-k8s.io/v1alpha1kind: ClusterProfilemetadata: name: spoke-cluster namespace: fleet-systemspec: clusterManager: name: kindEOF# Patch status with access provider info.# The extensions tell the credential plugin which Secret to read the token from.
STATUS_PATCH=$(cat <<EOF{ "status": { "accessProviders": [ { "name": "token-secretreader", "cluster": { "server": "${SPOKE_SERVER}", "certificate-authority-data": "${SPOKE_CA}", "extensions": [ { "name": "client.authentication.k8s.io/exec", "extension": { "secretName": "spoke-token", "secretNamespace": "knative-operator", "key": "token" } } ] } } ] }}EOF)
kubectl --context kind-hub patch clusterprofile spoke-cluster \
-n fleet-system --type merge --subresource=status -p "${STATUS_PATCH}"
4. Operator deployment
# Build and deploy the operator with ko.# KIND_CLUSTER_NAME ensures the image is loaded into the correct Kind cluster.
kubectl config use-context kind-hub
KO_DOCKER_REPO=kind.local KIND_CLUSTER_NAME=hub ko apply -f config/
kubectl --context kind-hub -n knative-operator \
wait --for=condition=Available deployment/knative-operator --timeout=120s
# Create clusterprofile-provider-file ConfigMap (provider name must match accessProviders[].name)
kubectl --context kind-hub -n knative-operator create configmap clusterprofile-provider-file \
--from-literal=config.json='{"providers":[{"name":"token-secretreader","execConfig":{"apiVersion":"client.authentication.k8s.io/v1","command":"/credential-plugin/kubeconfig-secretreader-plugin","provideClusterInfo":true}}]}'# Patch the operator deployment:# - Mount the credential plugin binary via image volume# - Mount the clusterprofile-provider-file config# - Add the --clusterprofile-provider-file flag
kubectl --context kind-hub -n knative-operator patch deployment knative-operator --type json -p '[ {"op":"add","path":"/spec/template/spec/containers/0/args","value":["--clusterprofile-provider-file=/etc/cluster-inventory/config.json"]}, {"op":"add","path":"/spec/template/spec/containers/0/volumeMounts","value":[ {"name":"cred-config","mountPath":"/etc/cluster-inventory","readOnly":true}, {"name":"credential-plugin","mountPath":"/credential-plugin","readOnly":true} ]}, {"op":"add","path":"/spec/template/spec/volumes","value":[ {"name":"cred-config","configMap":{"name":"clusterprofile-provider-file"}}, {"name":"credential-plugin","image":{"reference":"ghcr.io/labthrust/kubeconfig-secretreader-plugin:v0.0.1-linux-arm64"}} ]}]'
kubectl --context kind-hub -n knative-operator rollout status deployment/knative-operator --timeout=120s
5. Install Envoy Gateway on the spoke cluster
The net-gateway-api ingress provider requires Gateway API resources (GatewayClass, Gateway) to exist on the target cluster before deploying KnativeServing. Install Envoy Gateway and create the necessary resources on the spoke cluster.
Prerequisites
go install sigs.k8s.io/cloud-provider-kind@latest
Install Envoy Gateway
Since cloud-provider-kind installs Gateway API CRDs with server-side apply, helm install will fail with field ownership conflicts. Use helm template + kubectl apply --server-side --force-conflicts to take over ownership.
cat <<'EOF' > /tmp/values-eg.yamlconfig: envoyGateway: provider: type: Kubernetes kubernetes: deploy: type: GatewayNamespaceEOFexport ENVOY_GATEWAY_VERSION=v1.7.1
kubectl --context kind-spoke create namespace envoy-gateway-system
helm template eg oci://docker.io/envoyproxy/gateway-helm \
--version $ENVOY_GATEWAY_VERSION \
-n envoy-gateway-system \
-f /tmp/values-eg.yaml \
--include-crds | \
kubectl --context kind-spoke apply --server-side --force-conflicts -f -
After installing Envoy Gateway, start cloud-provider-kind in a separate terminal (provides LoadBalancer support for Kind):
sudo cloud-provider-kind
Create Gateway API resources (external)
kubectl --context kind-spoke apply -f - <<EOFapiVersion: v1kind: Namespacemetadata: name: eg-external---apiVersion: gateway.envoyproxy.io/v1alpha1kind: EnvoyProxymetadata: name: knative-external-config namespace: envoy-gateway-systemspec: provider: type: Kubernetes kubernetes: envoyService: name: knative-external---apiVersion: gateway.networking.k8s.io/v1kind: GatewayClassmetadata: name: eg-externalspec: controllerName: gateway.envoyproxy.io/gatewayclass-controller parametersRef: group: gateway.envoyproxy.io kind: EnvoyProxy name: knative-external-config namespace: envoy-gateway-system---apiVersion: gateway.networking.k8s.io/v1kind: Gatewaymetadata: name: eg-external namespace: eg-externalspec: gatewayClassName: eg-external listeners: - name: http port: 80 protocol: HTTP allowedRoutes: namespaces: from: All - name: tls port: 443 protocol: TLS tls: mode: Passthrough allowedRoutes: namespaces: from: AllEOF
Create Gateway API resources (internal)
The internal Gateway uses the ClusterIP service type, so it is not accessible from outside the cluster.
kubectl --context kind-spoke apply -f - <<EOFapiVersion: v1kind: Namespacemetadata: name: eg-internal---apiVersion: gateway.envoyproxy.io/v1alpha1kind: EnvoyProxymetadata: name: knative-internal-config namespace: envoy-gateway-systemspec: provider: type: Kubernetes kubernetes: envoyService: type: ClusterIP name: knative-internal---apiVersion: gateway.networking.k8s.io/v1kind: GatewayClassmetadata: name: eg-internalspec: controllerName: gateway.envoyproxy.io/gatewayclass-controller parametersRef: group: gateway.envoyproxy.io kind: EnvoyProxy name: knative-internal-config namespace: envoy-gateway-system---apiVersion: gateway.networking.k8s.io/v1kind: Gatewaymetadata: name: eg-internal namespace: eg-internalspec: gatewayClassName: eg-internal listeners: - name: http port: 80 protocol: HTTP allowedRoutes: namespaces: from: AllEOF
6. Deploy KnativeServing
kubectl --context kind-hub apply -f - <<EOFapiVersion: operator.knative.dev/v1beta1kind: KnativeServingmetadata: name: knative-serving namespace: knative-servingspec: clusterProfileRef: name: spoke-cluster namespace: fleet-system ingress: gateway-api: enabled: true config: config-gateway: external-gateways: | - class: eg-external gateway: eg-external/eg-external service: eg-external/knative-external supported-features: - HTTPRouteRequestTimeout local-gateways: | - class: eg-internal gateway: eg-internal/eg-internal service: eg-internal/knative-internal supported-features: - HTTPRouteRequestTimeout network: ingress.class: gateway-api.ingress.networking.knative.dev domain: example.com: ""EOF
7. Verify
# Verify resources deployed to spoke cluster
kubectl --context kind-spoke get deployments -n knative-serving
kubectl --context kind-spoke get configmaps -n knative-serving | grep root-owner
# Verify anchor ConfigMap exists with correct labels
kubectl --context kind-spoke get configmap knativeserving-knative-serving-root-owner \
-n knative-serving -o yaml
# Verify ownerReferences on namespace-scoped resources point to the anchor
kubectl --context kind-spoke get deployment activator -n knative-serving \
-o jsonpath='{.metadata.ownerReferences}'| jq .# Deploy a sample application to verify traffic routing
kubectl --context kind-spoke apply -f - <<EOFapiVersion: serving.knative.dev/v1kind: Servicemetadata: name: helloworld-go namespace: defaultspec: template: spec: containers: - image: gcr.io/knative-samples/helloworld-go env: - name: TARGET value: Go Sample v1EOF# Get the external gateway IP and testexport LB_IP=$(kubectl --context kind-spoke -n eg-external get svc knative-external \ -o jsonpath='{.status.loadBalancer.ingress[0].ip}')echo"$LB_IP helloworld-go.default.example.com"| sudo tee -a /etc/hosts
curl http://helloworld-go.default.example.com

8. Verify CR deletion and finalizer cleanup
# Delete the CR and wait for the finalizer to complete
kubectl --context kind-hub delete knativeserving knative-serving -n knative-serving
# Verify spoke cluster is clean
kubectl --context kind-spoke get all -n knative-serving
kubectl --context kind-spoke get clusterroles | grep knative
9. Cleanup
kind delete cluster --name hub
kind delete cluster --name spoke
Backward compatibility
# Deploy without --clusterprofile-provider-file and without clusterProfileRef# → must behave identically to previous releases
kubectl apply -f - <<EOFapiVersion: operator.knative.dev/v1beta1kind: KnativeServingmetadata: name: knative-serving namespace: knative-servingspec: version: "1.21"EOF
kubectl get knativeserving -n knative-serving -o jsonpath='{.status.conditions}'
Release Note
Add multi-cluster deployment support via the Cluster Inventory API (KEP-5339). When `spec.clusterProfileRef` is set on a KnativeServing or KnativeEventing CR, the operator deploys components to the remote cluster described by the referenced ClusterProfile. Requires the `--clusterprofile-provider-file` flag to be set on the operator deployment.

@knative-prowknative-prowBot
added
the 
size/XXL
Denotes a PR that changes 1000+ lines, ignoring generated files.
label
Apr 6, 2026
knative-prow[bot]
knative-prowBot
reviewed
Apr 6, 2026
View reviewed changes
@knative-prowknative-prowBot
left a comment


Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kahirokunn: 0 warnings.
Details
In response to this:
Fixes #
Proposed Changes
Release Note
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.
@codecov
codecovBot
commented
Apr 6, 2026•

edited

Loading

Copy link


Copy Markdown

Codecov Report
❌ Patch coverage is 60.40000% with 198 lines in your changes missing coverage. Please review.

✅ Project coverage is 63.84%. Comparing base (9c66d7b) to head (72fc85c).
⚠️ Report is 1 commits behind head on main.
Files with missing linesPatch %Lines
pkg/reconciler/common/multicluster.go62.18%121 Missing and 14 partials ⚠️
pkg/reconciler/knativeeventing/controller.go0.00%25 Missing ⚠️
pkg/reconciler/knativeserving/controller.go0.00%25 Missing ⚠️
pkg/reconciler/common/clusterprofile_informer.go86.53%3 Missing and 4 partials ⚠️
pkg/apis/operator/base/common.go0.00%2 Missing ⚠️
pkg/reconciler/common/access_provider_flag.go66.66%2 Missing ⚠️
pkg/reconciler/knativeeventing/eventing_tls.go0.00%2 Missing ⚠️
Additional details and impacted files
@@ Coverage Diff @@## main #2267 +/- ##
==========================================
- Coverage 64.58% 63.84% -0.75% 
==========================================
Files 51 55 +4 Lines 1999 2478 +479 ==========================================
+ Hits 1291 1582 +291 - Misses 606 777 +171 - Partials 102 119 +17 
☔ View full report in Codecov by Sentry.

📢 Have feedback on the report? Share it here.
 🚀 New features to boost your workflow: 
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
@kahirokunnkahirokunn
changed the title
Add multi-cluster deployment support via Cluster Inventory API[WIP] Add multi-cluster deployment support via Cluster Inventory APIApr 6, 2026
@knative-prowknative-prowBot
added
the 
do-not-merge/work-in-progress
Indicates that a PR should not merge because it is a work in progress.
label
Apr 6, 2026
@kahirokunn
kahirokunnforce-pushed
the
multi-cluster-support
branch
5 times, most recently
from
3e585a9 to
141c875CompareApril 7, 2026 06:40
@knative-prow-robotknative-prow-robot
added
the 
needs-rebase
Indicates a PR cannot be merged because it has merge conflicts with HEAD.
label
Apr 7, 2026
@kahirokunn
kahirokunnforce-pushed
the
multi-cluster-support
branch
2 times, most recently
from
d308bf0 to
3e742cbCompareApril 7, 2026 12:53
@kahirokunnkahirokunn
changed the title
[WIP] Add multi-cluster deployment support via Cluster Inventory APIAdd multi-cluster deployment support via Cluster Inventory APIApr 7, 2026
@knative-prowknative-prowBot
removed
the 
do-not-merge/work-in-progress
Indicates that a PR should not merge because it is a work in progress.
label
Apr 7, 2026
@kahirokunn
kahirokunnforce-pushed
the
multi-cluster-support
branch
from
3e742cb to
43919c6CompareApril 7, 2026 14:11
@knative-prow-robotknative-prow-robot
removed
the 
needs-rebase
Indicates a PR cannot be merged because it has merge conflicts with HEAD.
label
Apr 7, 2026
@kahirokunn
kahirokunnforce-pushed
the
multi-cluster-support
branch
from
43919c6 to
28a1e02CompareApril 7, 2026 14:22
@knative-prow-robotknative-prow-robot
added
the 
needs-rebase
Indicates a PR cannot be merged because it has merge conflicts with HEAD.
label
Apr 7, 2026
@kahirokunn
kahirokunnforce-pushed
the
multi-cluster-support
branch
from
28a1e02 to
88410efCompareApril 7, 2026 14:46
@knative-prow
knative-prowBot
commented
Apr 7, 2026

Copy link


Copy Markdown

There are empty aliases in OWNER_ALIASES, cleanup is advised.
@knative-prow-robotknative-prow-robot
removed
the 
needs-rebase
Indicates a PR cannot be merged because it has merge conflicts with HEAD.
label
Apr 7, 2026
@kahirokunnkahirokunn
changed the title
Add multi-cluster deployment support via Cluster Inventory API[WIP] Add multi-cluster deployment support via Cluster Inventory APIApr 7, 2026
@knative-prowknative-prowBot
added
the 
do-not-merge/work-in-progress
Indicates that a PR should not merge because it is a work in progress.
label
Apr 7, 2026
@knative-prow
knative-prowBot
commented
Apr 7, 2026

Copy link


Copy Markdown

There are empty aliases in OWNER_ALIASES, cleanup is advised.
@kahirokunn
kahirokunnforce-pushed
the
multi-cluster-support
branch
3 times, most recently
from
29a9317 to
6217b27CompareApril 8, 2026 04:05
@knative-prow-robotknative-prow-robot
added
the 
needs-rebase
Indicates a PR cannot be merged because it has merge conflicts with HEAD.
label
Apr 16, 2026
@kahirokunn
kahirokunnforce-pushed
the
multi-cluster-support
branch
2 times, most recently
from
9068c06 to
cecc1b8CompareApril 16, 2026 05:54
@knative-prow-robotknative-prow-robot
removed
the 
needs-rebase
Indicates a PR cannot be merged because it has merge conflicts with HEAD.
label
Apr 16, 2026
@knative-prow
knative-prowBot
commented
Apr 16, 2026

Copy link


Copy Markdown

There are empty aliases in OWNER_ALIASES, cleanup is advised.
@kahirokunnkahirokunn
changed the title
[WIP] Add multi-cluster deployment support via Cluster Inventory APIAdd multi-cluster deployment support via Cluster Inventory APIApr 16, 2026
@knative-prowknative-prowBot
removed
the 
do-not-merge/work-in-progress
Indicates that a PR should not merge because it is a work in progress.
label
Apr 16, 2026
@kahirokunn
kahirokunnforce-pushed
the
multi-cluster-support
branch
from
cecc1b8 to
725b1beCompareApril 16, 2026 08:34
@kahirokunn
kahirokunn
commented
Apr 17, 2026

Copy link


Copy Markdown

MemberAuthor
I didn't have the necessary permissions.
integration-tests-multicluster_operator_main will keep failing until the following is merged, so I'd like to ignore it using an override until then #2267
@kahirokunn
kahirokunn
commented
Apr 17, 2026

Copy link


Copy Markdown

MemberAuthor
/retest
houshengbo
houshengbo
reviewed
Apr 17, 2026
View reviewed changes
@houshengbohoushengbo
left a comment
•

edited

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review
The PR description is excellent — the verification steps, design explanation, example CRs, and backward compatibility section are thorough and well-written.
2 blocking items and 7 non-blocking items are noted in inline comments below.
Comment threadpkg/apis/operator/v1beta1/knativeserving_lifecycle.go
Outdated

Comment threadpkg/reconciler/common/clusterprofile_informer.go
Comment threadpkg/reconciler/common/multicluster.go
Comment threadpkg/reconciler/common/multicluster.go
Comment threadpkg/reconciler/common/multicluster.go
Comment threadpkg/reconciler/common/transformers.go
Comment threadpkg/reconciler/common/stages.go
Outdated

Comment threadtest/e2e/knativeserving_spoke_test.go
@kahirokunn
kahirokunn
commented
Apr 21, 2026

Copy link


Copy Markdown

MemberAuthor
@houshengbo Thank you for your feedback! I have addressed all the points you raised.
@kahirokunn
kahirokunn
commented
Apr 21, 2026

Copy link


Copy Markdown

MemberAuthor
I will check it
@kahirokunn
add multi-cluster deployment support via Cluster Inventory API
4845dee
Signed-off-by: kahirokunn <okinakahiro@gmail.com>
@kahirokunn
change MarkTargetClusterNotResolved to mark only TargetClusterResolved
d1e2351
Signed-off-by: kahirokunn <okinakahiro@gmail.com>
@kahirokunn
add doRefresh error path tests for ClusterProvider
c3c500f
Signed-off-by: kahirokunn <okinakahiro@gmail.com>
@kahirokunn
mark anchor ConfigMap as protected to surface deletion risk
6b7c4e0
Signed-off-by: kahirokunn <okinakahiro@gmail.com>
@kahirokunn
stabilise ClusterProvider shutdown and propagate namespace config
4c2c9b4
Signed-off-by: kahirokunn <okinakahiro@gmail.com>
@kahirokunn
add --remote-deployments-poll-interval CLI flag
5093973
Signed-off-by: kahirokunn <okinakahiro@gmail.com>
@kahirokunn
add multi-cluster e2e test for KnativeEventing
b901f31
Signed-off-by: kahirokunn <okinakahiro@gmail.com>
@kahirokunn
./hack/update-codegen.sh
72fc85c
@kahirokunn
kahirokunn
commented
Apr 21, 2026

Copy link


Copy Markdown

MemberAuthor
All CI passed! 🙌
@houshengbo
houshengbo
commented
Apr 22, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
/lgtm

/approve
@knative-prow
knative-prowBot
commented
Apr 22, 2026

Copy link


Copy Markdown

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: houshengbo, kahirokunn
The full list of commands accepted by this bot can be found here.
The pull request process is described here
Details
Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment

Approvers can cancel approval by writing /approve cancel in a comment
@kahirokunn
kahirokunn
commented
Apr 22, 2026

Copy link


Copy Markdown

MemberAuthor
Thank you😆
@kahirokunnkahirokunn
mentioned this pull request
Apr 23, 2026
 Merged

@kahirokunnkahirokunn
mentioned this pull request
May 26, 2026

Closed


This was referenced Aug 22, 2026
 Closed

 Closed


Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@knative-prowknative-prow[bot]knative-prow[bot] left review comments
@matzewmatzewAwaiting requested review from matzew
@houshengbohoushengboAwaiting requested review from houshengbo

Assignees

@houshengbohoushengbo

Labels

approvedIndicates a PR has been approved by an approver from all required OWNERS files.lgtmIndicates that a PR is ready to be merged.size/XXLDenotes a PR that changes 1000+ lines, ignoring generated files.

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

Support multi-cluster deployment via Cluster Inventory API


3 participants

@kahirokunn@houshengbo@knative-prow-robot

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
Add multi-cluster deployment support via Cluster Inventory API - #2267
Merged
knative-prow[bot] merged 8 commits into
knative:mainfrom 
kahirokunn:multi-cluster-support
Apr 22, 2026
Merged
Add multi-cluster deployment support via Cluster Inventory API#2267
knative-prow[bot] merged 8 commits into
knative:mainfrom 
kahirokunn:multi-cluster-support
Conversation
@kahirokunn
@kahirokunnkahirokunn
commented
Apr 6, 2026•

edited

Loading

Copy link


Copy Markdown

Member
Fixes#2264
Requires knative/infra#827 to be merged first so that the integration-tests-multicluster presubmit exists when this PR's CI runs.
Proposed Changes
  • Enable deploying Knative Serving and Eventing components to remote clusters by setting spec.clusterProfileRef on the CR, using the SIG-Multicluster Cluster Inventory API (KEP-5339, ClusterProfile) to discover target clusters without depending on a specific fleet manager.
  • When spec.clusterProfileRef is not set, behavior is completely unchanged — existing single-cluster deployments are unaffected.
  • On CR deletion, the operator finalizes resources on the remote cluster. If the remote cluster is unreachable, the finalizer retries; operators can remove it manually to force-delete.
Design
The core idea is to swap the manifestival manifest.Client at the start of the reconcile stage pipeline (ResolveTargetCluster), so that all subsequent stages — Apply, Delete, Get — transparently operate on the remote cluster.
For garbage collection on the remote cluster, an anchor ConfigMap pattern (inspired by k0smotron) is used:
  • An anchor ConfigMap ({kind}-{cr-name}-root-owner) is created on the remote cluster.
  • All namespace-scoped resources get an ownerReference pointing to this anchor, enabling Kubernetes-native GC.
  • Cluster-scoped resources (ClusterRole, etc.) have no ownerReference and are explicitly deleted by the finalizer.
  • Deleting the anchor ConfigMap cascades to all namespace-scoped resources via GC.
Example CR
KnativeServing on a remote cluster
apiVersion: operator.knative.dev/v1beta1kind: KnativeServingmetadata:
name: knative-serving-apacnamespace: knative-servingspec:
version: "1.21"clusterProfileRef:
name: apac-cluster-01namespace: fleet-systemconfig:
network:
ingress-class: "kourier.ingress.networking.knative.dev"ingress:
kourier:
enabled: true
KnativeEventing on a remote cluster
apiVersion: operator.knative.dev/v1beta1kind: KnativeEventingmetadata:
name: knative-eventing-apacnamespace: knative-eventingspec:
version: "1.21"clusterProfileRef:
name: apac-cluster-01namespace: fleet-system
Local cluster (unchanged behavior)
apiVersion: operator.knative.dev/v1beta1kind: KnativeServingmetadata:
name: knative-servingnamespace: knative-servingspec:
version: "1.21"# No clusterProfileRef → deploys to the local cluster as before
Verification Steps
E2E verification with Kind
Note: The credential plugin used below (kubeconfig-secretreader-plugin) is a community plugin.

An official plugin is being tracked at kubernetes-sigs/cluster-inventory-api#45 —

once released, switch to the official one.
1. Cluster setup
# Create Kind clusters
kind create cluster --name hub
kind create cluster --name spoke
# Install ClusterProfile CRD on hub
kubectl --context kind-hub apply -f https://raw.githubusercontent.com/kubernetes-sigs/cluster-inventory-api/main/config/crd/bases/multicluster.x-k8s.io_clusterprofiles.yaml
# Create namespaces on hub
kubectl --context kind-hub create namespace fleet-system
kubectl --context kind-hub create namespace knative-serving
kubectl --context kind-hub create namespace knative-operator
2. Spoke cluster credentials
# Create a ServiceAccount + token Secret on spoke for the operator to use
kubectl --context kind-spoke apply -f - <<EOFapiVersion: v1kind: ServiceAccountmetadata: name: knative-operator namespace: default---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRoleBindingmetadata: name: knative-operator-adminroleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cluster-adminsubjects:- kind: ServiceAccount name: knative-operator namespace: default---apiVersion: v1kind: Secretmetadata: name: knative-operator-token namespace: default annotations: kubernetes.io/service-account.name: knative-operatortype: kubernetes.io/service-account-tokenEOF# Wait for the token to be populatedforiin$(seq 1 30);do
TOKEN=$(kubectl --context kind-spoke get secret knative-operator-token -o go-template='{{.data.token | base64decode}}'2>/dev/null)&& [ -n"$TOKEN" ] &&break
sleep 1
done# Copy the token as a Secret on the hub cluster
kubectl --context kind-hub -n knative-operator create secret generic spoke-token \
--from-literal=token="${TOKEN}" --dry-run=client -o yaml | kubectl --context kind-hub apply -f -
3. ClusterProfile
# Get spoke cluster connection info (Docker-internal IP, not 127.0.0.1)
SPOKE_SERVER="https://$(docker inspect spoke-control-plane --format '{{ .NetworkSettings.Networks.kind.IPAddress }}'):6443"
SPOKE_CA=$(kubectl --context kind-spoke config view --minify --flatten -o jsonpath='{.clusters[0].cluster.certificate-authority-data}')# Create ClusterProfile (spec only — status is a subresource)
kubectl --context kind-hub apply -f - <<EOFapiVersion: multicluster.x-k8s.io/v1alpha1kind: ClusterProfilemetadata: name: spoke-cluster namespace: fleet-systemspec: clusterManager: name: kindEOF# Patch status with access provider info.# The extensions tell the credential plugin which Secret to read the token from.
STATUS_PATCH=$(cat <<EOF{ "status": { "accessProviders": [ { "name": "token-secretreader", "cluster": { "server": "${SPOKE_SERVER}", "certificate-authority-data": "${SPOKE_CA}", "extensions": [ { "name": "client.authentication.k8s.io/exec", "extension": { "secretName": "spoke-token", "secretNamespace": "knative-operator", "key": "token" } } ] } } ] }}EOF)
kubectl --context kind-hub patch clusterprofile spoke-cluster \
-n fleet-system --type merge --subresource=status -p "${STATUS_PATCH}"
4. Operator deployment
# Build and deploy the operator with ko.# KIND_CLUSTER_NAME ensures the image is loaded into the correct Kind cluster.
kubectl config use-context kind-hub
KO_DOCKER_REPO=kind.local KIND_CLUSTER_NAME=hub ko apply -f config/
kubectl --context kind-hub -n knative-operator \
wait --for=condition=Available deployment/knative-operator --timeout=120s
# Create clusterprofile-provider-file ConfigMap (provider name must match accessProviders[].name)
kubectl --context kind-hub -n knative-operator create configmap clusterprofile-provider-file \
--from-literal=config.json='{"providers":[{"name":"token-secretreader","execConfig":{"apiVersion":"client.authentication.k8s.io/v1","command":"/credential-plugin/kubeconfig-secretreader-plugin","provideClusterInfo":true}}]}'# Patch the operator deployment:# - Mount the credential plugin binary via image volume# - Mount the clusterprofile-provider-file config# - Add the --clusterprofile-provider-file flag
kubectl --context kind-hub -n knative-operator patch deployment knative-operator --type json -p '[ {"op":"add","path":"/spec/template/spec/containers/0/args","value":["--clusterprofile-provider-file=/etc/cluster-inventory/config.json"]}, {"op":"add","path":"/spec/template/spec/containers/0/volumeMounts","value":[ {"name":"cred-config","mountPath":"/etc/cluster-inventory","readOnly":true}, {"name":"credential-plugin","mountPath":"/credential-plugin","readOnly":true} ]}, {"op":"add","path":"/spec/template/spec/volumes","value":[ {"name":"cred-config","configMap":{"name":"clusterprofile-provider-file"}}, {"name":"credential-plugin","image":{"reference":"ghcr.io/labthrust/kubeconfig-secretreader-plugin:v0.0.1-linux-arm64"}} ]}]'
kubectl --context kind-hub -n knative-operator rollout status deployment/knative-operator --timeout=120s
5. Install Envoy Gateway on the spoke cluster
The net-gateway-api ingress provider requires Gateway API resources (GatewayClass, Gateway) to exist on the target cluster before deploying KnativeServing. Install Envoy Gateway and create the necessary resources on the spoke cluster.
Prerequisites
go install sigs.k8s.io/cloud-provider-kind@latest
Install Envoy Gateway
Since cloud-provider-kind installs Gateway API CRDs with server-side apply, helm install will fail with field ownership conflicts. Use helm template + kubectl apply --server-side --force-conflicts to take over ownership.
cat <<'EOF' > /tmp/values-eg.yamlconfig: envoyGateway: provider: type: Kubernetes kubernetes: deploy: type: GatewayNamespaceEOFexport ENVOY_GATEWAY_VERSION=v1.7.1
kubectl --context kind-spoke create namespace envoy-gateway-system
helm template eg oci://docker.io/envoyproxy/gateway-helm \
--version $ENVOY_GATEWAY_VERSION \
-n envoy-gateway-system \
-f /tmp/values-eg.yaml \
--include-crds | \
kubectl --context kind-spoke apply --server-side --force-conflicts -f -
After installing Envoy Gateway, start cloud-provider-kind in a separate terminal (provides LoadBalancer support for Kind):
sudo cloud-provider-kind
Create Gateway API resources (external)
kubectl --context kind-spoke apply -f - <<EOFapiVersion: v1kind: Namespacemetadata: name: eg-external---apiVersion: gateway.envoyproxy.io/v1alpha1kind: EnvoyProxymetadata: name: knative-external-config namespace: envoy-gateway-systemspec: provider: type: Kubernetes kubernetes: envoyService: name: knative-external---apiVersion: gateway.networking.k8s.io/v1kind: GatewayClassmetadata: name: eg-externalspec: controllerName: gateway.envoyproxy.io/gatewayclass-controller parametersRef: group: gateway.envoyproxy.io kind: EnvoyProxy name: knative-external-config namespace: envoy-gateway-system---apiVersion: gateway.networking.k8s.io/v1kind: Gatewaymetadata: name: eg-external namespace: eg-externalspec: gatewayClassName: eg-external listeners: - name: http port: 80 protocol: HTTP allowedRoutes: namespaces: from: All - name: tls port: 443 protocol: TLS tls: mode: Passthrough allowedRoutes: namespaces: from: AllEOF
Create Gateway API resources (internal)
The internal Gateway uses the ClusterIP service type, so it is not accessible from outside the cluster.
kubectl --context kind-spoke apply -f - <<EOFapiVersion: v1kind: Namespacemetadata: name: eg-internal---apiVersion: gateway.envoyproxy.io/v1alpha1kind: EnvoyProxymetadata: name: knative-internal-config namespace: envoy-gateway-systemspec: provider: type: Kubernetes kubernetes: envoyService: type: ClusterIP name: knative-internal---apiVersion: gateway.networking.k8s.io/v1kind: GatewayClassmetadata: name: eg-internalspec: controllerName: gateway.envoyproxy.io/gatewayclass-controller parametersRef: group: gateway.envoyproxy.io kind: EnvoyProxy name: knative-internal-config namespace: envoy-gateway-system---apiVersion: gateway.networking.k8s.io/v1kind: Gatewaymetadata: name: eg-internal namespace: eg-internalspec: gatewayClassName: eg-internal listeners: - name: http port: 80 protocol: HTTP allowedRoutes: namespaces: from: AllEOF
6. Deploy KnativeServing
kubectl --context kind-hub apply -f - <<EOFapiVersion: operator.knative.dev/v1beta1kind: KnativeServingmetadata: name: knative-serving namespace: knative-servingspec: clusterProfileRef: name: spoke-cluster namespace: fleet-system ingress: gateway-api: enabled: true config: config-gateway: external-gateways: | - class: eg-external gateway: eg-external/eg-external service: eg-external/knative-external supported-features: - HTTPRouteRequestTimeout local-gateways: | - class: eg-internal gateway: eg-internal/eg-internal service: eg-internal/knative-internal supported-features: - HTTPRouteRequestTimeout network: ingress.class: gateway-api.ingress.networking.knative.dev domain: example.com: ""EOF
7. Verify
# Verify resources deployed to spoke cluster
kubectl --context kind-spoke get deployments -n knative-serving
kubectl --context kind-spoke get configmaps -n knative-serving | grep root-owner
# Verify anchor ConfigMap exists with correct labels
kubectl --context kind-spoke get configmap knativeserving-knative-serving-root-owner \
-n knative-serving -o yaml
# Verify ownerReferences on namespace-scoped resources point to the anchor
kubectl --context kind-spoke get deployment activator -n knative-serving \
-o jsonpath='{.metadata.ownerReferences}'| jq .# Deploy a sample application to verify traffic routing
kubectl --context kind-spoke apply -f - <<EOFapiVersion: serving.knative.dev/v1kind: Servicemetadata: name: helloworld-go namespace: defaultspec: template: spec: containers: - image: gcr.io/knative-samples/helloworld-go env: - name: TARGET value: Go Sample v1EOF# Get the external gateway IP and testexport LB_IP=$(kubectl --context kind-spoke -n eg-external get svc knative-external \ -o jsonpath='{.status.loadBalancer.ingress[0].ip}')echo"$LB_IP helloworld-go.default.example.com"| sudo tee -a /etc/hosts
curl http://helloworld-go.default.example.com

8. Verify CR deletion and finalizer cleanup
# Delete the CR and wait for the finalizer to complete
kubectl --context kind-hub delete knativeserving knative-serving -n knative-serving
# Verify spoke cluster is clean
kubectl --context kind-spoke get all -n knative-serving
kubectl --context kind-spoke get clusterroles | grep knative
9. Cleanup
kind delete cluster --name hub
kind delete cluster --name spoke
Backward compatibility
# Deploy without --clusterprofile-provider-file and without clusterProfileRef# → must behave identically to previous releases
kubectl apply -f - <<EOFapiVersion: operator.knative.dev/v1beta1kind: KnativeServingmetadata: name: knative-serving namespace: knative-servingspec: version: "1.21"EOF
kubectl get knativeserving -n knative-serving -o jsonpath='{.status.conditions}'
Release Note
Add multi-cluster deployment support via the Cluster Inventory API (KEP-5339). When `spec.clusterProfileRef` is set on a KnativeServing or KnativeEventing CR, the operator deploys components to the remote cluster described by the referenced ClusterProfile. Requires the `--clusterprofile-provider-file` flag to be set on the operator deployment.

@knative-prowknative-prowBot
added
the 
size/XXL
Denotes a PR that changes 1000+ lines, ignoring generated files.
label
Apr 6, 2026
knative-prow[bot]
knative-prowBot
reviewed
Apr 6, 2026
View reviewed changes
@knative-prowknative-prowBot
left a comment


Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kahirokunn: 0 warnings.
Details
In response to this:
Fixes #
Proposed Changes
Release Note
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.
@codecov
codecovBot
commented
Apr 6, 2026•

edited

Loading

Copy link


Copy Markdown

Codecov Report
❌ Patch coverage is 60.40000% with 198 lines in your changes missing coverage. Please review.

✅ Project coverage is 63.84%. Comparing base (9c66d7b) to head (72fc85c).
⚠️ Report is 1 commits behind head on main.
Files with missing linesPatch %Lines
pkg/reconciler/common/multicluster.go62.18%121 Missing and 14 partials ⚠️
pkg/reconciler/knativeeventing/controller.go0.00%25 Missing ⚠️
pkg/reconciler/knativeserving/controller.go0.00%25 Missing ⚠️
pkg/reconciler/common/clusterprofile_informer.go86.53%3 Missing and 4 partials ⚠️
pkg/apis/operator/base/common.go0.00%2 Missing ⚠️
pkg/reconciler/common/access_provider_flag.go66.66%2 Missing ⚠️
pkg/reconciler/knativeeventing/eventing_tls.go0.00%2 Missing ⚠️
Additional details and impacted files
@@ Coverage Diff @@## main #2267 +/- ##
==========================================
- Coverage 64.58% 63.84% -0.75% 
==========================================
Files 51 55 +4 Lines 1999 2478 +479 ==========================================
+ Hits 1291 1582 +291 - Misses 606 777 +171 - Partials 102 119 +17 
☔ View full report in Codecov by Sentry.

📢 Have feedback on the report? Share it here.
 🚀 New features to boost your workflow: 
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
@kahirokunnkahirokunn
changed the title
Add multi-cluster deployment support via Cluster Inventory API[WIP] Add multi-cluster deployment support via Cluster Inventory APIApr 6, 2026
@knative-prowknative-prowBot
added
the 
do-not-merge/work-in-progress
Indicates that a PR should not merge because it is a work in progress.
label
Apr 6, 2026
@kahirokunn
kahirokunnforce-pushed
the
multi-cluster-support
branch
5 times, most recently
from
3e585a9 to
141c875CompareApril 7, 2026 06:40
@knative-prow-robotknative-prow-robot
added
the 
needs-rebase
Indicates a PR cannot be merged because it has merge conflicts with HEAD.
label
Apr 7, 2026
@kahirokunn
kahirokunnforce-pushed
the
multi-cluster-support
branch
2 times, most recently
from
d308bf0 to
3e742cbCompareApril 7, 2026 12:53
@kahirokunnkahirokunn
changed the title
[WIP] Add multi-cluster deployment support via Cluster Inventory APIAdd multi-cluster deployment support via Cluster Inventory APIApr 7, 2026
@knative-prowknative-prowBot
removed
the 
do-not-merge/work-in-progress
Indicates that a PR should not merge because it is a work in progress.
label
Apr 7, 2026
@kahirokunn
kahirokunnforce-pushed
the
multi-cluster-support
branch
from
3e742cb to
43919c6CompareApril 7, 2026 14:11
@knative-prow-robotknative-prow-robot
removed
the 
needs-rebase
Indicates a PR cannot be merged because it has merge conflicts with HEAD.
label
Apr 7, 2026
@kahirokunn
kahirokunnforce-pushed
the
multi-cluster-support
branch
from
43919c6 to
28a1e02CompareApril 7, 2026 14:22
@knative-prow-robotknative-prow-robot
added
the 
needs-rebase
Indicates a PR cannot be merged because it has merge conflicts with HEAD.
label
Apr 7, 2026
@kahirokunn
kahirokunnforce-pushed
the
multi-cluster-support
branch
from
28a1e02 to
88410efCompareApril 7, 2026 14:46
@knative-prow
knative-prowBot
commented
Apr 7, 2026

Copy link


Copy Markdown

There are empty aliases in OWNER_ALIASES, cleanup is advised.
@knative-prow-robotknative-prow-robot
removed
the 
needs-rebase
Indicates a PR cannot be merged because it has merge conflicts with HEAD.
label
Apr 7, 2026
@kahirokunnkahirokunn
changed the title
Add multi-cluster deployment support via Cluster Inventory API[WIP] Add multi-cluster deployment support via Cluster Inventory APIApr 7, 2026
@knative-prowknative-prowBot
added
the 
do-not-merge/work-in-progress
Indicates that a PR should not merge because it is a work in progress.
label
Apr 7, 2026
@knative-prow
knative-prowBot
commented
Apr 7, 2026

Copy link


Copy Markdown

There are empty aliases in OWNER_ALIASES, cleanup is advised.
@kahirokunn
kahirokunnforce-pushed
the
multi-cluster-support
branch
3 times, most recently
from
29a9317 to
6217b27CompareApril 8, 2026 04:05
@knative-prow-robotknative-prow-robot
added
the 
needs-rebase
Indicates a PR cannot be merged because it has merge conflicts with HEAD.
label
Apr 16, 2026
@kahirokunn
kahirokunnforce-pushed
the
multi-cluster-support
branch
2 times, most recently
from
9068c06 to
cecc1b8CompareApril 16, 2026 05:54
@knative-prow-robotknative-prow-robot
removed
the 
needs-rebase
Indicates a PR cannot be merged because it has merge conflicts with HEAD.
label
Apr 16, 2026
@knative-prow
knative-prowBot
commented
Apr 16, 2026

Copy link


Copy Markdown

There are empty aliases in OWNER_ALIASES, cleanup is advised.
@kahirokunnkahirokunn
changed the title
[WIP] Add multi-cluster deployment support via Cluster Inventory APIAdd multi-cluster deployment support via Cluster Inventory APIApr 16, 2026
@knative-prowknative-prowBot
removed
the 
do-not-merge/work-in-progress
Indicates that a PR should not merge because it is a work in progress.
label
Apr 16, 2026
@kahirokunn
kahirokunnforce-pushed
the
multi-cluster-support
branch
from
cecc1b8 to
725b1beCompareApril 16, 2026 08:34
@kahirokunn
kahirokunn
commented
Apr 17, 2026

Copy link


Copy Markdown

MemberAuthor
I didn't have the necessary permissions.
integration-tests-multicluster_operator_main will keep failing until the following is merged, so I'd like to ignore it using an override until then #2267
@kahirokunn
kahirokunn
commented
Apr 17, 2026

Copy link


Copy Markdown

MemberAuthor
/retest
houshengbo
houshengbo
reviewed
Apr 17, 2026
View reviewed changes
@houshengbohoushengbo
left a comment
•

edited

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review
The PR description is excellent — the verification steps, design explanation, example CRs, and backward compatibility section are thorough and well-written.
2 blocking items and 7 non-blocking items are noted in inline comments below.
Comment threadpkg/apis/operator/v1beta1/knativeserving_lifecycle.go
Outdated

Comment threadpkg/reconciler/common/clusterprofile_informer.go
Comment threadpkg/reconciler/common/multicluster.go
Comment threadpkg/reconciler/common/multicluster.go
Comment threadpkg/reconciler/common/multicluster.go
Comment threadpkg/reconciler/common/transformers.go
Comment threadpkg/reconciler/common/stages.go
Outdated

Comment threadtest/e2e/knativeserving_spoke_test.go
@kahirokunn
kahirokunn
commented
Apr 21, 2026

Copy link


Copy Markdown

MemberAuthor
@houshengbo Thank you for your feedback! I have addressed all the points you raised.
@kahirokunn
kahirokunn
commented
Apr 21, 2026

Copy link


Copy Markdown

MemberAuthor
I will check it
@kahirokunn
add multi-cluster deployment support via Cluster Inventory API
4845dee
Signed-off-by: kahirokunn <okinakahiro@gmail.com>
@kahirokunn
change MarkTargetClusterNotResolved to mark only TargetClusterResolved
d1e2351
Signed-off-by: kahirokunn <okinakahiro@gmail.com>
@kahirokunn
add doRefresh error path tests for ClusterProvider
c3c500f
Signed-off-by: kahirokunn <okinakahiro@gmail.com>
@kahirokunn
mark anchor ConfigMap as protected to surface deletion risk
6b7c4e0
Signed-off-by: kahirokunn <okinakahiro@gmail.com>
@kahirokunn
stabilise ClusterProvider shutdown and propagate namespace config
4c2c9b4
Signed-off-by: kahirokunn <okinakahiro@gmail.com>
@kahirokunn
add --remote-deployments-poll-interval CLI flag
5093973
Signed-off-by: kahirokunn <okinakahiro@gmail.com>
@kahirokunn
add multi-cluster e2e test for KnativeEventing
b901f31
Signed-off-by: kahirokunn <okinakahiro@gmail.com>
@kahirokunn
./hack/update-codegen.sh
72fc85c
@kahirokunn
kahirokunn
commented
Apr 21, 2026

Copy link


Copy Markdown

MemberAuthor
All CI passed! 🙌
@houshengbo
houshengbo
commented
Apr 22, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
/lgtm

/approve
@knative-prow
knative-prowBot
commented
Apr 22, 2026

Copy link


Copy Markdown

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: houshengbo, kahirokunn
The full list of commands accepted by this bot can be found here.
The pull request process is described here
Details
Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment

Approvers can cancel approval by writing /approve cancel in a comment
@kahirokunn
kahirokunn
commented
Apr 22, 2026

Copy link


Copy Markdown

MemberAuthor
Thank you😆
@kahirokunnkahirokunn
mentioned this pull request
Apr 23, 2026
 Merged

@kahirokunnkahirokunn
mentioned this pull request
May 26, 2026

Closed


This was referenced Aug 22, 2026
 Closed

 Closed


Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@knative-prowknative-prow[bot]knative-prow[bot] left review comments
@matzewmatzewAwaiting requested review from matzew
@houshengbohoushengboAwaiting requested review from houshengbo

Assignees

@houshengbohoushengbo

Labels

approvedIndicates a PR has been approved by an approver from all required OWNERS files.lgtmIndicates that a PR is ready to be merged.size/XXLDenotes a PR that changes 1000+ lines, ignoring generated files.

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

Support multi-cluster deployment via Cluster Inventory API


3 participants

@kahirokunn@houshengbo@knative-prow-robot