Cloudstack kubernetes Provider is not deployed in a shared network and routed mode isolated/vpc network

[kiranchavala]

problem

Cloudstack kubernetes Provider is not deployed in a shared network and routed mode isolated/vpc network

versions

Cloudstack 4.20
KVM

The steps to reproduce the bug

Steps to reproduce the issue

1. Create a isolated network or vpc network tier with routed mode or a shared network
2. Register the CKS iso
3. Launch a cks cluster in the isolated network or a vpc network tier
4. CKS cluster deploys successfully
5. Check the pods in the cks cluster

root@test-normal-control-196856da5b1:/var/log# kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-7c65d6cfc9-fmj7g 1/1 Running 0 44m
kube-system coredns-7c65d6cfc9-pzl99 1/1 Running 0 44m
kube-system etcd-test-normal-control-196856da5b1 1/1 Running 0 44m
kube-system kube-apiserver-test-normal-control-196856da5b1 1/1 Running 0 44m
kube-system kube-controller-manager-test-normal-control-196856da5b1 1/1 Running 0 44m
kube-system kube-proxy-9hgx8 1/1 Running 0 44m
kube-system kube-proxy-f9hg6 1/1 Running 0 44m
kube-system kube-scheduler-test-normal-control-196856da5b1 1/1 Running 0 44m
kube-system weave-net-pgqkf 2/2 Running 0 44m
kube-system weave-net-r2kjq 2/2 Running 1 (44m ago) 44m
kubernetes-dashboard dashboard-metrics-scraper-6b96ff7878-tqdkv 1/1 Running 0 

6. There is no Cloudstack kubernetes pod in the cluster

What to do about it?

Expected behaviour

Cloudstack should deploy the cloudstack-kubernetes-provider in the routed mode network or a shared network

Or we should document that cloudstack-kubernetes-provider is not supported in a routed mode network

[shwstppr]

This is explicitly done here, https://github.com/apache/cloudstack/blob/main/plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterActionWorker.java#L657-L661

Not sure even if the provider gets deployed on such networks whether it will work as expected cc @weizhouapache

[weizhouapache]

@shwstppr
thanks for checking

the original code was introduced by #5254, the provider is deployed on isolated network only.
I extended to not deploy on routed network/vpc in #9470

I do not know why the check was added there, but I think it is not a bug.
it is expected that the provider is not deployed on shared network or routed network.

[kiranchavala]

Thanks @shwstppr @weizhouapache for checking this,

I was able to deploy the cloudstack-kubernetes-provider in k8s cluster deployed by CAPC

but when acquiring a public IP its was throwing the following error

If necessary, I will log a improvement issue

I0506 11:00:26.188865 1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"nginx-deployment", UID:"cbb4fae5-3eae-4c50-a328-59c23ad0f4e0", APIVersion:"v1", ResourceVersion:"2494", FieldPath:""}): type: 'Normal' reason: 'EnsuringLoadBalancer' Ensuring load balancer
E0506 11:00:28.748580 1 controller.go:244] error processing service default/nginx-deployment (will retry): failed to ensure load balancer: error creating load balancer rule acbb4fae53eae4c50a32859c23ad0f4e-tcp-80: CloudStack API error 431 (CSExceptionErrorCode: 9999): In zone of type Advanced ip address can be associated only to the network of guest type Isolated with the SourceNat enabled
I0506 11:00:28.748789 1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"nginx-deployment", UID:"cbb4fae5-3eae-4c50-a328-59c23ad0f4e0", APIVersion:"v1", ResourceVersion:"2494", FieldPath:""}): type: 'Warning' reason: 'SyncLoadBalancerFailed' Error syncing load balancer: failed to ensure load balancer: error creating load balancer rule acbb4fae53eae4c50a32859c23ad0f4e-tcp-80: CloudStack API error 431 (CSExceptionErrorCode: 9999): In zone of type Advanced ip address can be associated only to the network of guest type Isolated with the SourceNat enabled
I0506 11:01:08.750166 1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"nginx-deployment", UID:"cbb4fae5-3eae-4c50-a328-59c23ad0f4e0", APIVersion:"v1", ResourceVersion:"2494", FieldPath:""}): type: 'Normal' reason: 'EnsuringLoadBalancer' Ensuring load balancer
E0506 11:01:11.147828 1 controller.go:244] error processing service default/nginx-deployment (will retry): failed to ensure load balancer: error creating load balancer rule acbb4fae53eae4c50a32859c23ad0f4e-tcp-80: CloudStack API error 431 (CSExceptionErrorCode: 9999): In zone of type Advanced ip address can be associated only to the network of guest type Isolated with the SourceNat enabled

[weizhouapache]

Thanks @shwstppr @weizhouapache for checking this,

I was able to deploy the cloudstack-kubernetes-provider in k8s cluster deployed by CAPC

but when acquiring a public IP its was throwing the following error

If necessary, I will log a improvement issue

I0506 11:00:26.188865 1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"nginx-deployment", UID:"cbb4fae5-3eae-4c50-a328-59c23ad0f4e0", APIVersion:"v1", ResourceVersion:"2494", FieldPath:""}): type: 'Normal' reason: 'EnsuringLoadBalancer' Ensuring load balancer
E0506 11:00:28.748580 1 controller.go:244] error processing service default/nginx-deployment (will retry): failed to ensure load balancer: error creating load balancer rule acbb4fae53eae4c50a32859c23ad0f4e-tcp-80: CloudStack API error 431 (CSExceptionErrorCode: 9999): In zone of type Advanced ip address can be associated only to the network of guest type Isolated with the SourceNat enabled
I0506 11:00:28.748789 1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"nginx-deployment", UID:"cbb4fae5-3eae-4c50-a328-59c23ad0f4e0", APIVersion:"v1", ResourceVersion:"2494", FieldPath:""}): type: 'Warning' reason: 'SyncLoadBalancerFailed' Error syncing load balancer: failed to ensure load balancer: error creating load balancer rule acbb4fae53eae4c50a32859c23ad0f4e-tcp-80: CloudStack API error 431 (CSExceptionErrorCode: 9999): In zone of type Advanced ip address can be associated only to the network of guest type Isolated with the SourceNat enabled
I0506 11:01:08.750166 1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"nginx-deployment", UID:"cbb4fae5-3eae-4c50-a328-59c23ad0f4e0", APIVersion:"v1", ResourceVersion:"2494", FieldPath:""}): type: 'Normal' reason: 'EnsuringLoadBalancer' Ensuring load balancer
E0506 11:01:11.147828 1 controller.go:244] error processing service default/nginx-deployment (will retry): failed to ensure load balancer: error creating load balancer rule acbb4fae53eae4c50a32859c23ad0f4e-tcp-80: CloudStack API error 431 (CSExceptionErrorCode: 9999): In zone of type Advanced ip address can be associated only to the network of guest type Isolated with the SourceNat enabled

@kiranchavala
Load balancer is not supported in Shared networks and Routed networks.

it might be the reason why the provider is not deployed by default.