Enable dynamic scaling of Kubernetes cluster offerings for KVM by bernardodemarco · Pull Request #13226 · apache/cloudstack

bernardodemarco · 2026-05-22T22:07:50Z

Description

PR #12975 introduced support for live scaling VMs with fixed service offerings on KVM. However, it did not remove a constraint in the Kubernetes cluster scaling workflow that prevented changing cluster compute offerings while the clusters were running.

This PR removes this leftover constraint, enabling dynamic scaling of Kubernetes cluster offerings.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
Build/CI
Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Screenshots (if appropriate):

How Has This Been Tested?

Created a k8s cluster with a compute offering of 2 vCPUs and 4096 MiB of memory (without CPU cap):
k8s cluster details

Scaled the k8s cluster and upgraded its offering to 2 vCPUs and 4096 MiB of memory (with CPU cap):

k8s cluster control plane VM XML

<maxMemoryslots='16'unit='KiB'>15327232</maxMemory>
<memoryunit='KiB'>4194304</memory>
<currentMemoryunit='KiB'>4194304</currentMemory>
<vcpuplacement='static'current='2'>8</vcpu>
<cputune>
<shares>835</shares>
<period>10000</period>
<quota>3300</quota>
</cputune>

Scaled the worker nodes compute offering to 3 vCPUs and 4096 MiB of memory (with CPU cap):

k8s cluster details

k8s cluster worker plane VM XML

<maxMemoryslots='16'unit='KiB'>15327232</maxMemory>
<memoryunit='KiB'>4194304</memory>
<currentMemoryunit='KiB'>4194304</currentMemory>
<vcpuplacement='static'current='3'>8</vcpu>
<vcpus>
<vcpuid='0'enabled='yes'hotpluggable='no'order='1'/>
<vcpuid='1'enabled='yes'hotpluggable='no'order='2'/>
<vcpuid='2'enabled='yes'hotpluggable='yes'order='3'/>
<vcpuid='3'enabled='no'hotpluggable='yes'/>
<vcpuid='4'enabled='no'hotpluggable='yes'/>
<vcpuid='5'enabled='no'hotpluggable='yes'/>
<vcpuid='6'enabled='no'hotpluggable='yes'/>
<vcpuid='7'enabled='no'hotpluggable='yes'/>
</vcpus>

codecov · 2026-05-22T22:17:22Z

Codecov Report

❌ Patch coverage is 0% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 18.10%. Comparing base (21b2025) to head (55ba389).
⚠️ Report is 206 commits behind head on main.

Files with missing lines	Patch %	Lines
...er/actionworkers/KubernetesClusterScaleWorker.java	0.00%	1 Missing ⚠️

Additional details and impacted files

@@ Coverage Diff @@## main #13226 +/- ##
============================================
+ Coverage 17.67% 18.10% +0.42% - Complexity 15792 16749 +957 
============================================
Files 5922 6037 +115 Lines 533165 542924 +9759 Branches 65208 66482 +1274 ============================================
+ Hits 94242 98292 +4050 - Misses 428276 433584 +5308 - Partials 10647 11048 +401

Flag	Coverage Δ
uitests	`3.51% <ø> (-0.18%)`	⬇️
unittests	`19.26% <0.00%> (+0.51%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

winterhazel · 2026-05-22T23:52:26Z

@blueorangutan package

blueorangutan · 2026-05-22T23:54:05Z

@winterhazel a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2026-05-22T23:54:58Z

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 17998

bernardodemarco · 2026-05-23T22:20:06Z

@blueorangutan package

blueorangutan · 2026-05-23T22:22:04Z

@bernardodemarco a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2026-05-23T23:07:35Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 18002

winterhazel · 2026-06-01T18:40:10Z

@DaanHoogland @weizhouapache can we run the CI here?

winterhazel

Code looks good, did not test.

DaanHoogland · 2026-06-14T17:29:43Z

@blueorangutan test

blueorangutan · 2026-06-14T17:32:03Z

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2026-06-15T06:07:26Z

[SF] Trillian test result (tid-16307)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 42446 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr13226-t16307-kvm-ol8.zip
Smoke tests completed. 141 look OK, 0 have errors, 10 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
all_test_volumes	`Skipped`	---	test_volumes.py
all_test_vpc_conserve_mode	`Skipped`	---	test_vpc_conserve_mode.py
all_test_vpc_ipv6	`Skipped`	---	test_vpc_ipv6.py
all_test_vpc_redundant	`Skipped`	---	test_vpc_redundant.py
all_test_vpc_router_nics	`Skipped`	---	test_vpc_router_nics.py
all_test_vpc_vpn	`Skipped`	---	test_vpc_vpn.py
all_test_webhook_delivery	`Skipped`	---	test_webhook_delivery.py
all_test_webhook_lifecycle	`Skipped`	---	test_webhook_lifecycle.py
all_test_host_maintenance	`Skipped`	---	test_host_maintenance.py
all_test_hostha_kvm	`Skipped`	---	test_hostha_kvm.py

JoaoJandre

CLGTM

winterhazel · 2026-06-15T12:01:43Z

The test failures seem unrelated. Can we attempt another run? @DaanHoogland

DaanHoogland · 2026-06-15T14:07:31Z

@blueorangutan package

blueorangutan · 2026-06-15T14:08:04Z

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2026-06-15T15:01:39Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 18265

DaanHoogland · 2026-06-17T18:04:06Z

@blueorangutan test

blueorangutan · 2026-06-17T18:06:08Z

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2026-06-17T18:50:47Z

[SF] Trillian Build Failed (tid-16348)

blueorangutan · 2026-06-19T03:43:32Z

[SF] Trillian test result (tid-16365)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 56072 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr13226-t16365-kvm-ol8.zip
Smoke tests completed. 149 look OK, 2 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_vpn_usage	`Error`	1.14	test_usage.py
test_01_redundant_vpc_site2site_vpn	`Failure`	380.86	test_vpc_vpn.py

weizhouapache · 2026-06-30T12:11:45Z

@bernardodemarco
is there any restrictions ?
for example, does it work with CKS clusters created in older ACS versions ?

bernardodemarco · 2026-06-30T13:34:34Z

Hello, @weizhouapache

is there any restrictions ?
for example, does it work with CKS clusters created in older ACS versions ?

No, there aren't any restrictions beyond the restrictions of the new live scaling workflow for KVM, see #12975 and #12908.

for example, does it work with CKS clusters created in older ACS versions

As explained with more details in #12908, the live scaling workflow for ACS requires the domains to be prepared for live scaling. The third section of the specification has more details on how to enable live scaling for already existing VMs prior to the upgrade to the new ACS release.

weizhouapache · 2026-06-30T13:56:10Z

Hello, @weizhouapache
is there any restrictions ?
for example, does it work with CKS clusters created in older ACS versions ?
No, there aren't any restrictions beyond the restrictions of the new live scaling workflow for KVM, see #12975 and #12908.
for example, does it work with CKS clusters created in older ACS versions
As explained with more details in #12908, the live scaling workflow for ACS requires the domains to be prepared for live scaling. The third section of the specification has more details on how to enable live scaling for already existing VMs prior to the upgrade to the new ACS release.

thanks @bernardodemarco

so, as I understand, the answer is no, right ?

bernardodemarco · 2026-06-30T14:04:21Z

so, as I understand, the answer is no, right ?

Yes, generally speaking we do not have restrictions.

I'll ask someone to test this one so that we can merge it by friday.

GeanJair

LGTM. Performed the following tests and it worked as intended.

Created a k8s cluster with a compute offering of 2 vCPUs and 4096 MiB of memory (Without CPU cap)

VM XML Without CPU cap

<maxMemory slots='16' unit='KiB'>8891392</maxMemory>
<memory unit='KiB'>4194304</memory>
<currentMemory unit='KiB'>4194304</currentMemory>
<vcpu placement='static' current='2'>6</vcpu>
<cputune>
<shares>2565</shares>
</cputune>

Scaled the k8s cluster and upgraded its offering to 2 vCPUs and 4096 MiB of memory (With CPU cap)

VM XML With CPU cap

<maxMemory slots='16' unit='KiB'>8891392</maxMemory>
<memory unit='KiB'>4194304</memory>
<currentMemory unit='KiB'>4194304</currentMemory>
<vcpu placement='static' current='2'>6</vcpu>
<cputune>
<shares>2565</shares>
<period>10000</period>
<quota>7700</quota>
</cputune>

After that, scaled the worker nodes compute offering to 3 vCPUs and 4096 MiB of memory (With CPU cap)

Cluster Details

worker plane VM XML

<maxMemory slots='16' unit='KiB'>8891392</maxMemory>
<memory unit='KiB'>4194304</memory>
<currentMemory unit='KiB'>4194304</currentMemory>
<vcpu placement='static' current='3'>6</vcpu>
<vcpus>
<vcpu id='0' enabled='yes' hotpluggable='no' order='1'/>
<vcpu id='1' enabled='yes' hotpluggable='no' order='2'/>
<vcpu id='2' enabled='yes' hotpluggable='yes' order='3'/>
<vcpu id='3' enabled='no' hotpluggable='yes'/>
<vcpu id='4' enabled='no' hotpluggable='yes'/>
<vcpu id='5' enabled='no' hotpluggable='yes'/>
</vcpus>

bernardodemarco · 2026-07-01T20:07:31Z

@GeanJS, thanks for testing!

weizhouapache · 2026-07-01T20:59:46Z

LGTM. Performed the following tests and it worked as intended.

Created a k8s cluster with a compute offering of 2 vCPUs and 4096 MiB of memory (Without CPU cap)

VM XML Without CPU cap

<maxMemory slots='16' unit='KiB'>8891392</maxMemory>
<memory unit='KiB'>4194304</memory>
<currentMemory unit='KiB'>4194304</currentMemory>
<vcpu placement='static' current='2'>6</vcpu>
<cputune>
<shares>2565</shares>
</cputune>

Scaled the k8s cluster and upgraded its offering to 2 vCPUs and 4096 MiB of memory (With CPU cap)

VM XML With CPU cap

<maxMemory slots='16' unit='KiB'>8891392</maxMemory>
<memory unit='KiB'>4194304</memory>
<currentMemory unit='KiB'>4194304</currentMemory>
<vcpu placement='static' current='2'>6</vcpu>
<cputune>
<shares>2565</shares>
<period>10000</period>
<quota>7700</quota>
</cputune>

After that, scaled the worker nodes compute offering to 3 vCPUs and 4096 MiB of memory (With CPU cap)

Cluster Details

worker plane VM XML

<maxMemory slots='16' unit='KiB'>8891392</maxMemory>
<memory unit='KiB'>4194304</memory>
<currentMemory unit='KiB'>4194304</currentMemory>
<vcpu placement='static' current='3'>6</vcpu>
<vcpus>
<vcpu id='0' enabled='yes' hotpluggable='no' order='1'/>
<vcpu id='1' enabled='yes' hotpluggable='no' order='2'/>
<vcpu id='2' enabled='yes' hotpluggable='yes' order='3'/>
<vcpu id='3' enabled='no' hotpluggable='yes'/>
<vcpu id='4' enabled='no' hotpluggable='yes'/>
<vcpu id='5' enabled='no' hotpluggable='yes'/>
</vcpus>

Thanks.

Can you disable dynamic scaling and create a CKS cluster, and then

scale cks cluster
enable dynamic scaling and scale cks cluster ?

GeanJair · 2026-07-03T14:28:46Z

Hello, @weizhouapache

When scaling k8s clusters with the dynamic scaling setting disabled, the following error message is returned, regardless of the hypervisor in use:

Error message returned when dynamic scaling is disabled

When a k8s cluster is created and the dynamic scaling is disabled, the metadata of the k8s cluster's VMs will indicate they are not dynamically scalable. As a consequence of that, regardless of the hypervisor, after enabling dynamic scaling, it will be required to mark the instances as dynamically scalable as well. Specially for KVM, there is another required step, that consists in stopping and starting the VMs so that their domains are recreated with the required structure for live scaling (see #12908 for more details). The image below demonstrates the error message returned when a scaling operation is attempted for the k8s cluster, but the domains are not prepared for it.

Error message returned when trying to scale a k8s cluster with unprepared KVM domains

remove constraint from scaling k8s workflow
55ba389

bernardodemarco added this to the 4.23.0 milestone May 22, 2026

boring-cyborgBot added the component:kubernetes label May 22, 2026

bernardodemarco mentioned this pull request May 22, 2026
Unable to Autoscale K8s in CS 4.21.0.0 #11994
Open

winterhazel added status:needs-testing status:needs-review labels May 22, 2026

DaanHoogland requested a review from kiranchavala May 26, 2026 08:17

DaanHoogland assigned kiranchavala May 26, 2026

winterhazel approved these changes Jun 1, 2026
View reviewed changes

github-actionsBot mentioned this pull request Jun 9, 2026
[repo-status] Daily Status Report – June 9, 2026 #13386
Closed

github-actionsBot mentioned this pull request Jun 14, 2026
[repo-status] Daily Status Report — June 14, 2026 #13418
Closed

JoaoJandre approved these changes Jun 15, 2026
View reviewed changes

github-actionsBot mentioned this pull request Jun 15, 2026
[repo-status] Daily Status Report — June 15, 2026 #13425
Closed

bernardodemarco removed the status:needs-review label Jun 22, 2026

GeanJair approved these changes Jul 1, 2026
View reviewed changes

winterhazel merged commit 5c4bc48 into apache:mainJul 1, 2026
26 of 27 checks passed

Uh oh!

Conversation

bernardodemarco commented May 22, 2026

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Screenshots (if appropriate):

How Has This Been Tested?

Uh oh!

codecovBot commented May 22, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

winterhazel commented May 22, 2026

Uh oh!

blueorangutan commented May 22, 2026

Uh oh!

blueorangutan commented May 22, 2026

Uh oh!

bernardodemarco commented May 23, 2026

Uh oh!

blueorangutan commented May 23, 2026

Uh oh!

blueorangutan commented May 23, 2026

Uh oh!

winterhazel commented Jun 1, 2026

Uh oh!

winterhazel left a comment

Choose a reason for hiding this comment

Uh oh!

DaanHoogland commented Jun 14, 2026

Uh oh!

blueorangutan commented Jun 14, 2026

Uh oh!

blueorangutan commented Jun 15, 2026

Uh oh!

JoaoJandre left a comment

Choose a reason for hiding this comment

Uh oh!

winterhazel commented Jun 15, 2026

Uh oh!

DaanHoogland commented Jun 15, 2026

Uh oh!

blueorangutan commented Jun 15, 2026

Uh oh!

blueorangutan commented Jun 15, 2026

Uh oh!

DaanHoogland commented Jun 17, 2026

Uh oh!

blueorangutan commented Jun 17, 2026

Uh oh!

blueorangutan commented Jun 17, 2026

Uh oh!

blueorangutan commented Jun 19, 2026

Uh oh!

weizhouapache commented Jun 30, 2026

Uh oh!

bernardodemarco commented Jun 30, 2026

Uh oh!

weizhouapache commented Jun 30, 2026

Uh oh!

bernardodemarco commented Jun 30, 2026

Uh oh!

GeanJair left a comment

Choose a reason for hiding this comment

Uh oh!

bernardodemarco commented Jul 1, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

weizhouapache commented Jul 1, 2026

Uh oh!

GeanJair commented Jul 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

codecovBot commented May 22, 2026•
edited
Loading

bernardodemarco commented Jul 1, 2026•
edited
Loading