Add advance settings to fine tune DRS imbalance calculation by vishesh92 · Pull Request #8521 · apache/cloudstack

vishesh92 · 2024-01-17T06:18:32Z

Description

Doc PR: apache/cloudstack-documentation#374

This PR addresses two issues:

Allow DRS to work as expected in clusters with hosts of different capacities - We do this by using ratio of free/total metric. This is controlled using a new setting drs.metric.use.ratio.
Imbalance can be greater than 1. For condensed algorithm, migrations don't take place when this happens. - This happens when the distribution of metrics is skewed. We fix this by filtering out values where ratio is less than drs.imbalance.condensed.skip.threshold if drs.metric.type is free and greater than drs.imbalance.condensed.skip.threshold if drs.metric.type is used to remove the skew and check if we need drs or not.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
build/CI

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

vishesh92 · 2024-01-17T06:18:57Z

@blueorangutan package

blueorangutan · 2024-01-17T06:20:03Z

@vishesh92 a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

codecov · 2024-01-17T06:25:57Z

Codecov Report

Attention: 15 lines in your changes are missing coverage. Please review.

Comparison is base (b0ac787) 30.80% compared to head (b917ecf) 30.94%.
Report is 4 commits behind head on 4.19.

Files	Patch %	Lines
...apache/cloudstack/cluster/ClusterDrsAlgorithm.java	86.44%	3 Missing and 5 partials ⚠️
.../apache/cloudstack/metrics/MetricsServiceImpl.java	76.92%	2 Missing and 1 partial ⚠️
...he/cloudstack/response/ClusterMetricsResponse.java	50.00%	1 Missing and 1 partial ⚠️
ui/src/config/section/infra/clusters.js	0.00%	1 Missing ⚠️
ui/src/views/infra/ClusterDRSTab.vue	0.00%	1 Missing ⚠️

Additional details and impacted files

@@ Coverage Diff @@## 4.19 #8521 +/- ##
============================================
+ Coverage 30.80% 30.94% +0.14% - Complexity 34043 34220 +177 
============================================
Files 5346 5346 Lines 375421 375496 +75 Branches 54598 54607 +9 ============================================
+ Hits 115647 116215 +568 + Misses 244489 243993 -496 - Partials 15285 15288 +3

Flag	Coverage Δ
simulator-marvin-tests	`24.83% <77.61%> (+0.19%)`	⬆️
uitests	`4.39% <0.00%> (-0.01%)`	⬇️
unit-tests	`16.55% <80.59%> (+0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

blueorangutan · 2024-01-17T07:25:24Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 8342

vishesh92 · 2024-01-17T19:55:43Z

@blueorangutan package

blueorangutan · 2024-01-17T19:56:03Z

@vishesh92 a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-01-17T21:01:40Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 8349

vishesh92 · 2024-01-17T21:11:41Z

@blueorangutan test

blueorangutan · 2024-01-17T21:12:03Z

@vishesh92 a [SL] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

kiranchavala · 2024-01-22T04:42:21Z

@blueorangutan package

blueorangutan · 2024-01-22T05:00:04Z

@kiranchavala a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-01-22T06:07:59Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 8402

vishesh92 · 2024-01-22T07:00:07Z

@blueorangutan package

kiranchavala · 2024-01-22T08:42:47Z

@blueorangutan package

blueorangutan · 2024-01-22T08:44:03Z

@kiranchavala a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

kiranchavala · 2024-01-24T06:41:53Z

@blueorangutan package

blueorangutan · 2024-01-24T06:44:04Z

@kiranchavala a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-01-24T07:51:33Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 8429

DaanHoogland

clgtm

vishesh92 · 2024-02-06T20:19:33Z

@blueorangutan package

blueorangutan · 2024-02-06T20:20:03Z

@vishesh92 a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-02-06T21:27:07Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 8558

vishesh92 · 2024-02-07T08:33:58Z

@blueorangutan package

blueorangutan · 2024-02-07T08:36:03Z

@vishesh92 a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-02-07T09:39:21Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 8567

kiranchavala · 2024-02-07T10:06:14Z

@blueorangutan package

blueorangutan · 2024-02-07T10:08:03Z

@kiranchavala a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-02-07T11:08:33Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 8569

kiranchavala · 2024-02-08T06:20:21Z

@blueorangutan ui

blueorangutan · 2024-02-08T06:21:03Z

@kiranchavala a Jenkins job has been kicked to build UI QA env. I'll keep you posted as I make progress.

blueorangutan · 2024-02-08T06:36:37Z

UI build: ✔️
Live QA URL: https://qa.cloudstack.cloud/simulator/pr/8521 (QA-JID-274)

kiranchavala · 2024-02-08T08:29:11Z

@blueorangutan package

blueorangutan · 2024-02-08T08:30:03Z

@kiranchavala a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-02-08T09:30:31Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 8584

kiranchavala · 2024-02-08T09:34:05Z

@blueorangutan test

blueorangutan · 2024-02-08T09:36:03Z

@kiranchavala a [SL] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2024-02-09T00:48:37Z

[SF] Trillian test result (tid-9128)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 53316 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8521-t9128-kvm-centos7.zip
Smoke tests completed. 127 look OK, 2 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_secure_vm_migration	`Error`	275.94	test_vm_life_cycle.py
test_02_unsecure_vm_migration	`Error`	226.46	test_vm_life_cycle.py
test_03_secured_to_nonsecured_vm_migration	`Error`	155.55	test_vm_life_cycle.py
test_04_nonsecured_to_secured_vm_migration	`Error`	155.48	test_vm_life_cycle.py
test_02_redundant_VPC_default_routes	`Failure`	395.79	test_vpc_redundant.py
test_05_rvpc_multi_tiers	`Failure`	463.75	test_vpc_redundant.py
test_05_rvpc_multi_tiers	`Error`	463.77	test_vpc_redundant.py

kiranchavala

Tested the new Advanced DRS related settings manually and also the UI improvements

drs.metric.type
drs.metric.use.ratio
drs.imbalance.condensed.skip.threshold

Scenarios

Drs.imbalance= 0.9 and drs.alogirithm=condensed >> The vms got migrated on a single host

Drs.imbalance= 0.4 and drs.alogirthm=balanced >> The vms got distributed across the hosts

Screenshots

Logs

Condensed

2024-02-08 05:35:54,311 DEBUG [o.a.c.c.Balanced] (qtp1491755116-22:ctx-ecfb388b ctx-9739cec5) (logid:8253e2f5) Cluster 2 needs DRS. Imbalance: 0.7158188976374373 Threshold: 0.5 Algorithm: balanced DRS metric: memory Metric Type: used Use ratio: false
2024-02-08 05:35:54,405 DEBUG [o.a.c.c.Balanced] (qtp1491755116-22:ctx-ecfb388b ctx-9739cec5) (logid:8253e2f5) Cluster 2 pre-imbalance: 0.7158188976374373 post-imbalance: 0.7820295697311479 Algorithm: balanced VM: f659a6f7-2915-4a18-9217-11cf8a03dce1 srcHost: 4 destHost: a16c16d2-a2bb-4dbf-93fb-838568f3a2fd
2024-02-08 05:35:54,413 DEBUG [o.a.c.c.Balanced] (qtp1491755116-22:ctx-ecfb388b ctx-9739cec5) (logid:8253e2f5) Cluster 2 pre-imbalance: 0.7158188976374373 post-imbalance: 0.34015067152490375 Algorithm: balanced VM: f659a6f7-2915-4a18-9217-11cf8a03dce1 srcHost: 4 destHost: b1f62d2b-fee5-4fab-b3c1-40ab3ad68a45
2024-02-08 05:35:54,413 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (qtp1491755116-22:ctx-ecfb388b ctx-9739cec5) (logid:8253e2f5) Plan for VM e1096698-c0f3-4c75-8c9a-fc3526c938dd to migrate from host 722009bc-e65f-465c-a55f-1e26b70b93ae to host b1f62d2b-fee5-4fab-b3c1-40ab3ad68a45
2024-02-08 05:35:54,417 DEBUG [o.a.c.c.Balanced] (qtp1491755116-22:ctx-ecfb388b ctx-9739cec5) (logid:8253e2f5) Cluster 2 does not need DRS. Imbalance: 0.34015067152490375 Threshold: 0.5 Algorithm: balanced DRS metric: memory Metric Type: used Use ratio: false

Balanced

2024-02-08 05:49:16,865 DEBUG [o.a.c.c.Condensed] (qtp1491755116-22:ctx-dbbaeaf5 ctx-975fb795) (logid:d6ccccf6) Cluster 2 needs DRS. Imbalance: 0.34015067152490375 Threshold: 0.8999999761581421 Algorithm: condensed DRS metric: memory Metric Type: used Use ratio: false SkipThreshold: 0.95
2024-02-08 05:49:17,055 DEBUG [o.a.c.c.Condensed] (qtp1491755116-22:ctx-dbbaeaf5 ctx-975fb795) (logid:d6ccccf6) Cluster 2 pre-imbalance: 0.7820295697311479 post-imbalance: 1.0523488093445659 Algorithm: condensed VM: f659a6f7-2915-4a18-9217-11cf8a03dce1 srcHost: 4 destHost: a16c16d2-a2bb-4dbf-93fb-838568f3a2fd
2024-02-08 05:49:17,060 DEBUG [o.a.c.c.Condensed] (qtp1491755116-22:ctx-dbbaeaf5 ctx-975fb795) (logid:d6ccccf6) Cluster 2 pre-imbalance: 0.7820295697311479 post-imbalance: 0.6428243465332251 Algorithm: condensed VM: f659a6f7-2915-4a18-9217-11cf8a03dce1 srcHost: 4 destHost: b1f62d2b-fee5-4fab-b3c1-40ab3ad68a45
2024-02-08 05:49:16,965 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (qtp1491755116-22:ctx-dbbaeaf5 ctx-975fb795) (logid:d6ccccf6) Plan for VM e1096698-c0f3-4c75-8c9a-fc3526c938dd to migrate from host b1f62d2b-fee5-4fab-b3c1-40ab3ad68a45 to host a16c16d2-a2bb-4dbf-93fb-838568f3a2fd
2024-02-08 05:49:17,060 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (qtp1491755116-22:ctx-dbbaeaf5 ctx-975fb795) (logid:d6ccccf6) Plan for VM 0381cd82-f9dd-417d-836f-99dd300720e7 to migrate from host 722009bc-e65f-465c-a55f-1e26b70b93ae to host a16c16d2-a2bb-4dbf-93fb-838568f3a2fd
2024-02-08 05:49:17,064 DEBUG [o.a.c.c.Condensed] (qtp1491755116-22:ctx-dbbaeaf5 ctx-975fb795) (logid:d6ccccf6) Cluster 2 does not need DRS. Imbalance: 1.0523488093445659 Threshold: 0.8999999761581421 Algorithm: condensed DRS metric: memory Metric Type: used Use ratio: false SkipThreshold: 0.95

yadvr · 2024-02-13T05:48:50Z

LGTM, thanks for the fixes and testing @vishesh92 @kiranchavala

) * Use free/total instead of free metric to calculate imbalance * Filter out hosts for condensed while checking imbalance * Make DRS more configurable * code refactor * Add unit tests * fixup * Fix validation for drs.imbalance.condensed.skip.threshold * Add logging and other minor changes for drs * Add some logging for drs * Change format for drs imbalance to string * Show drs imbalance as percentage * Fixup label for memorytotal in en.json

boring-cyborgBot added the component:api label Jan 17, 2024

DaanHoogland reviewed Jan 17, 2024
View reviewed changes

Comment threadplugins/drs/cluster/condensed/src/main/java/org/apache/cloudstack/cluster/Condensed.java Outdated
Comment threadplugins/drs/cluster/condensed/src/main/java/org/apache/cloudstack/cluster/Condensed.java Outdated

vishesh92 changed the title ~~Refactoring to allow DRS in a cluster with hosts of different capacities~~Add advance settings to fine tune DRS imbalance calculationJan 18, 2024

vishesh92 mentioned this pull request Jan 18, 2024
Document advance DRS settings apache/cloudstack-documentation#374
Merged

DaanHoogland reviewed Jan 18, 2024
View reviewed changes

Comment threadapi/src/main/java/org/apache/cloudstack/cluster/ClusterDrsAlgorithm.java Outdated

DaanHoogland reviewed Jan 18, 2024
View reviewed changes

Comment threadapi/src/main/java/org/apache/cloudstack/cluster/ClusterDrsAlgorithm.java Outdated

DaanHoogland reviewed Jan 18, 2024
View reviewed changes

Comment threadapi/src/main/java/org/apache/cloudstack/cluster/ClusterDrsAlgorithm.java Outdated

DaanHoogland reviewed Jan 18, 2024
View reviewed changes

Comment threadapi/src/main/java/org/apache/cloudstack/cluster/ClusterDrsAlgorithm.java Outdated

vishesh92 force-pushed the drs-fixes branch from 38db77b to 48b9bdbCompare January 19, 2024 06:46

vishesh92 marked this pull request as ready for review January 19, 2024 09:13

harikrishna-patnala reviewed Jan 20, 2024
View reviewed changes

Comment threadplugins/drs/cluster/condensed/src/main/java/org/apache/cloudstack/cluster/Condensed.java

DaanHoogland approved these changes Jan 24, 2024
View reviewed changes

boring-cyborgBot added the component:UI label Feb 6, 2024

Add some logging for drs
89efed2

Change format for drs imbalance to string
e90eae9

Show drs imbalance as percentage
16ec9ae

vishesh92 force-pushed the drs-fixes branch from 66ef8db to 16ec9aeCompare February 7, 2024 09:54

Fixup label for memorytotal in en.json
b917ecf

kiranchavala approved these changes Feb 12, 2024
View reviewed changes

yadvr merged commit 1955d8f into apache:4.19Feb 13, 2024

yadvr deleted the drs-fixes branch February 13, 2024 05:48

vishesh92 mentioned this pull request Jun 13, 2024
DRS failed for cluster if have another cluster in zone #8629
Closed

Uh oh!

Conversation

vishesh92 commented Jan 17, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

vishesh92 commented Jan 17, 2024

Uh oh!

blueorangutan commented Jan 17, 2024

Uh oh!

codecovBot commented Jan 17, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

blueorangutan commented Jan 17, 2024

Uh oh!

Uh oh!

Uh oh!

vishesh92 commented Jan 17, 2024

Uh oh!

blueorangutan commented Jan 17, 2024

Uh oh!

blueorangutan commented Jan 17, 2024

Uh oh!

vishesh92 commented Jan 17, 2024

Uh oh!

blueorangutan commented Jan 17, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kiranchavala commented Jan 22, 2024

Uh oh!

blueorangutan commented Jan 22, 2024

Uh oh!

blueorangutan commented Jan 22, 2024

Uh oh!

vishesh92 commented Jan 22, 2024

Uh oh!

kiranchavala commented Jan 22, 2024

Uh oh!

blueorangutan commented Jan 22, 2024

Uh oh!

kiranchavala commented Jan 24, 2024

Uh oh!

blueorangutan commented Jan 24, 2024

Uh oh!

blueorangutan commented Jan 24, 2024

Uh oh!

DaanHoogland left a comment

Choose a reason for hiding this comment

Uh oh!

vishesh92 commented Feb 6, 2024

Uh oh!

blueorangutan commented Feb 6, 2024

Uh oh!

blueorangutan commented Feb 6, 2024

Uh oh!

vishesh92 commented Feb 7, 2024

Uh oh!

blueorangutan commented Feb 7, 2024

Uh oh!

blueorangutan commented Feb 7, 2024

Uh oh!

kiranchavala commented Feb 7, 2024

Uh oh!

blueorangutan commented Feb 7, 2024

Uh oh!

blueorangutan commented Feb 7, 2024

Uh oh!

kiranchavala commented Feb 8, 2024

vishesh92 commented Jan 17, 2024•
edited
Loading

codecovBot commented Jan 17, 2024•
edited
Loading