Guest OS hypervisor mapping improvements by sureshanaparti · Pull Request #6561 · apache/cloudstack

sureshanaparti · 2022-07-21T10:19:12Z

Description

This PR includes the following Guest OS hypervisor mapping improvements.

Support for the Guest OS / hypervisor mapping addition from upgrade path (using GuestOsMapper)
Support to copy hypervisor mappings from older version from upgrade path (using GuestOsMapper)
Provision to validate the OS mapping name in hypervisor when enabled using the new parameter 'osmappingcheckenabled' in addGuestOsMapping API
Provision to get/check the hypervisor guest OS names (that can be used to add proper mapping) for VMware and XenServer using new API: getHypervisorGuestOsNames.
Display the Guest OS hypervisor mappings in UI.

Fixes: #3609

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

Manually tested add/update guest os mapping with the os name mapping check enabled, and tested the new getHypervisorGuestOsNames API with VMware and XenServer hypervisor.

(testenv-vmware) 🐱 > add guestosmapping hypervisor='VMware' hypervisorversion='5.5' osnameforhypervisor='centos' ostypeid=d6ea0862-0654-11ed-9169-1e00c10002c7 forced=true osmappingcheckenabled=true
🙈 Error: (HTTP 530, error code 4250) No host exists with hypervisor: VMware and version: 5.5, please specify available hypervisor and version
(testenv-vmware) 🐱 > add guestosmapping hypervisor='VMware' hypervisorversion='6.7.3' osnameforhypervisor='centos' ostypeid=d6ea0862-0654-11ed-9169-1e00c10002c7 forced=true osmappingcheckenabled=true
🙈 Error: (HTTP 530, error code 4250) Invalid hypervisor os mapping: centos for guest os: CentOS 7, hypervisor: VMware and version: 6.7.3
(testenv-vmware) 🐱 > get hypervisorguestosnames hypervisor='VMware' hypervisorversion='6.7.3' keyword='cent'
{
"hypervisorguestosnames": {
"guestoses": [
{
"osdisplayname": "Microsoft Windows Server 2003 Datacenter (64-bit)",
"osnameforhypervisor": "winNetDatacenter64Guest"
},
{
"osdisplayname": "Microsoft Windows Server 2003 Datacenter (32-bit)",
"osnameforhypervisor": "winNetDatacenterGuest"
},
{
"osdisplayname": "CentOS 8 (64-bit)",
"osnameforhypervisor": "centos8_64Guest"
},
{
"osdisplayname": "CentOS 7 (64-bit)",
"osnameforhypervisor": "centos7_64Guest"
},
{
"osdisplayname": "CentOS 6 (64-bit)",
"osnameforhypervisor": "centos6_64Guest"
},
{
"osdisplayname": "CentOS 6 (32-bit)",
"osnameforhypervisor": "centos6Guest"
},
{
"osdisplayname": "CentOS 4/5 or later (64-bit)",
"osnameforhypervisor": "centos64Guest"
},
{
"osdisplayname": "CentOS 4/5 or later (32-bit)",
"osnameforhypervisor": "centosGuest"
}
],
"guestosescount": 8,
"hypervisor": "VMware",
"hypervisorversion": "6.7.3"
}
}
(testenv-vmware) 🐱 > get hypervisorguestosnames hypervisor='VMware' hypervisorversion='6.7.3' keyword='core'
{
"hypervisorguestosnames": {
"guestoses": [
{
"osdisplayname": "CoreOS Linux (64-bit)",
"osnameforhypervisor": "coreos64Guest"
}
],
"guestosescount": 1,
"hypervisor": "VMware",
"hypervisorversion": "6.7.3"
}
}
(testenv-xen) 🐱 >get hypervisorguestosnames hypervisor='XenServer' hypervisorversion='7.1.0' keyword='core'
{
"hypervisorguestosnames": {
"guestoses": [
{
"osdisplayname": "CoreOS",
"osnameforhypervisor": "CoreOS"
}
],
"guestosescount": 1,
"hypervisor": "XenServer",
"hypervisorversion": "7.1.0"
}
}
(testenv-xen) 🐱 > get hypervisorguestosnames hypervisor='XenServer' hypervisorversion='7.1.0' keyword='cent'
{
"hypervisorguestosnames": {
"guestoses": [
{
"osdisplayname": "CentOS 4.8 (32-bit)",
"osnameforhypervisor": "CentOS 4.8 (32-bit)"
},
{
"osdisplayname": "CentOS 5 (32-bit)",
"osnameforhypervisor": "CentOS 5 (32-bit)"
},
{
"osdisplayname": "CentOS 7",
"osnameforhypervisor": "CentOS 7"
},
{
"osdisplayname": "CentOS 6 (64-bit)",
"osnameforhypervisor": "CentOS 6 (64-bit)"
},
{
"osdisplayname": "CentOS 4.6 (32-bit)",
"osnameforhypervisor": "CentOS 4.6 (32-bit)"
},
{
"osdisplayname": "CentOS 5 (64-bit)",
"osnameforhypervisor": "CentOS 5 (64-bit)"
},
{
"osdisplayname": "CentOS 4.7 (32-bit)",
"osnameforhypervisor": "CentOS 4.7 (32-bit)"
},
{
"osdisplayname": "CentOS 6 (32-bit)",
"osnameforhypervisor": "CentOS 6 (32-bit)"
},
{
"osdisplayname": "CentOS 4.5 (32-bit)",
"osnameforhypervisor": "CentOS 4.5 (32-bit)"
}
],
"guestosescount": 9,
"hypervisor": "XenServer",
"hypervisorversion": "7.1.0"
}
}
(testenv-xen) 🐱 > get hypervisorguestosnames hypervisor='XenServer' hypervisorversion='6.1.0'
{
"accountid": "aa2fc411-0654-11ed-8c69-1e00b50002e7",
"cmd": "org.apache.cloudstack.api.command.admin.guest.GetHypervisorGuestOsNamesCmd",
"completed": "2022-07-28T09:28:12+0000",
"created": "2022-07-28T09:28:12+0000",
"jobid": "e5189994-d441-42d3-bb75-26647210a7c1",
"jobprocstatus": 0,
"jobresult": {
"errorcode": 530,
"errortext": "No host exists with hypervisor: XenServer and version: 6.1.0, please specify available hypervisor and version"
},
"jobresultcode": 530,
"jobresulttype": "object",
"jobstatus": 2,
"userid": "aa310e7a-0654-11ed-8c69-1e00b50002e7"
}

acs-robot · 2022-07-21T10:20:10Z

Found UI changes, kicking a new UI QA build
@blueorangutan ui

blueorangutan · 2022-07-21T10:21:02Z

@acs-robot a Jenkins job has been kicked to build UI QA env. I'll keep you posted as I make progress.

sureshanaparti · 2022-07-21T10:21:32Z

@blueorangutan package

blueorangutan · 2022-07-21T10:22:03Z

@sureshanaparti a Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2022-07-21T10:27:17Z

UI build: ✔️
Live QA URL: http://qa.cloudstack.cloud:8080/client/pr/6561 (SL-JID-1990)

blueorangutan · 2022-07-21T10:59:03Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ debian ✔️ suse15. SL-JID 3807

- Checks the OS mapping name in hypervisor (VMware, XenServer) - Displays guest OS mappings in UI

… hypervisor, and code improvements

acs-robot · 2022-07-28T09:17:45Z

Found UI changes, kicking a new UI QA build
@blueorangutan ui

blueorangutan · 2022-07-28T09:18:03Z

@acs-robot a Jenkins job has been kicked to build UI QA env. I'll keep you posted as I make progress.

blueorangutan · 2022-07-28T09:23:13Z

UI build: ✔️
Live QA URL: http://qa.cloudstack.cloud:8080/client/pr/6561 (SL-JID-2021)

sureshanaparti · 2022-07-28T09:23:38Z

@blueorangutan package

blueorangutan · 2022-07-28T09:24:02Z

@sureshanaparti a Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2022-07-28T10:00:46Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ debian ✔️ suse15. SL-JID 3843

sureshanaparti · 2022-07-28T10:48:47Z

@blueorangutan test matrix

blueorangutan · 2022-07-28T10:49:03Z

@sureshanaparti a Trillian-Jenkins matrix job (centos7 mgmt + xs71, centos7 mgmt + vmware65, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

sureshanaparti · 2022-07-28T11:50:01Z

@blueorangutan package

acs-robot · 2022-07-28T11:50:58Z

Found UI changes, kicking a new UI QA build
@blueorangutan ui

blueorangutan · 2022-07-28T11:51:03Z

@sureshanaparti a Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2022-07-28T23:48:06Z

Trillian test result (tid-4555)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 45425 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr6561-t4555-kvm-centos7.zip
Smoke tests completed. 99 look OK, 1 have errors
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers	`Failure`	470.32	test_vpc_redundant.py

sureshanaparti · 2022-07-29T08:42:10Z

@blueorangutan package

acs-robot · 2022-07-29T08:42:52Z

Found UI changes, kicking a new UI QA build
@blueorangutan ui

blueorangutan · 2022-07-29T08:43:03Z

@sureshanaparti a Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2022-07-29T09:24:57Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ debian ✔️ suse15. SL-JID 3855

sonarqubecloud · 2022-07-29T09:43:04Z

SonarCloud Quality Gate failed.

0 Bugs
0 Vulnerabilities
0 Security Hotspots
12 Code Smells

1.3% Coverage
0.0% Duplication

sureshanaparti · 2022-08-07T17:49:46Z

@blueorangutan test

sureshanaparti · 2022-08-07T18:06:47Z

@blueorangutan help

blueorangutan · 2022-08-07T18:07:02Z

@sureshanaparti I understand these words: "help", "hello", "thanks", "package", "test"
Test command usage: test [mgmt os] [hypervisor] [keepEnv] [qemuEv]
Mgmt OS options: ['centos7', 'centos6', 'suse15', 'alma8', 'ubuntu18', 'ubuntu22', 'ubuntu20', 'rocky8']
Hypervisor options: ['kvm-centos6', 'kvm-centos7', 'kvm-rocky8', 'kvm-alma8', 'kvm-ubuntu18', 'kvm-ubuntu20', 'kvm-ubuntu22', 'kvm-suse15', 'vmware-55u3', 'vmware-60u2', 'vmware-65u2', 'vmware-67u3', 'vmware-70u1', 'vmware-70u2', 'vmware-70u3', 'xenserver-65sp1', 'xenserver-71', 'xenserver-74', 'xcpng74', 'xcpng76', 'xcpng80', 'xcpng81', 'xcpng82']
Note: when keepEnv is passed, you need to specify mgmt server os and hypervisor or use the matrix command.
when qemuEv is passed, it will deploy KVM hyperviosr hosts with qemu-kvm-ev, else it will default to stock qemu.
Package command usage: package [all(default value),kvm,xen,vmware,hyperv,ovm] - a comma separated list can be passed with package command to bundle the required hypervisor's systemVM templates. Not passing any argument will bundle all - kvm,xen and vmware templates.

Blessed contributors for kicking Trillian test jobs: ['rohityadavcloud', 'nvazquez', 'borisstoyanov', 'DaanHoogland', 'shwstppr', 'andrijapanicsb', 'Pearl1594', 'davidjumani', 'harikrishna-patnala', 'vladimirpetrov', 'weizhouapache', 'NuxRo']

DaanHoogland · 2022-08-17T16:22:35Z

I see nothing wrong with the code (even though sonar cloud does) . I have a question; how does this work with upgrade updates?

github-actions · 2022-12-05T18:54:03Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

codecov · 2022-12-12T19:09:30Z

Codecov Report

Merging #6561 (2a4dd78) into main (a3289f8) will decrease coverage by 0.09%.
The diff coverage is 7.07%.

❗ Current head 2a4dd78 differs from pull request most recent head 311d774. Consider uploading reports for the commit 311d774 to get more accurate results

@@ Coverage Diff @@## main #6561 +/- ##
============================================
- Coverage 11.59% 11.49% -0.10% + Complexity 7560 7534 -26 
============================================
Files 2494 2493 -1 Lines 247109 246949 -160 Branches 38619 38595 -24 ============================================
- Hits 28646 28399 -247 - Misses 214719 214951 +232 + Partials 3744 3599 -145

Impacted Files	Coverage Δ
...src/main/java/com/cloud/upgrade/GuestOsMapper.java	`4.67% <0.00%> (-0.83%)`	⬇️
...oud/hypervisor/vmware/resource/VmwareResource.java	`4.46% <0.00%> (-0.05%)`	⬇️
...src/main/java/com/cloud/api/ApiResponseHelper.java	`3.86% <0.00%> (-0.02%)`	⬇️
...in/java/com/cloud/server/ManagementServerImpl.java	`5.11% <0.00%> (-0.61%)`	⬇️
...java/com/cloud/hypervisor/vmware/mo/ClusterMO.java	`0.25% <0.00%> (-0.01%)`	⬇️
...in/java/com/cloud/hypervisor/vmware/mo/HostMO.java	`0.14% <0.00%> (-0.01%)`	⬇️
...CitrixGetHypervisorGuestOsNamesCommandWrapper.java	`8.33% <8.33%> (ø)`
...nbase/CitrixCheckGuestOsMappingCommandWrapper.java	`10.00% <10.00%> (ø)`
.../src/main/java/com/cloud/host/dao/HostDaoImpl.java	`29.20% <50.00%> (+0.31%)`	⬆️
...om/cloud/storage/dao/GuestOSHypervisorDaoImpl.java	`33.66% <50.00%> (+1.40%)`	⬆️
... and 268 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

github-actions · 2023-01-09T11:59:41Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

DaanHoogland

clgtm

DaanHoogland · 2023-01-09T16:18:43Z

+ if (cmd.getOsMappingCheckEnabled()) {
+ GuestOS guestOs = ApiDBUtils.findGuestOSById(guestOsHypervisorHandle.getGuestOsId());
+ if (guestOs == null) {
+ throw new InvalidParameterValueException("Unable to find the guest OS for the mapping");
+ }
+ checkGuestOSHypervisorMapping(HypervisorType.getType(guestOsHypervisorHandle.getHypervisorType()), guestOsHypervisorHandle.getHypervisorVersion(), guestOs.getDisplayName(), osNameForHypervisor);
+ }


extra method

sonarqubecloud · 2023-01-09T17:18:26Z

SonarCloud Quality Gate failed.

0 Bugs
0 Vulnerabilities
0 Security Hotspots
18 Code Smells

2.0% Coverage
0.0% Duplication

DaanHoogland · 2023-01-10T08:16:25Z

closing in favour of #6979

boring-cyborgBot added component:api component:management-server component:UI component:vmware component:XenServer labels Jul 21, 2022

shwstppr added this to the 4.18.0.0 milestone Jul 25, 2022

sureshanaparti added 2 commits July 28, 2022 14:43

Guest OS mapping improvements
ea5078b
- Checks the OS mapping name in hypervisor (VMware, XenServer) - Displays guest OS mappings in UI

Added API getHypervisorGuestOsNames to list the guest OS names in the…
7c6d56e
… hypervisor, and code improvements

sureshanaparti force-pushed the guest-os-mappings-improvements branch from 09b3eaf to 7c6d56eCompare July 28, 2022 09:14

boring-cyborgBot added the component:storage label Jul 28, 2022

sureshanaparti marked this pull request as ready for review July 28, 2022 09:25

apache deleted a comment from blueorangutan Jul 28, 2022

Some static analysis fixes
4c3a945

Removed commented code in listview
81cc716

yadvr requested review from DaanHoogland, Pearl1594, harikrishna-patnala and nvazquez August 16, 2022 10:50

github-actionsBot added the status:has-conflicts label Dec 5, 2022

Merge branch 'main' into guest-os-mappings-improvements
2a4dd78

github-actionsBot removed the status:has-conflicts label Dec 12, 2022

DaanHoogland mentioned this pull request Dec 12, 2022
Guest os mappings improvements #6979
Merged
12 tasks

github-actionsBot added the status:has-conflicts label Jan 9, 2023

Merge branch 'main' into guest-os-mappings-improvements
311d774

github-actionsBot removed the status:has-conflicts label Jan 9, 2023

DaanHoogland approved these changes Jan 9, 2023
View reviewed changes

DaanHoogland closed this Jan 10, 2023

DaanHoogland removed this from the 4.18.0.0 milestone Jan 10, 2023

Uh oh!

Conversation

sureshanaparti commented Jul 21, 2022• edited by DaanHoogland Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

Uh oh!

acs-robot commented Jul 21, 2022

Uh oh!

blueorangutan commented Jul 21, 2022

Uh oh!

sureshanaparti commented Jul 21, 2022

Uh oh!

blueorangutan commented Jul 21, 2022

Uh oh!

blueorangutan commented Jul 21, 2022

Uh oh!

blueorangutan commented Jul 21, 2022

Uh oh!

acs-robot commented Jul 28, 2022

Uh oh!

blueorangutan commented Jul 28, 2022

Uh oh!

blueorangutan commented Jul 28, 2022

Uh oh!

sureshanaparti commented Jul 28, 2022

Uh oh!

blueorangutan commented Jul 28, 2022

Uh oh!

blueorangutan commented Jul 28, 2022

Uh oh!

sureshanaparti commented Jul 28, 2022

Uh oh!

blueorangutan commented Jul 28, 2022

Uh oh!

sureshanaparti commented Jul 28, 2022

Uh oh!

acs-robot commented Jul 28, 2022

Uh oh!

blueorangutan commented Jul 28, 2022

Uh oh!

blueorangutan commented Jul 28, 2022

Uh oh!

sureshanaparti commented Jul 29, 2022

Uh oh!

acs-robot commented Jul 29, 2022

Uh oh!

blueorangutan commented Jul 29, 2022

Uh oh!

blueorangutan commented Jul 29, 2022

Uh oh!

sonarqubecloudBot commented Jul 29, 2022

Uh oh!

sureshanaparti commented Aug 7, 2022

Uh oh!

sureshanaparti commented Aug 7, 2022

Uh oh!

blueorangutan commented Aug 7, 2022

Uh oh!

DaanHoogland commented Aug 17, 2022

Uh oh!

github-actionsBot commented Dec 5, 2022

Uh oh!

codecovBot commented Dec 12, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actionsBot commented Jan 9, 2023

Uh oh!

DaanHoogland left a comment

Choose a reason for hiding this comment

Uh oh!

DaanHooglandJan 9, 2023

Choose a reason for hiding this comment

Uh oh!

sureshanaparti commented Jul 21, 2022•
edited by DaanHoogland
Loading

codecovBot commented Dec 12, 2022•
edited
Loading