Adding Centos8, Ubuntu 20.04, XCPNG8.1 Support by davidjumani · Pull Request #4068 · apache/cloudstack

davidjumani · 2020-05-11T04:52:13Z

Support for CentOS 8 as mgmt/kvm host
Support for Ubuntu 20.04 as mgmt/kvm host
Support for XCP-ng 8.1 as Hypervisor.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Changes have been made in smoke tests to use test template for Xenserver/SCP-ng as XCP-ng 8 and above does not support built-in PV template. Since test template at present does not support iso9660 filesystem, ISO attach-detach test for the VM will skip mounting of the ISO.

- Splits commands to create user and grant access on database, the old statement is no longer supported by MySQL 8.x - `NO_AUTO_CREATE_USER` is no longer supported by MySQL 8.x so remove that from db.properties conn parameters For mysql-server 8.x setup the following changes were added/tested to make it work with CloudStack in /etc/mysql/mysql.conf.d/mysqld.cnf and then restart the mysql-server process: server_id = 1 sql-mode="STRICT_TRANS_TABLES,NO_ENGINE_SUBSTITUTION,ERROR_FOR_DIVISION_BY_ZERO,NO_ZERO_DATE,NO_ZERO_IN_DATE,NO_ENGINE_SUBSTITUTION" innodb_rollback_on_timeout=1 innodb_lock_wait_timeout=600 max_connections=1000 log-bin=mysql-bin binlog-format = 'ROW' default-authentication-plugin=mysql_native_password Notice the last line above, this is to reset the old password based authentication used by MySQL 5.x. Developers can set empty password as follows: > sudo mysql -u root ALTER USER 'root'@'localhost' IDENTIFIED BY '';

davidjumani · 2020-05-11T04:56:58Z

@weizhouapache We'll be adding our changes here, including the python3 support!

yadvr · 2020-05-11T11:40:53Z

@davidjumani @shwstppr related PR for Ubuntu 20.04 with likely some overlapping parts - #4069

weizhouapache · 2020-05-12T07:47:47Z

@davidjumani I suggest to create separated PR for python2 to python3 upgrade
it seems to be ready for review, as you have changed many files.

yadvr · 2020-05-12T09:13:19Z

@weizhouapache we want to keep them in this PR as CentOS8 ships by default with python3, it may be tricky in some cases to test this PR without adding some python3 support.

weizhouapache · 2020-05-12T10:07:51Z

@weizhouapache we want to keep them in this PR as CentOS8 ships by default with python3, it may be tricky in some cases to test this PR without adding some python3 support.

@rhtyd @davidjumani
you can keep python3 changes in this branch, it is fine. It would be nice to create separated PR for python3 so we can test and merge it as soon as possible. it is important as it is required by both centos8 and ubuntu20.04 support.

we can upgrade to python3 without any changes for centos8 and ubuntu 20.04, as python3 can be tested on centos7 and ubuntu 18.04/ubuntu 16.04.

Comparing with a large PR with many changes, I prefer to create some small/medium PRs with one or two feature/improvement.

yadvr

Remove changes to unnecessary files in tools and elsewhere. Please keep the scope limited to management/usage server scripts and kvm-agent.

In libvirt repository, there are two related commits 2019-08-23 13:13 Daniel P. Berrangé ● rpm: don't enable socket activation in upgrade if --listen present 2019-08-22 14:52 Daniel P. Berrangé ● remote: forbid the --listen arg when systemd socket activation In libvirt.spec.in /bin/systemctl mask libvirtd.socket >/dev/null 2>&1 || : /bin/systemctl mask libvirtd-ro.socket >/dev/null 2>&1 || : /bin/systemctl mask libvirtd-admin.socket >/dev/null 2>&1 || : /bin/systemctl mask libvirtd-tls.socket >/dev/null 2>&1 || : /bin/systemctl mask libvirtd-tcp.socket >/dev/null 2>&1 || :

For more information: https://libvirt.org/kbase/backing_chains.html

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

vladimirpetrov · 2020-08-14T09:46:20Z

@blueorangutan help

blueorangutan · 2020-08-14T09:46:56Z

@vladimirpetrov I understand these words: "help", "hello", "thanks", "package", "test"
Test command usage: test [mgmt os] [hypervisor] [additional tests]
Mgmt OS options: ['centos6', 'centos7', 'centos8', 'ubuntu']
Hypervisor options: ['kvm-centos6', 'kvm-centos7', 'kvm-centos8', 'kvm-ubuntu', 'xenserver-71', 'xenserver-65sp1', 'vmware-67u3', 'vmware-65u2', 'vmware-60u2', 'vmware-55u3', 'xcpng76', 'xcpng80', 'xcpng81', 'xenserver-74', 'xcpng74']
Additional tests: list of space separated tests with paths relative to the test/integration directory, for example: component/test_acl_listvm.py component/test_volumes.py
Note: when additional tests are passed, you need to specify mgmt server os and hypervisor or use the matrix command.

Blessed contributors for kicking Trillian test jobs: ['rhtyd', 'nvazquez', 'PaulAngus', 'borisstoyanov', 'DaanHoogland', 'shwstppr', 'andrijapanicsb', 'Spaceman1984', 'Pearl1594', 'davidjumani', 'harikrishna-patnala', 'vladimirpetrov', 'sureshanaparti']

vladimirpetrov · 2020-08-14T09:48:25Z

@blueorangutan test centos8 kvm-centos8

blueorangutan · 2020-08-14T09:48:56Z

@vladimirpetrov a Trillian-Jenkins test job (centos8 mgmt + kvm-centos8) has been kicked to run smoke tests

yadvr · 2020-08-14T11:00:44Z

Env issue failed, rekicked job at the backend (job/acs-pr-trillian/2449/console)

blueorangutan · 2020-08-15T02:33:11Z

Trillian test result (tid-2450)
Environment: kvm-centos8 (x2), Advanced Networking with Mgmt server 8
Total time taken: 52615 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4068-t2450-kvm-centos8.zip
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_supported_versions.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_vpn.py
Intermittent failure detected: /marvin/tests/smoke/test_hostha_kvm.py
Smoke tests completed. 82 look OK, 3 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_01_add_delete_kubernetes_supported_version	`Error`	1807.51	test_kubernetes_supported_versions.py
test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL	`Failure`	430.26	test_vpc_redundant.py
test_04_rvpc_network_garbage_collector_nics	`Error`	3922.06	test_vpc_redundant.py
test_hostha_kvm_host_fencing	`Error`	170.85	test_hostha_kvm.py

blueorangutan · 2020-08-15T02:39:28Z

Trillian test result (tid-2452)
Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server 8
Total time taken: 52647 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4068-t2452-vmware-67u3.zip
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_supported_versions.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Smoke tests completed. 83 look OK, 2 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_01_add_delete_kubernetes_supported_version	`Error`	1808.71	test_kubernetes_supported_versions.py
test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL	`Failure`	501.15	test_vpc_redundant.py

blueorangutan · 2020-08-15T04:34:32Z

Trillian test result (tid-2451)
Environment: xenserver-74 (x2), Advanced Networking with Mgmt server 8
Total time taken: 59393 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4068-t2451-xenserver-74.zip
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_supported_versions.py
Intermittent failure detected: /marvin/tests/smoke/test_routers_network_ops.py
Intermittent failure detected: /marvin/tests/smoke/test_scale_vm.py
Intermittent failure detected: /marvin/tests/smoke/test_volumes.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermittent failure detected: /marvin/tests/smoke/test_host_maintenance.py
Smoke tests completed. 82 look OK, 3 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_01_add_delete_kubernetes_supported_version	`Error`	1807.42	test_kubernetes_supported_versions.py
test_01_scale_vm	`Failure`	27.73	test_scale_vm.py
test_11_migrate_volume_and_change_offering	`Error`	8.42	test_volumes.py

yadvr · 2020-08-15T17:50:31Z

@blueorangutan test centos8 xenserver-71

blueorangutan · 2020-08-15T17:50:56Z

@rhtyd a Trillian-Jenkins test job (centos8 mgmt + xenserver-71) has been kicked to run smoke tests

blueorangutan · 2020-08-16T09:45:39Z

Trillian test result (tid-2457)
Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 8
Total time taken: 55363 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4068-t2457-xenserver-71.zip
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_supported_versions.py
Intermittent failure detected: /marvin/tests/smoke/test_scale_vm.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Smoke tests completed. 83 look OK, 2 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_01_add_delete_kubernetes_supported_version	`Error`	1568.38	test_kubernetes_supported_versions.py
test_01_scale_vm	`Failure`	25.70	test_scale_vm.py

borisstoyanov

LGTM, based on marvin results. There are some failures, but I don't think they are related

andrijapanicsb · 2020-08-17T10:01:18Z

@blueorangutan test centos7 kvm-centos7

blueorangutan · 2020-08-17T10:02:01Z

@andrijapanicsb a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

andrijapanicsb · 2020-08-17T10:09:27Z

LGTM

Any outstanding issues with tests, will have to be resolved for 4.15

yadvr · 2020-08-17T10:57:56Z

Thanks @andrijapanicsb @borisstoyanov - agree failing tests are all intermittent failures. Merging this.

blueorangutan · 2020-08-18T00:19:20Z

Trillian test result (tid-2476)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 49408 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4068-t2476-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_supported_versions.py
Intermittent failure detected: /marvin/tests/smoke/test_outofbandmanagement.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermittent failure detected: /marvin/tests/smoke/test_hostha_kvm.py
Smoke tests completed. 82 look OK, 3 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_01_add_delete_kubernetes_supported_version	`Error`	1806.94	test_kubernetes_supported_versions.py
test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL	`Failure`	439.99	test_vpc_redundant.py
test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers	`Error`	361.84	test_vpc_redundant.py
test_04_rvpc_network_garbage_collector_nics	`Error`	3884.86	test_vpc_redundant.py
test_hostha_kvm_host_fencing	`Error`	8.76	test_hostha_kvm.py

andrijapanicsb · 2020-09-07T10:01:20Z

@weizhouapache were you able to test this PR against Ubuntu 20.04?. thx

weizhouapache · 2020-09-07T11:28:00Z

@andrijapanicsb tested some actions and found some issues, see PR #4301 and #4303

yadvr · 2020-09-07T19:53:46Z

Btw I'm running a custom build of 4.14 on Ubuntu 20.04 on raspberrypi4 using commits from Wei/this PR @andrijapanicsb

weizhouapache · 2020-09-11T11:49:07Z

@rhtyd @PaulAngus maybe we can work together to make 4.14 on ubuntu 20.04.

I have backported some changes in this PR to 4.14, it is tested ok.
https://github.com/ustcweizhou/cloudstack/commits/4.14-ubuntu2004

andrijapanicsb · 2020-09-11T11:53:35Z

not trying to be a smart one, but don't we only implement fixes on 4.14.x release? (i.e. no new features) - or am I confusing things here @weizhouapache ? Otherwise that sounds good (from a user perspective)

weizhouapache · 2020-09-11T14:15:13Z

not trying to be a smart one, but don't we only implement fixes on 4.14.x release? (i.e. no new features) - or am I confusing things here @weizhouapache ? Otherwise that sounds good (from a user perspective)

@andrijapanicsb true. 4.14 accepts only bug fixes. so I will not create PR for it.
However, it is good to provide packages for cloudstack users who want to run cloudstack 4.14.X on ubuntu 20.04.
The packages are not official. it could be something like ubuntu/ppa: https://launchpad.net/ubuntu/+ppas

vladimirpetrov · 2020-11-10T10:07:51Z

Management server with Ubuntu 20.04 - LGTM, here is what I've tested:

VM network - add/delete secondary IP, change IP address, change default NIC, add/remove NIC
Guest networks - create/remove/restart with clean up isolated, L2 and shared network, add/remove egress, firewall and port-forwarding rules
Images - register/delete ISO, register/delete template,
Volumes - create, attach/detach, migrate, create/restore snapshot, create template from volume, download volume, resize, delete
System VMs operations - start/stop, reboot, migrate, get diag. data, run diagnostics, destroy
Storages - add/remove primary and secondary storages

andrijapanicsb · 2020-11-10T10:16:23Z

It's worth mentioning, that on AMD EPYC CPUs, we had to upgrade Ubuntu kernel to 5.8.1+ in order to resolve a Qemu bug with CPU registers and stuff....where the VM can't be started at all.

andrijapanicsb · 2020-11-10T10:16:49Z

(same goes for CentOS 8, had to upgrade to Elrepo 5.8+ kernel to solve the same issues)

davidjumani force-pushed the adding-centos8-support branch from 5fa09d0 to eb9d639Compare May 11, 2020 10:13

yadvr mentioned this pull request May 11, 2020
Ubuntu 20.04 support (as hypervisor) #4069
Closed
5 tasks

yadvr added this to the 4.15.0.0 milestone May 11, 2020

weizhouapache mentioned this pull request May 12, 2020
[4.15] Support of next versions #4081
Closed

davidjumani force-pushed the adding-centos8-support branch from 5be4b2e to f5535b6Compare May 12, 2020 08:25

svenvogel added component:kvm component:management-server type:new-feature component:packaging component:usage-server labels May 12, 2020

yadvr requested changes May 15, 2020
View reviewed changes

yadvr reviewed May 15, 2020
View reviewed changes

Comment threadscripts/vm/hypervisor/xenserver/vmops Outdated

davidjumaniand others added 7 commits May 18, 2020 16:42

Adding centos8 packaging
83e8662

Ubuntu 20.04: Specify image format of backing file
9fef4a3
For more information: https://libvirt.org/kbase/backing_chains.html

Migrating cloudutils to python3
fc7113e

agent specific changes
28b1a2b
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

script fixes
e456aa0
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

agent: kvm - remove bridge-utils dependency
7f11aaa

davidjumani force-pushed the adding-centos8-support branch 2 times, most recently from f1aad4d to cae1374Compare May 18, 2020 11:44

Management scripts migrated to python3
67f40b5

davidjumani force-pushed the adding-centos8-support branch from cae1374 to 67f40b5Compare May 18, 2020 11:49

DaanHoogland changed the title ~~[WIP] Adding Centos8 Support~~Adding Centos8 SupportMay 18, 2020

borisstoyanov approved these changes Aug 17, 2020
View reviewed changes

yadvr merged commit d949302 into apache:masterAug 17, 2020

ACSGitBot added the type:new_feature label Nov 8, 2020

weizhouapache mentioned this pull request Feb 13, 2021
Qemu 2.10 requires -U flag to read volume metadata #4567
Merged
7 tasks

shwstppr mentioned this pull request Feb 24, 2021
packaging: update Requirements in README #4725
Merged
12 tasks

Uh oh!

Conversation

davidjumani commented May 11, 2020• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Types of changes

Uh oh!

davidjumani commented May 11, 2020

Uh oh!

yadvr commented May 11, 2020

Uh oh!

weizhouapache commented May 12, 2020• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yadvr commented May 12, 2020

Uh oh!

weizhouapache commented May 12, 2020• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yadvr left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

vladimirpetrov commented Aug 14, 2020

Uh oh!

blueorangutan commented Aug 14, 2020

Uh oh!

vladimirpetrov commented Aug 14, 2020

Uh oh!

blueorangutan commented Aug 14, 2020

Uh oh!

yadvr commented Aug 14, 2020

Uh oh!

blueorangutan commented Aug 15, 2020

Uh oh!

blueorangutan commented Aug 15, 2020

Uh oh!

blueorangutan commented Aug 15, 2020

Uh oh!

yadvr commented Aug 15, 2020

Uh oh!

blueorangutan commented Aug 15, 2020

Uh oh!

blueorangutan commented Aug 16, 2020

Uh oh!

borisstoyanov left a comment

Choose a reason for hiding this comment

Uh oh!

andrijapanicsb commented Aug 17, 2020

Uh oh!

blueorangutan commented Aug 17, 2020

Uh oh!

andrijapanicsb commented Aug 17, 2020

Uh oh!

yadvr commented Aug 17, 2020

Uh oh!

blueorangutan commented Aug 18, 2020

Uh oh!

andrijapanicsb commented Sep 7, 2020

Uh oh!

weizhouapache commented Sep 7, 2020

Uh oh!

yadvr commented Sep 7, 2020

Uh oh!

weizhouapache commented Sep 11, 2020

Uh oh!

andrijapanicsb commented Sep 11, 2020

Uh oh!

weizhouapache commented Sep 11, 2020

Uh oh!

vladimirpetrov commented Nov 10, 2020

Uh oh!

andrijapanicsb commented Nov 10, 2020

Uh oh!

andrijapanicsb commented Nov 10, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

davidjumani commented May 11, 2020•
edited
Loading

weizhouapache commented May 12, 2020•
edited
Loading

weizhouapache commented May 12, 2020•
edited
Loading

yadvr left a comment •
edited
Loading