Support multiple ceph monitors by weizhouapache · Pull Request #6792 · apache/cloudstack

weizhouapache · 2022-09-30T10:48:29Z

Description

This PR adds the support for multiple ceph monitors on UI and API.

What have been tested and work

vm life cycle
volume snapshot (created a separated PR kvm: fix backup volume snapshot fails on RBD storage #6790)
vm migration to another host
Stop a ceph monitor and test vm life cycle.

What have been tested but do not work (I believe they are not caused by this PR)

vm live migration with volume to another RBD primary storage
direct download to RBD primary storage
VM snapshots on RBD primary storage

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Screenshots (if appropriate):

How Has This Been Tested?

in agent.log Executing: qemu-img convert -O qcow2 -U --image-opts driver=raw,file.filename=rbd:cloudstack/c2a23920-9b0f-4e38-b68c-2ca2d040d9c5@79c0cb77-3733-4621-9555-9d8ba6d169b8:mon_host=10.0.32.254,10.0.32.207,10.0.32.208:auth_supported=cephx:id=cloudstack:key=AQDwHTNjjHXRKRAAJb+AToFr6x4a1AvKUc4Ksg==:rbd_default_format=2:client_mount_timeout=30 /mnt/7ceab3e8-1091-3c9e-84f9-2a1e20eb4051/snapshots/2/1490/79c0cb77-3733-4621-9555-9d8ba6d169b8 qemu-img: warning: short-form boolean option '10.0.32.207' deprecatedPlease use 10.0.32.207=on insteadqemu-img: Could not open 'driver=raw,file.filename=rbd:cloudstack/c2a23920-9b0f-4e38-b68c-2ca2d040d9c5@79c0cb77-3733-4621-9555-9d8ba6d169b8:mon_host=10.0.32.254,10.0.32.207,10.0.32.208:auth_supported=cephx:id=cloudstack:key=AQDwHTNjjHXRKRAAJb+AToFr6x4a1AvKUc4Ksg==:rbd_default_format=2:client_mount_timeout=30': Could not open 'rbd:cloudstack/c2a23920-9b0f-4e38-b68c-2ca2d040d9c5@79c0cb77-3733-4621-9555-9d8ba6d169b8:mon_host=10.0.32.254': No such file or directory

volume snapshot fails on RBD storage with the following error qemu-img: Could not open 'driver=raw,file.filename=rbd:cloudstack/test_wei.img:mon_host=10.0.32.254:auth_supported=cephx:id=cloudstack:key=AQDwHTNjjHXRKRAAJb+AToFr6x4a1AvKUc4Ksg==:rbd_default_format=2:client_mount_timeout=30': Could not open 'rbd:cloudstack/test_wei.img:mon_host=10.0.32.254:auth_supported=cephx:id=cloudstack:key=AQDwHTNjjHXRKRAAJb+AToFr6x4a1AvKUc4Ksg==:rbd_default_format=2:client_mount_timeout=30': No such file or directory However, it works without using image options Therefore, do not pass the image options if the image format is not QCOW2 and LUKS.

weizhouapache · 2022-09-30T10:48:43Z

@blueorangutan package

blueorangutan · 2022-09-30T10:49:03Z

@weizhouapache a Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2022-09-30T11:45:12Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ debian ✔️ suse15. SL-JID 4326

wido

Overall looks good! Just a few IPv6 requests

codecov · 2022-09-30T13:39:34Z

Codecov Report

Merging #6792 (7ad0b52) into main (713a236) will increase coverage by 0.03%.
The diff coverage is 55.17%.

@@ Coverage Diff @@## main #6792 +/- ##
============================================
+ Coverage 10.60% 10.63% +0.03% - Complexity 6849 6866 +17 
============================================
Files 2466 2466 Lines 244549 244563 +14 Branches 38262 38265 +3 ============================================
+ Hits 25936 26014 +78 + Misses 215331 215262 -69 - Partials 3282 3287 +5

Impacted Files	Coverage Δ
...ypervisor/kvm/resource/LibvirtDomainXMLParser.java	`48.49% <0.00%> (ø)`
...cycle/CloudStackPrimaryDataStoreLifeCycleImpl.java	`0.00% <0.00%> (ø)`
.../cloud/hypervisor/kvm/storage/KVMPhysicalDisk.java	`70.45% <70.00%> (+0.18%)`	⬆️
...om/cloud/hypervisor/kvm/resource/LibvirtVMDef.java	`67.04% <85.71%> (+2.88%)`	⬆️
...hypervisor/kvm/resource/LibvirtStoragePoolDef.java	`77.77% <100.00%> (+1.98%)`	⬆️
...isor/kvm/resource/LibvirtStoragePoolXMLParser.java	`60.65% <100.00%> (+60.65%)`	⬆️
...apache/cloudstack/utils/qemu/QemuImageOptions.java	`73.07% <100.00%> (+16.55%)`	⬆️
...rg/apache/cloudstack/quota/QuotaStatementImpl.java	`36.28% <0.00%> (-3.99%)`	⬇️
... and 2 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

weizhouapache · 2022-10-05T08:05:52Z

@wido
I have made some changes to support ceph monitors with IPv6 addresses. some test cases are added as well.

I am seriously wondering if you are using ceph with IPv6 address in your production, because I faced some blocker issues in my testing
(1) failed to copy template to ceph storage when deploy a vm. this is fixed by 4a242a7
(2) failed to take volume snapshot, this is also fixed by commit 4a242a7
(3) failed to start vm
when I create storage pool, the ipv6 addresses must have brackets ("[" and "]"), but the brackets must be removed in vm definition xml. this is fixed by 34f8732

With all commits in this PR, the RBD storage pools with single or multiple Ipv4/ipv6 monitors work well in my testing.
I will continue on testing.
now it is ready for code reviews. @wido @DaanHoogland @rohityadavcloud @borisstoyanov

weizhouapache · 2022-10-05T08:09:10Z

@blueorangutan package

blueorangutan · 2022-10-05T08:10:03Z

@weizhouapache a Jenkins job has been kicked to build packages. It will be bundled with

SystemVM template(s). I'll keep you posted as I make progress.

sonarqubecloud · 2022-10-05T09:11:22Z

SonarCloud Quality Gate failed.

1 Bug
0 Vulnerabilities
0 Security Hotspots
8 Code Smells

61.6% Coverage
0.0% Duplication

blueorangutan · 2022-10-05T09:14:06Z

Packaging result: ✔️ el7 ✖️ el8 ✖️ debian ✔️ suse15. SL-JID 4355

blueorangutan · 2022-10-06T08:19:03Z

@weizhouapache a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

weizhouapache · 2022-10-06T09:08:08Z

@blueorangutan package

blueorangutan · 2022-10-06T09:09:03Z

@weizhouapache a Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2022-10-06T09:58:30Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ debian ✔️ suse15. SL-JID 4363

weizhouapache · 2022-10-06T11:28:49Z

@blueorangutan test ubuntu22 kvm-ubuntu22

blueorangutan · 2022-10-06T11:29:03Z

@weizhouapache a Trillian-Jenkins test job (ubuntu22 mgmt + kvm-ubuntu22) has been kicked to run smoke tests

blueorangutan · 2022-10-06T20:17:46Z

Trillian test result (tid-5075)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 41746 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr6792-t5075-kvm-centos7.zip
Smoke tests completed. 104 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

blueorangutan · 2022-10-07T00:07:36Z

Trillian test result (tid-5076)
Environment: kvm-ubuntu22 (x2), Advanced Networking with Mgmt server u22
Total time taken: 43956 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr6792-t5076-kvm-ubuntu22.zip
Smoke tests completed. 103 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_08_upgrade_kubernetes_ha_cluster	`Failure`	592.41	test_kubernetes_clusters.py

weizhouapache · 2022-10-18T07:34:21Z

when testing this PR we have found an issue that

centos7 kvm host can NOT connect to ceph 17 (single monitor or multiple monitors)
centos7 kvm host can connecto to ceph 15

thanks @NuxRo

NuxRo

Did some testing, looks good to me!

DaanHoogland

clgtm

DaanHoogland · 2022-10-21T07:35:22Z

@wido by the looks of it all your comments are met, right?

yadvr · 2022-10-21T08:23:31Z

LGTM - let's merge this @DaanHoogland ?

DaanHoogland · 2022-10-21T08:36:27Z

We are not in "freeze" @rohityadavcloud ;)

zap51 · 2023-07-23T14:46:05Z

@rohityadavcloud @DaanHoogland A PR has been created on Ceph to include this.

weizhouapache added 6 commits September 30, 2022 11:29

utils: add getUriInfo in UriUtils
bf9f1ff

kvm plugin: support multiple hosts in vm def xml
5621f72

kvm plugin: parse storage pool xml with multiple hosts
cab8f65

kvm QemuImageOptions: use setFormat
ac63fe3

weizhouapache requested review from DaanHoogland, wido and yadvr September 30, 2022 10:48

boring-cyborgBot added component:kvm component:primary-storage component:storage labels Sep 30, 2022

weizhouapache mentioned this pull request Sep 30, 2022
RBD: support multiple MON hosts #4380
Closed
5 tasks

wido requested changes Sep 30, 2022
View reviewed changes

DaanHoogland reviewed Sep 30, 2022
View reviewed changes

Comment thread...rvisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/LibvirtStoragePoolXMLParser.java Outdated

weizhouapache added 6 commits October 5, 2022 09:43

kvmapache#6790: fix code smell
783e79b

Updateapache#6790: use uppercase
d8a661c

UriUtils: support RBD url with Ipv6 address
2a20362

kvm: fix RBD storage pool definition xml
f61f624

kvm: add slash before colon in IPv6 addresses for RBDStringBuilder
4a242a7

kvm: remove Ipv6 brackets in vm definition xml
34f8732

UI: add description of rados monitors
7ad0b52

boring-cyborgBot added the component:UI label Oct 6, 2022

NuxRo approved these changes Oct 20, 2022
View reviewed changes

DaanHoogland requested a review from wido October 21, 2022 06:48

DaanHoogland approved these changes Oct 21, 2022
View reviewed changes

DaanHoogland added this to the 4.18.0.0 milestone Oct 21, 2022

DaanHoogland self-assigned this Oct 21, 2022

yadvr marked this pull request as ready for review October 21, 2022 08:23

yadvr approved these changes Oct 21, 2022
View reviewed changes

DaanHoogland merged commit 48ffa5d into apache:mainOct 21, 2022

weizhouapache mentioned this pull request Oct 21, 2022
KVM HA Enhancements for Ceph RBD Primary Storage #5862
Closed
12 tasks

weizhouapache deleted the 4.18-multiple-ceph-monitors branch December 9, 2022 10:38

weizhouapache mentioned this pull request Mar 13, 2023
Failover of RADOS monitor when intergrate Cloudstack with CEPH RBD #7334
Closed

tuanhoangth1603 mentioned this pull request Mar 13, 2023
HA - Load balance ceph-mon with HAProxy - Cloudstack v4.17 #7336
Closed

DaanHoogland removed their assignment Jun 13, 2023

weizhouapache mentioned this pull request Dec 13, 2023
Cannot create primary on StorPool #8352
Closed

weizhouapache mentioned this pull request Feb 4, 2026
Support for custom SSH port for KVM hosts from the host url on add host and the configuration #12571
Merged
14 tasks

Uh oh!

Conversation

weizhouapache commented Sep 30, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Screenshots (if appropriate):

How Has This Been Tested?

Uh oh!

weizhouapache commented Sep 30, 2022

Uh oh!

blueorangutan commented Sep 30, 2022

Uh oh!

blueorangutan commented Sep 30, 2022

Uh oh!

wido left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecovBot commented Sep 30, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

weizhouapache commented Oct 5, 2022

Uh oh!

weizhouapache commented Oct 5, 2022

Uh oh!

blueorangutan commented Oct 5, 2022

Uh oh!

sonarqubecloudBot commented Oct 5, 2022

Uh oh!

blueorangutan commented Oct 5, 2022

Uh oh!

blueorangutan commented Oct 6, 2022

Uh oh!

weizhouapache commented Oct 6, 2022

Uh oh!

blueorangutan commented Oct 6, 2022

Uh oh!

blueorangutan commented Oct 6, 2022

Uh oh!

weizhouapache commented Oct 6, 2022

Uh oh!

blueorangutan commented Oct 6, 2022

Uh oh!

blueorangutan commented Oct 6, 2022

Uh oh!

blueorangutan commented Oct 7, 2022

Uh oh!

weizhouapache commented Oct 18, 2022

Uh oh!

NuxRo left a comment

Choose a reason for hiding this comment

Uh oh!

DaanHoogland left a comment

Choose a reason for hiding this comment

Uh oh!

DaanHoogland commented Oct 21, 2022

Uh oh!

yadvr commented Oct 21, 2022

Uh oh!

DaanHoogland commented Oct 21, 2022

Uh oh!

zap51 commented Jul 23, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

weizhouapache commented Sep 30, 2022•
edited
Loading

codecovBot commented Sep 30, 2022•
edited
Loading

zap51 commented Jul 23, 2023•
edited
Loading