api,server,ui: snapshot copy, multi-zone replica

[shwstppr]

Description

This PR adds new functionality to copy snapshots across zones and take snapshots for multiple zones.

Copy functionality is similar to template copy. The source zone acts as the web server from where the destination zone(s) can download the snapshot files. For this purpose, a new API - copySnapshot has been added. The response for copySnapshot will be returning zone and download details from the first destination zone of the request. This behaviour is similar to the copyTemplate API.

In a similar manner, multiple zones can be selected while taking the snapshots or creating snapshot policies. For this snapshot will be taken in the base zone(in which volume is present) and then copied to the additional zones. A new parameter - zoneids has been added to createSnapshot and createSnapshotPolicy APIs.

As snapshots can be present on multiple zones (secondary stores), a new parameter zoneid has been added to delete the snapshot copy on a specific zone.

listSnapshots API has been updated to allow listing snapshot entries for different zones/datastores. New parameters - showUnique, locationType have been added.

Events generated during snapshot operations will now be linked to the snapshot itself rather than the volume of the snapshot.

listSnapshotPolicies and createSnapshotPolicy APIs will return zone details of the zones in which backup will be scheduled for the policy.

---

New API added
copySnapshot

Request and response params updated for APIs

- listSnapshots
- deleteSnapshot
- createTemplate
- listZones
- listSnapshotPolicies
- createSnapshotPolicy

UI updated for

• Snapshot detail view
• Create snapshot form
• Create snapshot policy form
• Create volume (from snapshot) form
• Create template (from snapshot) form

Doc PR: apache/cloudstack-documentation#344

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

• Major
• Minor

Bug Severity

• BLOCKER
• Critical
• Major
• Minor
• Trivial

Screenshots (if appropriate):

How Has This Been Tested?

Manual snapshot operations in a multi-zone environment

Smoke test added:

[root@ref-trl-5659-k-M7-abhishek-kumar-marvin marvin]# nosetests --with-xunit --xunit-file=results.xml --with-marvin --marvin-config=./ref-trl-5659-k-M7-abhishek-kumar-advanced-cfg -s -a tags=advanced --hypervisor=KVM tests/component/test_snapshot_copy.py
/usr/local/lib/python3.6/site-packages/paramiko/transport.py:32: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography. The next release of cryptography will remove support for Python 3.6.
from cryptography.hazmat.backends import default_backend
==== Marvin Init Started ====
=== Marvin Parse Config Successful ===
=== Marvin Setting TestData Successful===
==== Log Folder Path: /marvin/MarvinLogs/Sep_13_2023_22_14_19_WGW1DW All logs will be available here ====
=== Marvin Init Logging Successful===
==== Marvin Init Successful ====
=== TestName: test_01_take_snapshot_multi_zone | Status : SUCCESS ===
=== TestName: test_02_copy_snapshot_multi_zone | Status : SUCCESS ===
=== TestName: test_03_take_snapshot_multi_zone_delete_single_zone | Status : SUCCESS ===
=== TestName: test_04_copy_snapshot_multi_zone_delete_all | Status : SUCCESS ===
=== TestName: test_05_take_snapshot_multi_zone_create_volume_additional_zone | Status : SUCCESS ===
=== TestName: test_06_take_snapshot_multi_zone_create_template_additional_zone | Status : SUCCESS ===
=== Final results are now copied to: /marvin//MarvinLogs/test_snapshot_copy_WS0393 ===

[shwstppr]

@blueorangutan package

[blueorangutan]

@shwstppr a [SF] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 6972

[shwstppr]

@blueorangutan package

[blueorangutan]

@shwstppr a [SF] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[codecov]

Codecov Report

Merging #7873 (ba05e7c) into main (c9d2303) will increase coverage by 10.10%.
The diff coverage is 22.67%.

@@ Coverage Diff @@## main #7873 +/- ##
=============================================
+ Coverage 19.07% 29.17% +10.10% - Complexity 18250 30616 +12366 
=============================================
Files 4967 5111 +144 Lines 336878 360624 +23746 Branches 48395 52697 +4302 =============================================
+ Hits 64246 105222 +40976 + Misses 263345 240946 -22399 - Partials 9287 14456 +5169 

Flag	Coverage Δ
simulator-marvin-tests	25.13% <17.53%> (+4.88%)	⬆️
uitests	4.79% <0.00%> (-0.06%)	⬇️
unit-tests	14.52% <15.17%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
api/src/main/java/com/cloud/event/EventTypes.java	95.88% <ø> (ø)
.../main/java/com/cloud/storage/VolumeApiService.java	100.00% <ø> (ø)
...n/java/org/apache/cloudstack/api/ApiConstants.java	40.90% <ø> (ø)
...k/api/command/user/snapshot/CreateSnapshotCmd.java	45.67% <100.00%> (+3.17%)	⬆️
...k/api/command/user/snapshot/DeleteSnapshotCmd.java	80.00% <100.00%> (+1.05%)	⬆️
...k/api/command/user/template/CreateTemplateCmd.java	59.74% <100.00%> (+7.10%)	⬆️
...cloudstack/api/command/user/zone/ListZonesCmd.java	93.75% <100.00%> (+0.41%)	⬆️
...g/apache/cloudstack/api/response/ZoneResponse.java	92.04% <100.00%> (-0.98%)	⬇️
...java/org/apache/cloudstack/query/QueryService.java	100.00% <ø> (ø)
...ack/engine/subsystem/api/storage/SnapshotInfo.java	100.00% <ø> (ø)
... and 75 more

... and 1638 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[blueorangutan]

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 6974

[blueorangutan]

[LL]Trillian test result (tid-6861)
Environment: vmware-70u3 (x2), Advanced Networking with Mgmt server r8
Total time taken: 48294 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7873-t6861-vmware-70u3.zip
Smoke tests completed. 113 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

[DaanHoogland]

@DaanHoogland, I have to run a few more tests with the latest changes

ok, we have an interest in putting this in 4.19. Can you give an E.T.A. on these, @slavkap ?

[shwstppr]

@blueorangutan package

[blueorangutan]

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 7456

[shwstppr]

@blueorangutan test matrix

[blueorangutan]

@shwstppr a [SL] Trillian-Jenkins matrix job (centos7 mgmt + xenserver71, rocky8 mgmt + vmware67u3, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

[blueorangutan]

[SF] Trillian test result (tid-8046)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 39837 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7873-t8046-kvm-centos7.zip
Smoke tests completed. 113 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

[blueorangutan]

[SF] Trillian test result (tid-8045)
Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server r8
Total time taken: 48666 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7873-t8045-vmware-67u3.zip
Smoke tests completed. 112 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_02_upgrade_kubernetes_cluster	Failure	588.34	test_kubernetes_clusters.py

[shwstppr]

@blueorangutan test centos7 xs71

[blueorangutan]

@shwstppr [SL] unsupported parameters provided. Supported mgmt server os are: centos7, centos6, suse15, alma8, ubuntu18, ubuntu22, ubuntu20, rocky8, alma9. Supported hypervisors are: kvm-centos6, kvm-centos7, kvm-rocky8, kvm-alma8, kvm-alma9, kvm-ubuntu18, kvm-ubuntu20, kvm-ubuntu22, kvm-suse15, vmware-55u3, vmware-60u2, vmware-65u2, vmware-67u3, vmware-70u1, vmware-70u2, vmware-70u3, vmware-80, vmware-80u1, xenserver-65sp1, xenserver-71, xenserver-74, xcpng74, xcpng76, xcpng80, xcpng81, xcpng82

[shwstppr]

@blueorangutan test centos7 xenserver-71

[blueorangutan]

@shwstppr a [SL] Trillian-Jenkins test job (centos7 mgmt + xenserver-71) has been kicked to run smoke tests

[slavkap]

CLGTM, tested with StorPool and NFS primary storage

[blueorangutan]

[SF] Trillian test result (tid-8051)
Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 7
Total time taken: 39412 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7873-t8051-xenserver-71.zip
Smoke tests completed. 113 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

[weizhouapache]

Merging based on approvals, QA testing.

great work @shwstppr !