Create new Instance from VM backup by abh1sar · Pull Request #10140 · apache/cloudstack

abh1sar · 2024-12-25T05:12:50Z

Doc PR: apache/cloudstack-documentation#474

Description

This PR adds the ability to create a new Instance from a VM backup for Dummy, NAS and Veeam backup Providers.
This will still work if the original Instance used to create the backup was expunged.
Design doc : https://cwiki.apache.org/confluence/display/CLOUDSTACK/Create+Instance+from+Backup

Other Miscellaneous changes

Framework level enhancements
a. Set name and description for backups
b. Show interval type for backup schedule in UI
c. Improve backup search in ui by keyword
NAS backup provider enhancements
a. Improved error logging and handling
b. Quiesce instance option during backup create
Backup And Object storage capacity tracking and alerts.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
build/CI
test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

1. Create Instance from Backup

Test Case	Notes
Create VM from backup using default metadata (no overrides)	Validated for all providers
Create VM from backup with custom service offering	Service Offering used and Disk size validation enforced
Create VM from backup with custom data volume sizes	Fails if smaller than original
Create VM with new IP/MAC (original instance not expunged)	New IP/MAC allocated
Create VM with original IP/MAC (original instance expunged)	Only works if IP is unassigned
Create VM from backup having ISO	Verified that ISO is detached before boot
Create VM from backup having ISO with multiple data volumes	Verified volume configurations
Create VM with SSH Key	SSH-key access is there
Create VM with multiple data volumes	All volumes created and attached
Create VM from backup using default metadata after deleting template	Proper error is thrown
Create VM from backup using default metadata after disabling service offering	Proper error is thrown
Create VM from backup using default metadata after deleting network	Proper error is thrown
Create VM from backup using default metadata after disabling disk offering	Proper error is thrown

2. Expunge/Unmanage VM with Backups

Test Case	Notes
Remove backup offering from VM with backups	Offering removed, backups retained
Expunge VM with retained backups	VM deleted, backups remain usable
Attempt to delete backup after expunging source VM	No dependency on VM existence
Create VM from backup of an expunged instance	Created as expected
Create VM from backup of an unmanaged instance	Created as expected
Reassign new backup offering to the VM and trigger new backup	New backups tracked under new offering
Verify database entry for expunged VM is retained	Entry present as expected

3. Usage Tracking

Test Case	Notes
Usage tracked per `(VM ID, Backup Offering ID)`	Validated in usage records
Metrics returned per tuple for all providers (NAS, Veeam, Dummy)	As expected
Backup offering removal does not delete usage record if backups exist	Usage still visible in `usage_vm_backup` table
Usage record deleted only when last backup for offering is deleted	Verified record purged post-delete
Usage metrics for expunged VMs retained and reported	As expected

4. UI Validation

Test Case	Notes
“Create Instance from Backup” button appears in backup list	UI shows proper dialog
Pre-filled config from backup metadata when clicked on `Configure`	All fields populated correctly
Volume details and limits correctly shown and enforced	Volume size limits shown
`Use IP Addresses from Backup` switch appears only when original VM is expunged	As expected
Multiple data volumes rendered correctly in UI form	As expected

5. Regression Tests

Test Case	Notes
Standard DeployVM API with different combinations of disk offerings, service offerings and ovverride root disk size	Pass
Backup Schedule and Delete flows	Pass
Restore Backup	Pass
Restore Volume and Attach to Instance	Pass
VM lifecycle commands	Pass

How did you try to break this feature and the system with this change?

codecov · 2024-12-25T05:20:17Z

Codecov Report

❌ Patch coverage is 40.97744% with 1256 lines in your changes missing coverage. Please review.
✅ Project coverage is 17.35%. Comparing base (70468a6) to head (226e82a).
⚠️ Report is 9 commits behind head on main.

Files with missing lines	Patch %	Lines
...loudstack/api/command/user/vm/BaseDeployVMCmd.java	47.91%	173 Missing and 39 partials ⚠️
...rg/apache/cloudstack/backup/BackupManagerImpl.java	64.09%	111 Missing and 61 partials ⚠️
.../src/main/java/com/cloud/vm/UserVmManagerImpl.java	44.25%	125 Missing and 40 partials ⚠️
...ack/api/command/user/vm/CreateVMFromBackupCmd.java	6.15%	60 Missing and 1 partial ⚠️
...rg/apache/cloudstack/backup/NASBackupProvider.java	34.40%	58 Missing and 3 partials ⚠️
...rg/apache/cloudstack/backup/dao/BackupDaoImpl.java	21.81%	42 Missing and 1 partial ⚠️
.../apache/cloudstack/backup/DummyBackupProvider.java	0.00%	42 Missing ⚠️
...che/cloudstack/backup/NetworkerBackupProvider.java	0.00%	35 Missing ⚠️
.../apache/cloudstack/backup/VeeamBackupProvider.java	5.88%	32 Missing ⚠️
...a/org/apache/cloudstack/backup/BackupDetailVO.java	18.91%	29 Missing and 1 partial ⚠️
... and 47 more

Additional details and impacted files

@@ Coverage Diff @@## main #10140 +/- ##
============================================
+ Coverage 17.17% 17.35% +0.17% - Complexity 15010 15186 +176 
============================================
Files 5869 5883 +14 Lines 521710 524302 +2592 Branches 63509 63962 +453 ============================================
+ Hits 89614 90974 +1360 - Misses 422039 423051 +1012 - Partials 10057 10277 +220

Flag	Coverage Δ
uitests	`3.63% <ø> (-0.13%)`	⬇️
unittests	`18.38% <40.97%> (+0.22%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

leo79901 · 2024-12-31T06:52:13Z

Yes, we need this !

…d data volumes before restore.

abh1sar · 2025-01-06T12:00:02Z

@blueorangutan package

blueorangutan · 2025-01-06T12:02:04Z

@abh1sar a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-01-06T13:19:25Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 11996

github-actions · 2025-01-08T14:37:51Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

github-actions · 2025-07-29T14:59:06Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

…p-latest

sureshanaparti · 2025-07-30T05:01:33Z

@blueorangutan package

blueorangutan · 2025-07-30T05:02:03Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

sureshanaparti · 2025-07-30T05:02:42Z

@blueorangutan ui

blueorangutan · 2025-07-30T05:04:08Z

@sureshanaparti a Jenkins job has been kicked to build UI QA env. I'll keep you posted as I make progress.

blueorangutan · 2025-07-30T05:24:04Z

UI build: ✔️
Live QA URL: https://qa.cloudstack.cloud/simulator/pr/10140 (QA-JID-703)

blueorangutan · 2025-07-30T07:32:48Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 14457

sureshanaparti · 2025-07-30T07:39:18Z

@blueorangutan test

blueorangutan · 2025-07-30T07:40:05Z

@sureshanaparti a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

github-actions · 2025-07-30T13:03:34Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

blueorangutan · 2025-07-31T07:01:39Z

[SF] Trillian test result (tid-13974)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 80618 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10140-t13974-kvm-ol8.zip
Smoke tests completed. 138 look OK, 8 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
ContextSuite context=TestRAMCPUResourceAccounting>:setup	`Error`	0.00	test_resource_accounting.py
test_01_list_sec_storage_vm	`Failure`	0.05	test_ssvm.py
test_03_ssvm_internals	`Failure`	0.05	test_ssvm.py
test_05_stop_ssvm	`Failure`	0.04	test_ssvm.py
test_07_reboot_ssvm	`Failure`	0.05	test_ssvm.py
test_11_destroy_ssvm	`Failure`	917.32	test_ssvm.py
ContextSuite context=TestVMWareStoragePolicies>:setup	`Error`	0.00	test_storage_policy.py
test_01_create_template	`Error`	1.14	test_templates.py
test_CreateTemplateWithDuplicateName	`Error`	1.11	test_templates.py
test_02_create_template_with_checksum_sha1	`Error`	65.65	test_templates.py
test_03_create_template_with_checksum_sha256	`Error`	65.63	test_templates.py
test_04_create_template_with_checksum_md5	`Error`	65.64	test_templates.py
test_05_create_template_with_no_checksum	`Error`	65.63	test_templates.py
ContextSuite context=TestTemplates>:setup	`Error`	240.85	test_templates.py
ContextSuite context=TestISOUsage>:setup	`Error`	0.00	test_usage.py
test_01_snapshot_usage	`Error`	1.19	test_usage.py
test_01_template_usage	`Error`	1.20	test_usage.py
test_01_volume_usage	`Error`	130.86	test_usage.py
test_06_download_detached_volume	`Error`	300.10	test_volumes.py
test_13_migrate_volume_and_change_offering	`Error`	126.36	test_volumes.py
ContextSuite context=TestIpv6Vpc>:setup	`Error`	0.00	test_vpc_ipv6.py
ContextSuite context=TestRVPCSite2SiteVpn>:setup	`Error`	0.00	test_vpc_vpn.py
ContextSuite context=TestVPCSite2SiteVPNMultipleOptions>:setup	`Error`	0.00	test_vpc_vpn.py
ContextSuite context=TestVpcRemoteAccessVpn>:setup	`Error`	0.00	test_vpc_vpn.py
ContextSuite context=TestVpcSite2SiteVpn>:setup	`Error`	0.00	test_vpc_vpn.py

sureshanaparti · 2025-07-31T07:07:44Z

@abh1sar can you check the conflicts

abh1sar · 2025-07-31T07:47:10Z

@blueorangutan package

blueorangutan · 2025-07-31T07:48:04Z

@abh1sar a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

abh1sar · 2025-07-31T08:35:54Z

Tried to create VM from backup and delete the backup before the process ends, the deletion was allowed but the VM was not restored. I think that these processes should be either blocked from executing at the same time, or be queued.
The processes are queued once they reach the agent. i.e, if the new instance is in Restoring state then backup is deleted, the agent will process the restore first and then backup delete. If the new instance is still being created/starting/stopping before restore, and the backup is deleted, delete is not blocked and restore will throw an error that backup is deleted. IMO this is not a must have, and I'll be happy to fix this as a future improvement.
Could we map it as an issue then?

Created issue #11356

blueorangutan · 2025-07-31T08:59:13Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 14488

sureshanaparti · 2025-07-31T09:10:32Z

@blueorangutan test

blueorangutan · 2025-07-31T09:12:03Z

@sureshanaparti a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2025-08-01T00:27:58Z

[SF] Trillian test result (tid-13987)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 52137 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10140-t13987-kvm-ol8.zip
Smoke tests completed. 145 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_list_sec_storage_vm	`Failure`	0.05	test_ssvm.py
test_03_ssvm_internals	`Failure`	0.04	test_ssvm.py
test_05_stop_ssvm	`Failure`	0.04	test_ssvm.py
test_11_destroy_ssvm	`Error`	3.23	test_ssvm.py
test_12_destroy_cpvm	`Error`	3.19	test_ssvm.py

abh1sar added 2 commits December 25, 2024 08:24

Initial api and service layer changes
35ed72c

Nas backup provider changes
e094e14

boring-cyborgBot added component:api component:backup component:kvm labels Dec 25, 2024

Add hypervisor, service_offering and template to backups table.
27feeda

abh1sar added 3 commits January 1, 2025 17:15

Create backup_details table and add details related to vm.
f98da37

Initial UI form
04a2deb

Save networkIds and diskOfferingIds in backup_details. Create require…
d3439ad
…d data volumes before restore.

boring-cyborgBot added component:storage component:UI labels Jan 3, 2025

Restore backup to VM plugin change for VeeamBackupProvider
753d80f

abh1sar added 3 commits January 7, 2025 02:50

UI form updates
72c943f

Create instance from backup using just the backupid
7ae078e

Remove zone and iso from DeployVMFromBackup.vue
31f89df

github-actionsBot added the status:has-conflicts label Jan 8, 2025

abh1sar added 3 commits January 9, 2025 20:57

Create multiple data volumes using the deployVirtualMachine Api
7e3f083

Create multiple data volumes for createVMFromBackup api and ui
f46fad8

Merge remote-tracking branch 'upstream/main' into instance
d941809

boring-cyborgBot added component:compute component:kubernetes component:orchestration labels Jan 9, 2025

github-actionsBot removed the status:has-conflicts label Jan 9, 2025

Fix build failure after merge
10f7beb

abh1sar added 4 commits July 29, 2025 22:04

Merge remote-tracking branch 'upstream/main' into instance-from-backu…
59aac60
…p-latest

fix backupmanagertest
629127b

Merge remote-tracking branch 'upstream/main' into instance-from-backu…
6e5fa8d
…p-latest

Merge branch 'main' into instance-from-backup
f9cf862

Merge remote-tracking branch 'upstream/main' into instance-from-backup
226e82a

abh1sar mentioned this pull request Jul 31, 2025
Disallow or block Backup deletion if create Instance from Backup is in progress #11356
Closed

abh1sar mentioned this pull request Jul 31, 2025
Backroll Plugin #8251
Draft
13 tasks

abh1sar mentioned this pull request Sep 2, 2025
Create Instance from backup on another Zone (DRaaS use case) #11560
Merged
14 tasks

sureshanaparti mentioned this pull request Sep 24, 2025
Restore from NAS backup is failing for VM with template / disks < 1 GB #11707
Closed

sureshanaparti mentioned this pull request Jan 29, 2026
Alert type fixes and some code improvements #12546
Open
14 tasks

winterhazel mentioned this pull request Apr 20, 2026
Release reserved storage resources on VM deployment failure #13048
Merged
12 tasks

Uh oh!

Conversation

abh1sar commented Dec 25, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Other Miscellaneous changes

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

1. Create Instance from Backup

2. Expunge/Unmanage VM with Backups

3. Usage Tracking

4. UI Validation

5. Regression Tests

How did you try to break this feature and the system with this change?

Uh oh!

codecovBot commented Dec 25, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

leo79901 commented Dec 31, 2024

Uh oh!

abh1sar commented Jan 6, 2025

Uh oh!

blueorangutan commented Jan 6, 2025

Uh oh!

blueorangutan commented Jan 6, 2025

Uh oh!

github-actionsBot commented Jan 8, 2025

Uh oh!

github-actionsBot commented Jul 29, 2025

Uh oh!

sureshanaparti commented Jul 30, 2025

Uh oh!

blueorangutan commented Jul 30, 2025

Uh oh!

sureshanaparti commented Jul 30, 2025

Uh oh!

blueorangutan commented Jul 30, 2025

Uh oh!

blueorangutan commented Jul 30, 2025

Uh oh!

blueorangutan commented Jul 30, 2025

Uh oh!

sureshanaparti commented Jul 30, 2025

Uh oh!

blueorangutan commented Jul 30, 2025

Uh oh!

github-actionsBot commented Jul 30, 2025

Uh oh!

blueorangutan commented Jul 31, 2025

Uh oh!

sureshanaparti commented Jul 31, 2025

Uh oh!

abh1sar commented Jul 31, 2025

Uh oh!

blueorangutan commented Jul 31, 2025

Uh oh!

abh1sar commented Jul 31, 2025

Uh oh!

blueorangutan commented Jul 31, 2025

Uh oh!

sureshanaparti commented Jul 31, 2025

Uh oh!

blueorangutan commented Jul 31, 2025

Uh oh!

blueorangutan commented Aug 1, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

12 participants

abh1sar commented Dec 25, 2024•
edited
Loading

codecovBot commented Dec 25, 2024•
edited
Loading