Introduce new backup provider (KBOSS)

[JoaoJandre]

Description

This PR adds a new incremental backup provider for KVM. This provider does not need any external plugins or integrations. The design document which goes into details of the implementation can be found on https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=406622120.

The validation process which is detailed in the design document will be added to this PR soon. The validation process is already in this PR.
The file extraction process will be added in a later PR.

This PR adds a few new APIs:

• Since we are proposing a solution where ACS is the provider, we will not have an external ID to inform on the importBackupOffering. Thus, a new API called createBackupOffering will be created, which will define a new backup offering from scratch. It will have the following parameters:

  Parameter	Description	Default Value	Required
  name	Specifies the name of the offering	-	Yes
  description	The description of the backup offering	-	Yes
  compress	Specifies whether the offering supports backup compression	false	No
  validate	Specifies whether the offering supports backup validation	false	No
  validationSteps	A comma-separated list of which validation steps should be performed.	screenshot	No
  validationInterval	Specifies the interval (in hours) between two validations of the same backup	false	No
  allowQuickRestore	Specifies whether the offering supports quick restore	false	No
  allowExtractFile	Specifies whether the offering supports file extraction from backups	false	No
  compressionlibrary	Compression library, for offerings that support compression. Accepted values are zstd and zlib. If the image only supports zlib, it will be used regardless of this parameter.	zstd	No
  backupchainsize	Backup chain size for backups created with this offering.	-	No
  zoneId	The zone ID for the offering	-	Yes
  userDrivenBackups	Whether users are allowed to create adhoc backups and backup schedules	-	Yes
  domainIds	the ID of the containing domain(s), null for public offerings	-	No

• The listBackupServiceJobs has the following parameters

  Parameter	Description
  id	List only the job with the specified ID
  backupid	List jobs associated with the specified backup
  hostid	List jobs associated with the specified host. When this parameter is provided, the executing parameter is implicit
  zoneid	List jobs associated with the specified zone
  type	List jobs of the specified type. Accepts StartCompression, FinalizeCompression and BackupValidation
  executing	List jobs that are currently executing
  scheduled	List jobs scheduled to run in the future

  By default, lists all jobs that have not been removed.

• The API downloadValidationScreenshot was added to allow downloading the images generated during the screenshot step. The API has the following parameter:

  Parameter	Description
  backupId	ID of the backup whose screenshot should be downloaded

• The API finishBackupChain was added to allow users to force a backup chain on a VM to end. This API may be called on VMs that are on the BackupError state and it will do a best-effort attempt at normalizing the VM and returning it to its original state, as well as finishing the backup chain on the VM. The API has the following parameter:

  Parameter	Description
  vmId	ID of the VM to finish the chain.

The PR also adds parameters to the following APIs:

• The isolated parameter was added to the createBackup and createBackupSchedule APIs
• The quickRestore parameter was added to the restoreBackup, restoreVolumeFromBackupAndAttachToVM and createVMFromBackup APIs
• The hostId parameter was added to the restoreBackup and restoreVolumeFromBackupAndAttachToVM APIs, which can only be used by root admins and only when quick restore is true.

New settings were also added:

Configuration	Description	Default Value
backup.chain.size	Determines the max size of a backup chain. If cloud admins set it to 1 , all the backups will be full backups. With values lower than 1, the backup chain will be unlimited, unless it is stopped by another process. Please note that unlimited backup chains have a higher chance of getting corrupted, as new backups will be dependent on all of the older ones.	8
kboss.timeout	Timeout, in seconds, to execute KBOSS commands. After the command times out, the Management Server will still wait for another kboss.timeout seconds to receive a response from the Agent.	43200
backup.compression.task.enabled	Determines whether the task responsible for scheduling compression jobs is active. If not, compression jobs will not run	true
backup.compression.max.concurrent.operations.per.host	Maximum number of concurrent compression jobs. Compression finalization jobs ignore this setting	5
backup.compression.max.concurrent.operations	Maximum number of compression jobs that can be executed at the same time in the zone. Values lower than 1 disable the limit.	10
backup.compression.max.job.retries	Maximum number of attempts for executing compression jobs	2
backup.compression.retry.interval	Interval, in minutes, between attempts to run compression jobs	60
backup.compression.timeout	Timeout, in seconds, for running compression jobs	28800
backup.compression.minimum.free.storage	Minimum required available storage to start the backup compression process. This setting accepts a real number that is multiplied by the total size of the backup to determine the necessary available space. By default, the storage must have the same amount of available space as the space occupied by the backup.	1
backup.compression.coroutines	Number of coroutines used for the compression process, each coroutine has its own thread	1
backup.compression.rate.limit	Compression rate limit, in MB/s. Values less than 1 disable the limit	0
backup.validation.task.enabled	Determines whether the task responsible for scheduling validation jobs is active. If it is not active, validation jobs will not run.	true
backup.validation.interval	Interval, in hours, between two validations of the same backup	24
backup.validation.max.concurrent.operations	Maximum number of validation jobs that can be executed at the same time in the zone. Values lower than 1 disable the limit.	10
backup.validation.max.concurrent.operations.per.host	Maximum number of validation jobs that can be executed at the same time on each host. Values lower than 1 disable the limit.	1
backup.validation.boot.default.timeout	Default timeout, in seconds, for the boot validation step	240
backup.validation.script.default.timeout	Default timeout, in seconds, for the script validation step	60
backup.validation.screenshot.default.wait	Default waiting time, in seconds, before executing the VM screenshot	60
backup.validation.end.chain.on.fail	If true, ends the current backup chain if backup validation fails and the backup belongs to that chain.	true
enforce.resource.limit.on.backup.validation.vm	If true, the creation of validation VMs is bound by the account/domain resource limits.	false

Two new host-level configurations, configured in agent.properties, were added: backup.validation.max.concurrent.operations.per.host and backup.compression.max.concurrent.operations.per.host. When configured, they override the values set in the global configurations with the same names.

Six new VM configurations were added:

Configuration	Description
backupValidationCommand	Command to be executed during the command execution step. The step will not run if this configuration does not have a value
backupValidationCommandArguments	Arguments to be passed to the command executed during the command execution step
backupValidationCommandExpectedResult	Expected result of the command execution, coded in Base64. If not provided, the process exit code will be checked for value 0
backupValidationCommandTimeout	Timeout for the command executed during the command execution step. Overrides the global configuration
backupValidationScreenshotWait	Waiting time before executing the screenshot during the screenshot step. Overrides the global configuration
backupValidationBootTimeout	Timeout for the VM boot. Overrides the global configuration

By default, the VM configurations backupValidationCommandTimeout, backupValidationScreenshotWait, and backupValidationBootTimeout are read-only for regular users. This behavior can be changed through the global configuration user.vm.readonly.details.

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)
• Build/CI
• Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

• Major
• Minor

Bug Severity

• BLOCKER
• Critical
• Major
• Minor
• Trivial

Screenshots (if appropriate):

How Has This Been Tested?

Tests related to disk-only VM snapshots

N	Test	Result
1	Take disk-only VM snapshot	ok
2	Take disk-only VM snapshot again	ok
3	Stop VM, revert to snapshot 2, start VM	Correct deltas found in the VM volume chain
4	Stop VM, revert to snapshot 1, start VM	Correct deltas found in the VM volume chain
5	Take disk-only VM snapshot	ok
6	Remove disk-only VM snapshot 1	Marked as removed, not removed from storage
7	Remove disk-only VM snapshot 3	Merged with current volume
8	Remove disk-only VM snapshot 2	Removed, snap 1 merged with the current volume

Basic tests with backup

Using backup.chain.size=3

N	Test	Result
1	With the VM stopped, I created a backup (b1)	Full backup created
2	Started VM, wrote data, created a second backup (b2)	Incremental backup created
3	Stopped the VM, went back to backup 1, started	Ok, VM without data
4	Stopped the VM, went back to backup 2, started	Ok, data from the backup created in test 2 present
5	Created 4 backups (b3, b4, b5, b6)	b3 was a full, b4 and b5 incremental, b6 full
6	Removing the last backup (b6)	the delta on the primary was merged with the volume
7	Removed backups b4 and b5	they were marked as removed, but not deleted from storage
8	Batch removing the remaining backups	Ok, all removed
9	Created a new backup	ok
10	Detached the VM from the offer	Deltas were merged on primary
11	Removing this last backup	ok

Interactions with other functionalities

I created a new VM with a root disk and a data disk for the tests below.

N	Test	Result
1	Took a new backup and migrated the VM	ok
2	Migrated the VM + one of the volumes	ok, the migrated volume had no delta on the primary, the other volume still had a delta
3	Took a new backup	For the volume that was not migrated the backup was incremental, for the migrated volume, it was a full backup
4	I took 2 backups	OK, the finished normally
5	Try restoring one of the backups from before the migration	OK
6	Created file 1, created backup b1	OK
7	Created file 2, created VM snap s1	OK
8	Created file 3, created VM snap s2	OK
9	Created file 4, created backup b2	OK
10	Created file 5, created backup b3	OK
11	Stopped the VM, restored VM snap s1, started	Files 1 and 2 present
12	Stopped the VM, restored VM snap s2, started	Files 1, 2 and 3 present
13	Removed VM snapshots	ok
14	Restored backup b1, started	file 1 present
15	Restored backup b2, started	files 1, 2, 3 and 4 present
16	Restored backup b3, started	files 1, 2, 3, 4 and 5 present
17	Took a new backup b4	ok
18	Attached a new volume, wrote data, took a backup b5	ok
19	Stopped the VM, restored backup b4, started the VM	the new volume was not affected by the restoration
20	Detached the volume, restored backup b5	a new volume was created and attached to the VM, the files were there
21	Created a backup	ok
22	Created a volume snapshot	OK
23	Revert volume snapshot	I verified that the delta on the primary left by the last backup was removed

Configuration Tests

• I changed the value of the backup.compression.task.enabled setting and verified that no new jobs were started. I verified that when returned to true, they were executed.
• I changed the value of the backup.compression.max.concurrent.compressions.per.host setting and verified that the number of jobs executed simultaneously for each host was relative to the value of the setting. I also verified that the value -1 does not limit the number of jobs executed by the host.
• I verified that the number of retries respects the backup.compression.max.job.retries setting.
• I verified that the time between retries respects the backup.compression.retry.interval setting.
• I changed the value of the backup.compression.minimum.free.storage setting and verified that the job failed if there was not enough free space.
• I changed the value of the backup.compression.coroutines setting and verified that the value passed to qemu-img was reflected.
• I changed the value of the backup.compression.rate.limit setting and verified that the value was passed to qemu-img.

Compression Tests

Tests performed with an offer that provides compressed backups support

Test	Result
Create full backup	Backup created and compressed
Create incremental backup	Backup created and compressed
Create 10 backups of the same machine	Backups created sequentially, but compressed in parallel

Validation tests

Test Case	Result
Attach a backup offer with screenshot validation and perform a backup	Backup completed successfully
Attach a backup offer with wait for boot validation and perform a backup	Boot was detected and backup completed successfully
Attach a backup offer with wait for boot and command execution validation, but do not specify a command, then perform a backup	Backup completed successfully, but no command was executed
Attach a backup offer with wait for boot and command execution validation, configuring the command echo	Backup completed successfully
Attach a backup offer with wait for boot and command execution validation, configuring the command echo with argument ababa	Backup completed successfully
Attach a backup offer with wait for boot and command execution validation, configuring the command echo, argument ababa, and expected output absdui	Backup failed, expected output YWJhYmEK
Attach a backup offer with wait for boot and command execution validation, configuring the command echo, argument ababa, and expected output YWJhYmEK	Backup completed successfully
Attach a backup offer with screenshot, wait for boot, and command execution validation, configuring the command echo, argument ababa, and expected output YWJhYmEK	Backup completed successfully
Download a validation screenshot	URL generated and download successful
Attempt to download a screenshot from a backup without a screenshot	URL was not generated

Tests with restoreVolumeFromBackupAndAttachToVM

N	Test	Result
1	Restore volume A from VM 1 to the same VM while it is stopped	New volume created, restored, and attached
2	Restore volume B from VM 1 to the same VM while it is running	New volume created, restored, and attached
3	Restore volume B from VM 1 to VM 2 while it is running	New volume created, restored, and attached
4	Restore volume A from VM 1 to VM 2 while it is running, even though the VM was deleted	New volume created, restored, and attached
5	Restore a volume to a VM using quickrestore	New volume created, attached, and consolidated with the backup
6	Restore a volume to a stopped VM using quickrestore and specifying the hostId	New volume created, attached, VM started on the specified host, and volume consolidated

Tests with restoreBackup

N	Test	Result
1	Restore VM without quickrestore	OK
2	Restore VM with quickrestore	Volumes restored, VM started, and volumes consolidated
3	Restore VM with quickrestore and specifying hostId	Volumes restored, VM started on the specified host, and volumes consolidated
4	Detach a volume from the VM and repeat test 3	Detached volume duplicated, attached to the VM, restored, and VM started on the host, volumes consolidated

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 3.41%. Comparing base (24fd90a) to head (5de00c8).
⚠️ Report is 1 commits behind head on main.

❗ There is a different number of reports uploaded between BASE (24fd90a) and HEAD (5de00c8). Click for more details.

HEAD has 1 upload less than BASE

Flag	BASE (24fd90a)	HEAD (5de00c8)
unittests	1	0

Additional details and impacted files

@@ Coverage Diff @@## main #12758 +/- ##
=============================================
- Coverage 19.50% 3.41% -16.09% 
=============================================
Files 6303 487 -5816 Lines 569288 41860 -527428 Branches 69792 7910 -61882 =============================================
- Hits 111028 1429 -109599 + Misses 446106 40232 -405874 + Partials 12154 199 -11955 

Flag	Coverage Δ
uitests	3.41% <ø> (-0.01%)	⬇️
unittests	?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[yadvr]

@JoaoJandre just heads up - my colleagues have been working on an incremental backup feature for NAS B&R (using nbd/qemu bitmap tracking & checkpoints). We're also working on a new Veeam-KVM integration for CloudStack whose PR may may be out soon. My colleagues can further help review and advise on this.

/cc @weizhouapache@abh1sar@shwstppr@sureshanaparti@DaanHoogland@harikrishna-patnala

Just my 2cents on the design & your comments - NAS is more than just NFS, but any (mountable) shared storage such as CephFS, cifs/samba etc. Enterprise users usually don't want to mix using secondary storage with backup repositories, which is why NAS B&R introduced a backup-provider agnostic concept of backup repositories which can be explored by other backup providers.

[JoaoJandre]

Just my 2cents on the design & your comments - NAS is more than just NFS, but any (mountable) shared storage such as CephFS, cifs/samba etc.

At the time of writing that part, I believe it was only NFS that was supported. I'll update the relevant part.

Enterprise users usually don't want to mix using secondary storage with backup repositories, which is why NAS B&R introduced a backup-provider agnostic concept of backup repositories which can be explored by other backup providers.

The secondary storage selector feature (introduced in 2023 by #7659) allows you to specialize secondary storages. This PR extended the feature so that you may also create selectors for backups.

[abh1sar]

Hi Joao,

This looks promising. Incremental backups, quick restore and file restore features have been missing from CloudStack KVM.

I am having trouble understanding some of the design choices though:

1. What’s the reason behind strong coupling with secondary storage?

   • I am wondering if the Backup Repository will provide a more flexible alternative. The user would be free to add an external storage server or use the secondary storage by simply adding it as a backup repository? It will be very easy for user to have multiple backup repository attached to multiple backup offerings which can be assigned to instance as required.

     This will also be consistent with other backup providers like Veeam and NAS which have the concept of backup repository.

     The backup repository feature also comes with a separate capacity tracking and email alerts.

   • If a secondary storage is needed just for backup’s purpose, how will it be ensured that templates and snapshots are not copied over to it?

2. About Qemu compression

   • Have you measured / compared the performance of qemu-img compression with other compression methods?
   • As I understand, qemu-img compresses the qcow2 file at a cluster granularity (usually 64kb). That might not fare well when compared to storage level compression. In production environments, the operator might choose to have compression at the storage layer if they are using an enterprise storage like NetApp. Even something open source like ZFS might perform better than qemu-img compress due to the granularity limitation that qemu compression has.
   • I am making this point because the compression part is introducing a fair bit of complexity due to the interaction with SSVM, and I am just wondering if the gains are worth the trouble and should compression be offloaded to the storage completely.

3. Do we need a separate backup offering table and api?

   • Why not add column or details to backup_offering or backup_offering_details? Other offerings can also benefit from these settings.

4. What’s the reason behind using virDomainSnapshotCreate to create backup files and not virDomainBackupBegin like incremental volume snapshots and NAS backup?

   • Did you face any issues with checkpoints and bitmaps?

[JoaoJandre]

Hi Joao,

Hello, @abh1sar

This looks promising. Incremental backups, quick restore and file restore features have been missing from CloudStack KVM.

I am having trouble understanding some of the design choices though:

1. What’s the reason behind strong coupling with secondary storage?
* I am wondering if the Backup Repository will provide a more flexible alternative. The user would be free to add an external storage server or use the secondary storage by simply adding it as a backup repository? It will be very easy for user to have multiple backup repository attached to multiple backup offerings which can be assigned to instance as required.

I don't see why we should force the coupling of backup offerings with backup repositories, what is the benefit?

 This will also be consistent with other backup providers like Veeam and NAS which have the concept of backup repository.
The backup repository feature also comes with a separate capacity tracking and email alerts.

The secondary storage also has both features. Although the capacity is not reported to the users currently.

 * If a secondary storage is needed just for backup’s purpose, how will it be ensured that templates and snapshots are not copied over to it?

The secondary storage selectors feature (introduced in 2023 through #7659) allows you to specialize secondary storages. Quoting from the PR description: "This PR aims to add the possibility to direct resources (Volumes, Templates, Snapshots and ISOs) to a specific secondary storage through rules written in JavaScript that will only affect new allocated resources". For a few years it has been possible to have secondary storages that only receive snapshots or templates for example. This PR introduces the possibility to add selectors for backups, so that you have secondary storages that are specific for backups.

Furthermore, my colleagues are working on a feature to allow using alternative secondary storage solutions, such as CephFS, iSCSI and S3, while preserving compatibility with features destined to NFS storages. This feature may be extended in the future to allow essentially any type of secondary storage. Thus, the flexibility for secondary storages will soon grow.

2. About Qemu compression
* Have you measured / compared the performance of qemu-img compression with other compression methods?

Using any other type of backup-level compression will be worse then using qemu-img compression. This is because when restoring the backup, we must have access to the whole backing chain. If we use other types of compression, we will have to decompress the whole chain before restoring. Using qemu-img, the backing files are still valid and do not need to be decompressed, we actually never have to decompress ever. This is the great benefit of using qemu-img.

In any case, here is a brief comparison of using qemu-img with the zstd library and 8 threads and using the pigz implementation of multi-threaded compression, also using 8 threads. The original file is the root volume of a VM that I use.

Command	Time	Original file size	Final file size
qemu-img convert -c -p -W -m 8 -f qcow2 -O qcow2 -o compression_type=zstd	real 3m51.944s - user 16m11.970s - sys 4m14.987s	43 G	35G
pigz -p8	real 6m13.799s - user 44m33.300s - sys 1m54.801s	43G	34G
pigz --zip -p8	real 6m2.729s - user 44m38.401s - sys 1m47.663s	43G	34G

Compression using qemu-img was a lot faster, with a bit smaller compression ratio. Furthermore, we have to consider that the qemu-img compressed image can be used as-is, while the other images must be decompressed, further adding to the processing time of backing up/restoring a backup.

 * As I understand, qemu-img compresses the qcow2 file at a cluster granularity (usually 64kb). That might not fare well when compared to storage level compression. In production environments, the operator might choose to have compression at the storage layer if they are using an enterprise storage like NetApp. Even something open source like ZFS might perform better than qemu-img compress due to the granularity limitation that qemu compression has.

The compression feature is optional, if you are using storage-level compression, you probably will not use backup-level compression. However, many environments do not have storage-level compression, thus having the possibility of backup-level compression is still very interesting.

 * I am making this point because the compression part is introducing a fair bit of complexity due to the interaction with SSVM, and I am just wondering if the gains are worth the trouble and should compression be offloaded to the storage completely.

The compression does not add any interaction with the SSVM.

3. Do we need a separate backup offering table and api?
* Why not add column or details to backup_offering or backup_offering_details? Other offerings can also benefit from these settings.

I did not want to add dozens of parameters to the import backup offering API which are only really going to be used for one provider. This way, the original design of the API is preserved.

Furthermore, you may note that the APIs are intentionally not called createKnibBackupOffering, but createNativeBackupOffering. If other native providers want to use these offerings, they may do so by extending their implementations.

4. What’s the reason behind using virDomainSnapshotCreate to create backup files and not virDomainBackupBegin like incremental volume snapshots and NAS backup?
* Did you face any issues with checkpoints and bitmaps?

There are two main issues with using bitmaps:

1. They are prone to corruption, while this can be mitigated in some ways, since the incremental volume snapshot feature was added, we have noticed multiple cases of bitmap corruption with different causes. It is possible to detect the corruption and delete corrupt bitmaps, but this would add more complexity to the feature.
2. Using bitmaps is not compatible with the file-based incremental VM snapshot feature added in File-based disk-only VM snapshot with KVM as hypervisor #10632. After some internal discussion and feedback from users, we have come to the conclusion that being able to use both the incremental VM snapshot and backup features at the same time is very interesting.

At the end of the day, this PR adds a new backup provider option for users. They will be free to choose the provider that best fits their needs. This is one of the reasons why it was done as a new backup provider; KNIB and other backup providers do not have to cancel each-other out.

[abh1sar]

Hi @JoaoJandre
Please find my response inline.

I don't see why we should force the coupling of backup offerings with backup repositories, what is the benefit?

It has a big benefit for use cases where someone wants multiple tiers of backup storage with different cost and recovery characteristics. For example, long-term archival backups might go to cheap cold storage like tape, while backups that require faster recovery (better RTO) may use more expensive SSD-backed storage.

You can have multiple backup offerings for different use cases and VMs can be attached to the required offerings.
Furthermore, the backup storage usage is tracked per Backup Offering. So, users can use different offerings as per requirement and get charged accordingly.

The secondary storage also has both features. Although the capacity is not reported to the users currently.

Capacity is just for Admins for Backup Repository also, so that is not the issue.
But the Secondary Storage capacity tracking and alerts would be mixed up for Templates, Snapshots, Volumes and Backups.
With backup repositories, you will get separate capacity tracking and alerts just for the Backup Infrastructure.

Compression using qemu-img was a lot faster, with a bit smaller compression ratio. Furthermore, we have to consider
that the qemu-img compressed image can be used as-is, while the other images must be decompressed, further adding
to the processing time of backing up/restoring a backup.

It does have its benefits, but do keep in mind that the decompression cost will be paid by Reads. Also if someone is using a restored VM, they might see the physical size increase disproportionately to the actual data written due to decompression. It's still a useful feature and it's good that it is optional.

I did not want to add dozens of parameters to the import backup offering API which are only really going to be used for one provider. This way, the original design of the API is preserved.

If there is possibility these parameters can be added for other providers they should be added to the existing API.
Some of these like allowQuickRestore, allowExtractFile might as well be added for other providers such as Veeam and NAS in the future.

Furthermore, you may note that the APIs are intentionally not called createKnibBackupOffering, but createNativeBackupOffering. If other native providers want to use these offerings, they may do so by extending their implementations.

• What defines a provider as Native?
• Do the native offerings differ in functionality or usage to the existing BackupOfferings?
• How will this look like in the UI? Will the user see Backup Offering and another Native Backup Offering?
• Again, coupling an offering to storage has its benefits which current Backup Offerings already have.

[JoaoJandre]

Hello, @abh1sar

It has a big benefit for use cases where someone wants multiple tiers of backup storage with different cost and recovery characteristics. For example, long-term archival backups might go to cheap cold storage like tape, while backups that require faster recovery (better RTO) may use more expensive SSD-backed storage.

You can have multiple backup offerings for different use cases and VMs can be attached to the required offerings. Furthermore, the backup storage usage is tracked per Backup Offering. So, users can use different offerings as per requirement and get charged accordingly.

You can have exactly that using this implementation. Essentially backup repositories and secondary storages are the same, with different names. Using selectors, you can have offerings that are tied to specific secondary storages, or you can have offerings that go into multiple storages, it was made to be flexible.

Capacity is just for Admins for Backup Repository also, so that is not the issue. But the Secondary Storage capacity tracking and alerts would be mixed up for Templates, Snapshots, Volumes and Backups. With backup repositories, you will get separate capacity tracking and alerts just for the Backup Infrastructure.

Again, you can have dedicated secondary storages using selectors.

If there is possibility these parameters can be added for other providers they should be added to the existing API. Some of these like allowQuickRestore, allowExtractFile might as well be added for other providers such as Veeam and NAS in the future.

While I was not the one that introduced the backup framework, looking at its design, it was clearly intended to have as little configuration as possible on the ACS side while leaving these details to the backup provider. If we add these parameters to the import backup offering API, I'm sure a lot of users will be confused when they do nothing for Veeam and Dell Networker. I did not intend to warp the original design of configuring the offerings on the provider side and only importing it to ACS.

This is why I created the concept of native offerings. As with KNIB (and NAS) the provider is ACS, and thus the configurations can still be made on the "backup provider", and the import API will follow the same logic it always had.

* What defines a provider as `Native`?

A provider is native if the backup implementation is made by ACS. Thus, the current native providers would be NAS and KNIB. Veeam and Networker are external providers.

* Do the native offerings differ in functionality or usage to the existing BackupOfferings?

They are used to configure the details that would be configured in the backup provider if it was an external provider, such as Veeam for example.

* How will this look like in the UI? Will the user see `Backup Offering` and another `Native Backup Offering`?

I have yet to add it to the GUI, if you have any suggestions for the GUI design, you are free to give your opinion. The GUI for the native backup offerings will be added in the future.

* Again, coupling an offering to storage has its benefits which current Backup Offerings already have.

Again, you absolutely can do this with KNIB. But you may also choose not to do it, its flexible.

[weizhouapache]

@JoaoJandre
Although I’m not fully convinced that the mixed use of Secondary Storage is a good approach, I appreciate the work you’ve put into this. It would probably have been better to discuss the idea with the community earlier in the process, rather than presenting the design document and PR after most of the development had already been completed.

Would it be possible to change the name Native to something clearer? For example, "Secondary Storage Backup" might be more descriptive and easier for end users to understand. The term "Native" is not commonly used in CloudStack.

[github-actions]

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

[JoaoJandre]

Would it be possible to change the name Native to something clearer? For example, "Secondary Storage Backup" might be more descriptive and easier for end users to understand. The term "Native" is not commonly used in CloudStack.

I don't see any issue with "Native", and I'm not claiming that KNIB is the only "Native" provider. As I explained before:

• Native = CloudStack is the backup provider, implementing all of the processes necessary for the backup to be executed.
• External = CloudStack integrates an external solution that is responsible for the backups, such as Veeam and Dell Networker.

This PR does not remove or substitute anything, the provider I am proposing is a new option for users to use.

[JoaoJandre]

Also. The Validation was added in the last commits. All the features that were meant to be included in this PR are already here.

[weizhouapache]

Would it be possible to change the name Native to something clearer? For example, "Secondary Storage Backup" might be more descriptive and easier for end users to understand. The term "Native" is not commonly used in CloudStack.

I don't see any issue with "Native", and I'm not claiming that KNIB is the only "Native" provider. As I explained before:

• Native = CloudStack is the backup provider, implementing all of the processes necessary for the backup to be executed.
• External = CloudStack integrates an external solution that is responsible for the backups, such as Veeam and Dell Networker.

This PR does not remove or substitute anything, the provider I am proposing is a new option for users to use.

We do not use the term "Native" in CloudStack. Plugin names are typically based on the name of the external device or backup repository they integrate with.

Please follow the same convention.

[bernardodemarco]

Hello, @weizhouapache, all

We do not use the term "Native" in CloudStack

Yes, we currently do not have any APIs using the term Native. But, this does not imply or require us to avoid using this expression when it is suitable.

Plugin names are typically based on the name of the external device or backup repository they integrate with.

Yes, that is correct. In the case of the new backup plugin, it is fully implemented by ACS, including backup creation, chain management, compression, validation and other related processes. Therefore, its name is fully coherent with its functionality. In other words, KNIB (KVM Native Incremental Backup) stands for:

• KVM: provides support for the KVM hypervisor;
• Native: Fully native to Apache CloudStack, i.e., all processes are implemented by the cloud platform orchestrator itself;
• Incremental Backup: Incremental backups are supported.

Please follow the same convention.

Sorry, but is this a established convention? I am not aware of any guideline stating that the term Native is not supported.

Again, if there is any semantic inconsistency with the plugin name KNIB (KVM Native Incremental Backup), we can address it. However, the plugin supports incremental backups, for KVM, natively in ACS; thus, the name accurately reflects what it delivers. It does not promote any concept or capability that is not actually provided.

---

@JoaoJandre, btw, thanks for all the effort you put into designing and implementing this backup plugin. It will certainly be a stable and efficient solution, with strong adoption across cloud providers and users.

[weizhouapache]

Hello, @weizhouapache, all

We do not use the term "Native" in CloudStack

Yes, we currently do not have any APIs using the term Native. But, this does not imply or require us to avoid using this expression when it is suitable.

Plugin names are typically based on the name of the external device or backup repository they integrate with.

Yes, that is correct. In the case of the new backup plugin, it is fully implemented by ACS, including backup creation, chain management, compression, validation and other related processes. Therefore, its name is fully coherent with its functionality. In other words, KNIB (KVM Native Incremental Backup) stands for:

• KVM: provides support for the KVM hypervisor;
• Native: Fully native to Apache CloudStack, i.e., all processes are implemented by the cloud platform orchestrator itself;
• Incremental Backup: Incremental backups are supported.

Please follow the same convention.

Sorry, but is this a established convention? I am not aware of any guideline stating that the term Native is not supported.

Again, if there is any semantic inconsistency with the plugin name KNIB (KVM Native Incremental Backup), we can address it. However, the plugin supports incremental backups, for KVM, natively in ACS; thus, the name accurately reflects what it delivers. It does not promote any concept or capability that is not actually provided.

@JoaoJandre, btw, thanks for all the effort you put into designing and implementing this backup plugin. It will certainly be a stable and efficient solution, with strong adoption across cloud providers and users.

I don’t want to continue debating this topic, so I’ll just summarize my view:

• The mixed use of Secondary Storage is not a good approach.
• The term "Native" is unclear and may confuse users. It would be better to follow the existing naming conventions used for other network/storage/backup plugins and choose a more descriptive name.
• Please move API/DB/Java classes into the plugin as much as possible to keep the implementation isolated.

[blueorangutan]

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 18535

[shwstppr]

@blueorangutan test

[blueorangutan]

@shwstppr a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

[blueorangutan]

[SF] Trillian test result (tid-16530)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 54149 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr12758-t16530-kvm-ol8.zip
Smoke tests completed. 155 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

[winterhazel]

@JoaoJandre here are some more reviews. These are mostly trivial, so they can be addressed in a follow-up PR.

[github-actions]

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

[JoaoJandre]

@blueorangutan package

[blueorangutan]

@JoaoJandre a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 18551

[weizhouapache]

@blueorangutan test

[blueorangutan]

@weizhouapache a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

[blueorangutan]

[SF] Trillian Build Failed (tid-16549)

[weizhouapache]

@blueorangutan test

[blueorangutan]

@weizhouapache a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

[winterhazel]

Looks good. I left a few more trivial remarks that I noticed during my testing, which can be addressed in a follow-up PR.

In addition to @bernardodemarco's tests in #12758 (review), I also tested the following operations for local and shared mountpoint primary storages:

Backup creation

• Create full backups,
• Create incremental backups.
• Create backups with quiesce.
• Backup schedules.

Backup restore

• Restore full backups.
• Restore incremental backups.
• Restore backups using quick restore.
• Restore and attach backup volumes to VMs.
• Restore and attach backup volumes to VMs, using quick restore.
• Create VMs from backups.
• Create VMs from backups, using quick restore.

Backup chain finalization

• Finalization of backup chains via finishBackupChain.
• Normalization of VMs in BackupError via finishBackupChain.
  • Due to a BackupException before the backup process started in the Agent.
  • Because the Agent was unable to send an answer to the Management Server after the backup ended.

Backup compression

• Backup compression with zstd.

Backup validation

• Backup validation with wait_for_boot and screenshot steps.

Backup deletion

• Delete full backups.
• Delete incremental backups.
• Attempt to delete backups while compression or validation jobs are active for it (the deletion fails).

[winterhazel]

@blueorangutan package

[blueorangutan]

@winterhazel a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 18556

[blueorangutan]

[SF] Trillian test result (tid-16550)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 54486 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr12758-t16550-kvm-ol8.zip
Smoke tests completed. 140 look OK, 16 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
ContextSuite context=TestIpv4Routing>:setup	Error	0.00	test_ipv4_routing.py
test_01_native_to_native_network_migration	Error	4.90	test_migration.py
test_05_isolated_network_full_lifecycle	Error	7.53	test_network_extension_namespace.py
test_08_custom_action_policy_based_routing	Error	7.65	test_network_extension_namespace.py
test_11_ipv6_dualstack_isolated_network	Error	8.15	test_network_extension_namespace.py
ContextSuite context=TestNonStrictAffinityGroups>:setup	Error	0.00	test_nonstrict_affinity_group.py
ContextSuite context=TestNetworkPermissions>:setup	Error	0.00	test_network_permissions.py
ContextSuite context=TestSharedNetworkWithConfigDrive>:setup	Error	50.86	test_network.py
ContextSuite context=TestIsolatedNetworksPasswdServer>:setup	Error	0.00	test_password_server.py
test_CRUD_operations_userdata	Error	19.04	test_register_userdata.py
test_deploy_vm_with_registered_userdata	Error	3.80	test_register_userdata.py
test_deploy_vm_with_registered_userdata_with_override_policy_allow	Error	3.82	test_register_userdata.py
test_deploy_vm_with_registered_userdata_with_override_policy_append	Error	3.89	test_register_userdata.py
test_deploy_vm_with_registered_userdata_with_override_policy_deny	Error	3.66	test_register_userdata.py
test_deploy_vm_with_registered_userdata_with_params	Error	3.76	test_register_userdata.py
test_link_and_unlink_userdata_to_template	Error	3.58	test_register_userdata.py
test_user_userdata_crud	Error	3.69	test_register_userdata.py
ContextSuite context=TestRouterDHCPHosts>:setup	Error	0.00	test_router_dhcphosts.py
ContextSuite context=TestRouterDHCPOpts>:setup	Error	0.00	test_router_dhcphosts.py
ContextSuite context=TestRouterDns>:setup	Error	0.00	test_router_dns.py
ContextSuite context=TestRouterDnsService>:setup	Error	0.00	test_router_dnsservice.py
test_01_RVR_Network_FW_PF_SSH_default_routes_egress_true	Error	0.16	test_routers_network_ops.py
test_02_RVR_Network_FW_PF_SSH_default_routes_egress_false	Error	0.14	test_routers_network_ops.py
test_03_RVR_Network_check_router_state	Error	0.14	test_routers_network_ops.py
ContextSuite context=TestSharedFSLifecycle>:setup	Error	0.00	test_sharedfs_lifecycle.py
test_01_ssl_offloading_isolated_network	Error	0.21	test_ssl_offloading.py
ContextSuite context=TestVmAutoScaling>:setup	Error	0.00	test_vm_autoscaling.py
test_04_deploy_vnf_appliance	Error	102.63	test_vnf_templates.py

[winterhazel]

I went through the test failures in #12758 (comment) and checked that they are unrelated to the changes in here.

• ContextSuite context=TestIpv4Routing>:setup | Error | 0.00 | test_ipv4_routing.py -- unrelated. Failed due to an attempt to create an ASN range that conflicts with an existing one. This PR does not touch this part of the code.
• test_01_native_to_native_network_migration | Error | 4.90 | test_migration.py -- unrelated. Failed due to an attempt to create a non-VPC network using an offering for VPC networks. This PR does not touch this part of the code.
• test_05_isolated_network_full_lifecycle | Error | 7.53 | test_network_extension_namespace.py -- unrelated. Failed due to an attempt to create a non-VPC network using an offering for VPC networks.
• test_08_custom_action_policy_based_routing | Error | 7.65 | test_network_extension_namespace.py -- unrelated. Failed due to an attempt to create a non-VPC network using an offering for VPC networks.
• test_11_ipv6_dualstack_isolated_network | Error | 8.15 | test_network_extension_namespace.py -- unrelated. Failed due to an attempt to create a non-VPC network using an offering for VPC networks.
• ContextSuite context=TestNonStrictAffinityGroups>:setup | Error | 0.00 | test_nonstrict_affinity_group.py -- unrelated. Failed due to an attempt to create a non-VPC network using an offering for VPC networks.
• ContextSuite context=TestNetworkPermissions>:setup | Error | 0.00 | test_network_permissions.py -- unrelated. Failed due to an attempt to create a non-VPC network using an offering for VPC networks.
• ContextSuite context=TestSharedNetworkWithConfigDrive>:setup | Error | 50.86 | test_network.py -- unrelated. Failed due to an attempt to create a non-VPC network using an offering for VPC networks.
• ContextSuite context=TestIsolatedNetworksPasswdServer>:setup | Error | 0.00 | test_password_server.py -- unrelated. Failed due to an attempt to create a non-VPC network using an offering for VPC networks.
• test_CRUD_operations_userdata | Error | 19.04 | test_register_userdata.py -- unrelated. Failed due to an attempt to create a non-VPC network using an offering for VPC networks.
• test_deploy_vm_with_registered_userdata | Error | 3.80 | test_register_userdata.py -- unrelated. Failed due to an attempt to create a non-VPC network using an offering for VPC networks.
• test_deploy_vm_with_registered_userdata_with_override_policy_allow | Error | 3.82 | test_register_userdata.py -- unrelated. Failed due to an attempt to create a non-VPC network using an offering for VPC networks.
• test_deploy_vm_with_registered_userdata_with_override_policy_append | Error | 3.89 | test_register_userdata.py -- unrelated. Failed due to an attempt to create a non-VPC network using an offering for VPC networks.
• test_deploy_vm_with_registered_userdata_with_override_policy_deny | Error | 3.66 | test_register_userdata.py -- unrelated. Failed due to an attempt to create a non-VPC network using an offering for VPC networks.
• test_deploy_vm_with_registered_userdata_with_params | Error | 3.76 | test_register_userdata.py -- unrelated. Failed due to an attempt to create a non-VPC network using an offering for VPC networks.
• test_link_and_unlink_userdata_to_template | Error | 3.58 | test_register_userdata.py -- unrelated. Failed due to an attempt to create a non-VPC network using an offering for VPC networks.
• test_user_userdata_crud | Error | 3.69 | test_register_userdata.py -- unrelated. Failed due to an attempt to create a non-VPC network using an offering for VPC networks.
• ContextSuite context=TestRouterDHCPHosts>:setup | Error | 0.00 | test_router_dhcphosts.py -- unrelated. Failed due to an attempt to create a non-VPC network using an offering for VPC networks.
• ContextSuite context=TestRouterDHCPOpts>:setup | Error | 0.00 | test_router_dhcphosts.py -- unrelated. Failed due to an attempt to create a non-VPC network using an offering for VPC networks.
• ContextSuite context=TestRouterDns>:setup | Error | 0.00 | test_router_dns.py -- unrelated. Failed due to an attempt to create a non-VPC network using an offering for VPC networks.
• ContextSuite context=TestRouterDnsService>:setup | Error | 0.00 | test_router_dnsservice.py -- unrelated. Failed due to an attempt to create a non-VPC network using an offering for VPC networks.
• test_01_RVR_Network_FW_PF_SSH_default_routes_egress_true | Error | 0.16 | test_routers_network_ops.py -- unrelated. Failed due to an attempt to create a non-VPC network using an offering for VPC networks.
• test_02_RVR_Network_FW_PF_SSH_default_routes_egress_false | Error | 0.14 | test_routers_network_ops.py -- unrelated. Failed due to an attempt to create a non-VPC network using an offering for VPC networks.
• test_03_RVR_Network_check_router_state | Error | 0.14 | test_routers_network_ops.py -- unrelated. Failed due to an attempt to create a non-VPC network using an offering for VPC networks.
• ContextSuite context=TestSharedFSLifecycle>:setup | Error | 0.00 | test_sharedfs_lifecycle.py -- unrelated. Failed due to an attempt to create a non-VPC network using an offering for VPC networks.
• test_01_ssl_offloading_isolated_network | Error | 0.21 | test_ssl_offloading.py -- unrelated. Failed due to an attempt to create a non-VPC network using an offering for VPC networks.
• ContextSuite context=TestVmAutoScaling>:setup | Error | 0.00 | test_vm_autoscaling.py -- unrelated. Failed due to an attempt to create a non-VPC network using an offering for VPC networks.
• test_04_deploy_vnf_appliance | Error | 102.63 | test_vnf_templates.py -- unrelated. Failed due to an attempt to create a non-VPC network using an offering for VPC networks.

As the CI passed for this PR before merging the latest changes in main that had network-related changes (see #12758 (comment)), all the changes since then (commits e48ef53 and 5de00c8) do not change existing behaviors for other plugins/workflows, and the new failures are unrelated, I am merging these changes.