Allow parallel execution of NAS backup and delete commands

[jmsperu]

Summary

• Change executeInSequence() from true to false in TakeBackupCommand and DeleteBackupCommand
• Allows the KVM agent to process multiple backup/delete operations concurrently via its existing worker thread pool
• RestoreBackupCommand and PrepareForBackupRestorationCommand remain sequential (they modify VM state)

Motivation

Currently all backup commands are serialized on the agent — a large VM backup (e.g. 100+ GB taking 2+ hours) blocks all other backup and delete operations on the same host. This is the root cause of backup schedule delays and timeouts in environments with many VMs per host.

Each backup operation:

• Mounts its own temporary NFS directory (mktemp -d -t csbackup.XXXXX)
• Operates on independent VM disks via separate QEMU block jobs
• Has no shared state with other backup operations

There is no technical reason to serialize them. The agent already has a thread pool (requestHandler) that can execute multiple commands concurrently — this change simply allows backup commands to use it.

Impact

• Hosts with 10+ VMs will see significantly faster backup completion (backups run in parallel instead of queuing)
• NFS bandwidth is shared across concurrent backups (can be controlled with the bandwidth throttle flag from PR nasbackup.sh: add bandwidth throttle via -b flag #12846)
• No change to management server scheduling — it already submits backups as independent async jobs

Test plan

• Schedule 3+ VM backups at the same time on one host — verify they run concurrently (check virsh domjobinfo on multiple VMs)
• Verify each backup gets its own mount point (no mount conflicts)
• Run backup + delete concurrently — verify no interference
• Verify restore operations still execute sequentially
• Monitor host I/O during concurrent backups — consider using -b bandwidth throttle if NFS saturates

[codecov]

Codecov Report

❌ Patch coverage is 85.71429% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 16.26%. Comparing base (61afb4c) to head (da82917).
⚠️ Report is 35 commits behind head on 4.20.

Files with missing lines	Patch %	Lines
...rg/apache/cloudstack/backup/NASBackupProvider.java	0.00%	1 Missing ⚠️

Additional details and impacted files

@@ Coverage Diff @@## 4.20 #12847 +/- ##
============================================
+ Coverage 16.24% 16.26% +0.01% - Complexity 13411 13435 +24 
============================================
Files 5664 5667 +3 Lines 500463 500737 +274 Branches 60779 60804 +25 ============================================
+ Hits 81308 81451 +143 - Misses 410059 410178 +119 - Partials 9096 9108 +12 

Flag	Coverage Δ
uitests	4.14% <ø> (-0.01%)	⬇️
unittests	17.12% <85.71%> (+0.02%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Copilot]

Pull request overview

This PR changes CloudStack backup agent command scheduling by making the NAS backup take and delete commands eligible for parallel execution on the agent (instead of being forced through the single-threaded “in-sequence” queue).

Changes:

• Make TakeBackupCommand.executeInSequence() return false.
• Make DeleteBackupCommand.executeInSequence() return false.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
core/src/main/java/org/apache/cloudstack/backup/TakeBackupCommand.java	Marks backup creation command as non-sequential (parallelizable) on the agent.
core/src/main/java/org/apache/cloudstack/backup/DeleteBackupCommand.java	Marks backup deletion command as non-sequential (parallelizable) on the agent.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

[winterhazel]

@JoaoJandre could you have a look at whether this change would affect KBOSS?

[JoaoJandre]

@JoaoJandre could you have a look at whether this change would affect KBOSS?

@winterhazel KBOSS uses its own commands to send messages to the agent, thus this does not affect it.
In any case, KBOSS commands have this flag as false as well.

[winterhazel]

@jmsperu could you have a look at @abh1sar's comment at #12847 (comment)?

[jmsperu]

Thanks for the ping @winterhazel — and @abh1sar's point at #12847 (comment) is well taken. DeleteBackup needs sequential execution because parallel deletes against the same NAS path can race on the cleanup of the chain (we've also seen the rm -rf vs mount/unmount interleave produce stuck mounts). Will restrict the parallelization to TakeBackup only in the next push — DeleteBackup keeps its current sequential behaviour. Thanks @JoaoJandre for the KBOSS confirmation too.

[jmsperu]

Pushed b2cd0bf — reverted the DeleteBackupCommand change so only TakeBackupCommand flips to executeInSequence() = false. Delete stays serialized as you suggested @abh1sar (incremental chains share parent qcow2 files on the NAS, concurrent sibling/descendant deletes can race on shared file refs).

[winterhazel]

@blueorangutan package

[blueorangutan]

@winterhazel a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[jmsperu]

Notes on the current red checks — all three appear unrelated to this change (which is a single executeInSequence() boolean flip on TakeBackupCommand and touches no other code path):

• codecov/project — reports 4.15% (-12.10%) project-coverage delta. That number is noise: the patch surface is one line so the codecov math is degenerate. codecov/patch (the one that actually reflects this PR) passes.
• build (component/test_persistent_networks ...) — failure is test_vpc_force_delete throwing builtins.Exception plus test_admin_project_creation going EXCEPTION on one retry while the other retry passed. Known flake pattern; VPC teardown / project-creation tests are not on any code path reachable from a backup executeInSequence flip.
• build (component/test_lb_secondary_ip ...) — failure is test_03_delete_instance in test_memory_limits. Memory-limits resource-accounting tests are timing-sensitive on the simulator and have no overlap with backup orchestration.

Happy to rebase to retrigger if a reviewer prefers a fresh CI run, but the change itself is unaffected by these matrices. Other 14 checks green, 8 still running.

[blueorangutan]

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 17997

[abh1sar]

Code LGTM.

[DaanHoogland]

@blueorangutan test

[blueorangutan]

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

[blueorangutan]

[SF] Trillian test result (tid-16181)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 50661 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr12847-t16181-kvm-ol8.zip
Smoke tests completed. 141 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

[sbrueseke]

Does it make sense to add new global settings parameter so be able to edit this?

[jmsperu]

@sbrueseke good suggestion — added a global setting backup.nas.parallel.execution.enabled (Advanced, Zone-scoped, default true). It gates the take-backup command's parallelism, so an operator can force sequential execution per zone without a code change if they ever hit contention on a busy NAS. Default keeps the new parallel behaviour.

[DaanHoogland]

@blueorangutan package

[DaanHoogland]

@sbrueseke are you willing to test this?