NAS backup: compression, encryption, bandwidth throttle, integrity check by jmsperu · Pull Request #12898 · apache/cloudstack

jmsperu · 2026-03-26T22:14:37Z

Summary

Adds four optional, zone-scoped features to NAS backup operations on KVM, all disabled by default:

Compression (-c): Uses qcow2 internal compression (qemu-img convert -c) to reduce backup size
LUKS Encryption (-e): Encrypts backup files at rest using LUKS via qemu-img convert --object secret
Bandwidth Throttle (-b): Limits backup I/O — virsh blockjob --bandwidth for running VMs, qemu-img convert -r + ionice for stopped VMs
Integrity Check (--verify): Runs qemu-img check on each backup file after creation

Configuration Keys (Zone scope)

Setting	Type	Default	Description
`nas.backup.compression.enabled`	Boolean	false	Enable qcow2 compression for backup files
`nas.backup.encryption.enabled`	Boolean	false	Enable LUKS encryption for backup files
`nas.backup.encryption.passphrase`	String (Secure)	""	Passphrase for LUKS encryption
`nas.backup.bandwidth.limit.mbps`	Integer	0	Bandwidth limit in MiB/s (0 = unlimited)
`nas.backup.integrity.check`	Boolean	false	Run qemu-img check after backup

Architecture

NASBackupProvider reads zone-scoped ConfigKeys and populates a details map on TakeBackupCommand
TakeBackupCommand carries the details map from management server to KVM agent
LibvirtTakeBackupCommandWrapper extracts the details and translates them to nasbackup.sh CLI flags
nasbackup.sh implements the actual compression, encryption, throttling, and verification logic

Files Changed

scripts/vm/hypervisor/kvm/nasbackup.sh — new -c, -b, -e, --verify flags with encrypt_backup() and verify_backup() functions
core/.../TakeBackupCommand.java — added details map (HashMap) with getter/setter/addDetail
plugins/backup/nas/.../NASBackupProvider.java — 5 new ConfigKeys, populate command details in takeBackup()
plugins/hypervisors/kvm/.../LibvirtTakeBackupCommandWrapper.java — extract details, build dynamic CLI args, temp passphrase file lifecycle

Notes

All existing functionality (quiesce, cleanup, RBD support, stats) is preserved unchanged
Encryption passphrase is written to a temp file on the KVM host and deleted after backup completes
The passphrase ConfigKey uses the "Secure" category so it is not exposed in API responses
Combines and supersedes PRs nasbackup.sh: add optional backup compression via -c flag #12844, nasbackup.sh: add bandwidth throttle via -b flag #12846, nasbackup.sh: add LUKS encryption for backup files via -e flag #12848, nasbackup.sh: verify backup integrity with qemu-img check #12845

Test plan

Verify backup works with all four features disabled (default) — no behavioral change
Enable nas.backup.compression.enabled at zone scope, take backup, verify qcow2 files are compressed
Enable nas.backup.bandwidth.limit.mbps (e.g. 50), take backup of running VM, verify virsh blockjob bandwidth is applied
Enable nas.backup.bandwidth.limit.mbps, take backup of stopped VM, verify qemu-img -r rate limit is applied
Enable nas.backup.encryption.enabled with passphrase, take backup, verify files are LUKS encrypted (qemu-img info shows encryption)
Enable nas.backup.integrity.check, take backup, verify qemu-img check runs and passes
Test with multiple features enabled simultaneously (compression + integrity check)
Verify restore still works for backups created with compression/encryption
Test with RBD storage pools — verify bandwidth throttle applies correctly

… integrity check Adds four optional features to NAS backup operations, configurable at zone scope via CloudStack global settings: - Compression (-c): qcow2 internal compression of backup files Config: nas.backup.compression.enabled (default: false) - LUKS Encryption (-e): encrypt backup files at rest using qemu-img Config: nas.backup.encryption.enabled (default: false) Config: nas.backup.encryption.passphrase (Secure category) - Bandwidth Throttle (-b): limit backup I/O bandwidth via virsh blockjob for running VMs or qemu-img -r for stopped VMs Config: nas.backup.bandwidth.limit.mbps (default: 0/unlimited) - Integrity Check (--verify): qemu-img check after backup creation Config: nas.backup.integrity.check (default: false) All features are disabled by default and fully backward compatible. Settings are read from zone-scoped ConfigKeys in NASBackupProvider, passed to the KVM agent via TakeBackupCommand details map, and translated to nasbackup.sh CLI flags in LibvirtTakeBackupCommandWrapper. Changes: - nasbackup.sh: add -c, -b, -e, --verify flags with encrypt_backup() and verify_backup() helper functions - TakeBackupCommand.java: add details map for passing config to agent - NASBackupProvider.java: add 5 ConfigKeys, populate command details - LibvirtTakeBackupCommandWrapper.java: extract details, build CLI args, handle passphrase temp file lifecycle Combines and supersedes PRs apache#12844, apache#12846, apache#12848, apache#12845

codecov · 2026-03-27T05:31:12Z

Codecov Report

❌ Patch coverage is 62.10526% with 36 lines in your changes missing coverage. Please review.
✅ Project coverage is 17.70%. Comparing base (a289bb0) to head (30572d2).
⚠️ Report is 83 commits behind head on 4.22.

Files with missing lines	Patch %	Lines
...ource/wrapper/LibvirtTakeBackupCommandWrapper.java	52.54%	27 Missing and 1 partial ⚠️
...rg/apache/cloudstack/backup/TakeBackupCommand.java	40.00%	6 Missing ⚠️
...rg/apache/cloudstack/backup/NASBackupProvider.java	92.30%	0 Missing and 2 partials ⚠️

Additional details and impacted files

@@ Coverage Diff @@## 4.22 #12898 +/- ##
============================================
+ Coverage 17.67% 17.70% +0.02% - Complexity 15792 15845 +53 
============================================
Files 5922 5925 +3 Lines 533123 533623 +500 Branches 65201 65284 +83 ============================================
+ Hits 94246 94453 +207 - Misses 428236 428491 +255 - Partials 10641 10679 +38

Flag	Coverage Δ
uitests	`3.69% <ø> (-0.01%)`	⬇️
unittests	`18.77% <62.10%> (+0.02%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copilot

Pull request overview

Adds optional, zone-scoped enhancements for KVM NAS backups (compression, LUKS encryption, bandwidth throttling, and post-backup integrity verification) by plumbing config from management server → TakeBackupCommand details → KVM agent wrapper → nasbackup.sh flags.

Changes:

Add new CLI flags and implementation in nasbackup.sh for compression (-c), encryption (-e), bandwidth throttling (-b), and verification (--verify).
Extend TakeBackupCommand with a details map to carry optional settings to the agent.
Add zone-scoped NAS backup ConfigKeys and populate command details; update KVM wrapper to translate details into script args and manage a temporary passphrase file.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 10 comments.

File	Description
`scripts/vm/hypervisor/kvm/nasbackup.sh`	Implements compression/encryption/throttle/verify logic and argument parsing for NAS backup operations.
`core/src/main/java/org/apache/cloudstack/backup/TakeBackupCommand.java`	Adds a `details` map to carry optional backup feature settings from management to agent.
`plugins/backup/nas/src/main/java/org/apache/cloudstack/backup/NASBackupProvider.java`	Introduces zone-scoped ConfigKeys and passes enabled settings into `TakeBackupCommand` details.
`plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtTakeBackupCommandWrapper.java`	Builds dynamic `nasbackup.sh` command args from `TakeBackupCommand` details and writes an encryption passphrase temp file.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-03-30T09:24:49Z

+ if [[ ! -f "$ENCRYPT_PASSFILE" ]]; then
+ echo "Encryption passphrase file not found: $ENCRYPT_PASSFILE"
+ exit 1
+ fi


encrypt_backup calls exit 1 on missing/invalid passphrase file, which bypasses cleanup()/unmount logic and can leave the NAS mount + temp dir behind. Prefer returning a non-zero status and letting callers invoke cleanup() (or add a trap-based cleanup) so failures don’t leak mounts/directories.

Copilot · 2026-03-30T09:24:50Z

+ if [[ $failed -ne 0 ]]; then
+ echo "One or more backup files failed verification"
+ exit 1
+ fi


verify_backup exits directly on failure, which skips cleanup()/unmount in the calling backup paths and can leave the NAS store mounted and temp directories behind. Return failure to the caller and perform cleanup/unmount before exiting.

Copilot · 2026-03-30T09:24:50Z

+ } catch (IOException e) {
+ logger.error("Failed to create encryption passphrase file", e);
+ return new BackupAnswer(command, false, "Failed to create encryption passphrase file: " + e.getMessage());
+ }


If details indicates encryption is enabled but the passphrase is missing/empty, the wrapper silently skips adding -e and the backup proceeds unencrypted. This should fail the command (or at least log and return an error) to avoid reporting a successful encrypted backup when encryption was requested.
Suggested change
}
}
} else {
logger.error("Encryption requested for backup but no encryption passphrase was provided");
returnnewBackupAnswer(command, false, "Encryption requested but encryption_passphrase is missing or empty");

Copilot · 2026-03-30T09:24:51Z

+ // Pass optional backup enhancement settings from zone-scoped configs
+ Long zoneId = vm.getDataCenterId();
+ if (Boolean.TRUE.equals(NASBackupCompressionEnabled.valueIn(zoneId))) {
+ command.addDetail("compression", "true");
+ }
+ if (Boolean.TRUE.equals(NASBackupEncryptionEnabled.valueIn(zoneId))) {
+ command.addDetail("encryption", "true");
+ String passphrase = NASBackupEncryptionPassphrase.valueIn(zoneId);
+ if (passphrase != null && !passphrase.isEmpty()) {
+ command.addDetail("encryption_passphrase", passphrase);
+ }
+ }
+ Integer bandwidthLimit = NASBackupBandwidthLimitMbps.valueIn(zoneId);
+ if (bandwidthLimit != null && bandwidthLimit > 0) {
+ command.addDetail("bandwidth_limit", String.valueOf(bandwidthLimit));
+ }
+ if (Boolean.TRUE.equals(NASBackupIntegrityCheckEnabled.valueIn(zoneId))) {
+ command.addDetail("integrity_check", "true");
+ }


The new zone-scoped feature flags are translated into TakeBackupCommand details here, but the existing NASBackupProviderTest.takeBackupSuccessfully doesn’t assert the details map contents. Add/extend unit tests to verify the correct details are added for each config (compression, bandwidth limit, integrity check, and encryption+passphrase; and that encryption without passphrase fails).

Copilot · 2026-03-30T09:24:51Z

@@ -35,6 +37,7 @@ public class TakeBackupCommand extends Command {
 private Boolean quiesce;
 @LogLevel(LogLevel.Log4jLevel.Off)
 private String mountOptions;


details may carry sensitive values (e.g., an encryption passphrase). CloudStack’s Gson logging uses LoggingExclusionStrategy with @LogLevel to exclude fields, so leaving this unannotated can leak secrets in debug logs. Annotate details with @LogLevel(Off) (or avoid putting secrets in this map).
Suggested change
privateStringmountOptions;
privateStringmountOptions;
@LogLevel(LogLevel.Log4jLevel.Off)

Copilot · 2026-03-30T09:24:52Z

- if ! qemu-img convert -O qcow2 "$disk" "$output" > "$logFile" 2> >(cat >&2); then
+ if ! ionice -c 3 qemu-img convert $([[ "$COMPRESS" == "true" ]] && echo "-c") $([[ -n "$BANDWIDTH" ]] && echo "-r" "${BANDWIDTH}M") -O qcow2 "$disk" "$output" > "$logFile" 2> >(cat >&2); then
 echo "qemu-img convert failed for $disk $output"
 cleanup


On qemu-img convert failure, this calls cleanup but then continues execution (no exit/return). If cleanup succeeds, the function proceeds to later steps with an unmounted/removed dest, which can cause confusing follow-on failures and potentially report incorrect results. Exit the script (or return 1) after cleanup here.
Suggested change
cleanup
cleanup
return 1

- nasbackup.sh: Replace exit 1 with return 1 in encrypt_backup and verify_backup so callers can run cleanup before terminating - nasbackup.sh: Append (>>) instead of truncate (>) agent.log in qemu-img convert for stopped VM backups - nasbackup.sh: Add return 1 after cleanup on qemu-img convert failure to stop execution - nasbackup.sh: Callers of encrypt_backup/verify_backup now check return code and run cleanup on failure - LibvirtTakeBackupCommandWrapper: Fail with error when encryption is enabled but passphrase is missing instead of silently skipping - LibvirtTakeBackupCommandWrapper: Delete temp passphrase file in finally block, set 0600 permissions, use explicit UTF-8 charset - NASBackupProvider: Throw CloudRuntimeException when encryption is enabled but passphrase is null/empty - NASBackupProviderTest: Add tests for compression, bandwidth, integrity check, encryption+passphrase, and encryption-without- passphrase failure scenarios - TakeBackupCommand: Add @loglevel(Off) to details field to prevent passphrase leaking in debug logs - TakeBackupCommand: Normalize null to empty HashMap in setDetails

sureshanaparti · 2026-04-01T09:54:33Z

@blueorangutan package

blueorangutan · 2026-04-01T09:56:05Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2026-04-01T10:18:17Z

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 17323

sureshanaparti · 2026-04-01T10:19:22Z

@jmsperu can you check/fix the build failure.

Address remaining Copilot review feedback on PR apache#12898: - Replace `2>&1 | tee -a` with `>> logFile 2>&1` in encrypt_backup, compress, and mount_operation to prevent tee from masking non-zero exit codes of qemu-img and mount commands - Add `return 1` after cleanup on virsh backup job failure to prevent continuing execution with broken state

The test helper overrideConfigValue() was only setting _value on ConfigKey, but zone-scoped configs (valueIn(zoneId)) fall back to _defaultValue when s_depot is null in test context. Also set _defaultValue via ReflectionTestUtils to ensure valueIn() returns the expected test value. Fixes: 4 assertion failures (compression, bandwidth, encryption, integrity_check details all returned null) and 1 error (encryption without passphrase expected CloudRuntimeException but got NullPointerException from null config value).

jmsperu · 2026-04-02T15:37:40Z

@sureshanaparti Fixed. The test failures were caused by overrideConfigValue() in NASBackupProviderTest only setting _value on ConfigKey, but the zone-scoped configs (valueIn(zoneId)) fall back to _defaultValue when s_depot is null in the test context. All 5 config values (compression, bandwidth, encryption, encryption passphrase, integrity check) were returning null instead of the test values.

The fix also sets _defaultValue via ReflectionTestUtils so valueIn() correctly resolves test values.

Also addressed in the previous commit: replaced 2>&1 | tee -a with >> logFile 2>&1 in nasbackup.sh to prevent tee from masking non-zero exit codes, and added return 1 after cleanup on virsh backup job failure.

Could you please retrigger the build? @blueorangutan package

blueorangutan · 2026-04-02T15:40:08Z

@jmsperu a [SL] Jenkins job has been kicked to build packages. It will be bundled with` SystemVM template(s). I'll keep you posted as I make progress.

blueorangutan · 2026-04-02T16:31:55Z

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 17346

abh1sar · 2026-05-14T04:17:54Z

Hi @jmsperu
Please check and address pending review comments from Copilot and fix the conflict.

P.S: Since you are working on the incremental backup PR as well, I'll suggest giving that one priority over this.
Let's try to get it merged for the 4.23 release which is in 2 weeks.
And resume work on this PR after the incremental backup PR is merged.

Conflict in LibvirtTakeBackupCommandWrapper.java was in the NAS-backup command invocation block — upstream switched to a String[] literal and dropped cmdArgs; resolved by keeping our cmdArgs list (needed for conditional compression / encryption / bandwidth flags) and folding in upstream's timeout var. Copilot review fixes: - Detail keys (compression / encryption / encryption_passphrase / bandwidth_limit / integrity_check) now declared as constants in the wrapper instead of being scattered as string literals. - encrypt_backup() in nasbackup.sh now preserves -c when COMPRESS is true; previously the encrypt-step re-convert produced an uncompressed-but-encrypted qcow2 and silently discarded the earlier compression work. - verify_backup() now passes the LUKS secret via --object + --image-opts when ENCRYPT_PASSFILE is set, so qemu-img check can actually open encrypted backups (previously --verify always failed when encryption was enabled). TakeBackupCommand.setDetails() is already null-safe (this.details = details != null ? details : new HashMap<>()) — no change needed. Note (Copilot): one further suggestion was to promote the detail keys to a class shared between MS and agent. Held off for now to keep the diff focused; can do as a follow-up if maintainers want.

jmsperu · 2026-05-22T21:41:31Z

Picking this back up — will rebase against current main to clear the merge conflict and address the open Copilot review comments in the next push. ETA next 24-48 hrs. Thanks for the patience @abh1sar.

Addresses the final outstanding Copilot review point on apache#12898 (shared constants). The detail-map keys exchanged between NASBackupProvider, the KVM agent wrapper, and the nasbackup.sh interface were defined twice — once as private string literals in the provider and once as static finals in the wrapper — which makes a typo or rename silently break the wire protocol. Moved the constants to TakeBackupCommand so they live on the command class itself, and updated both the provider's addDetail() sites and the wrapper's details.get() sites to use them. Test file updated to assert via the same constants for consistency. No behaviour change. All 14 NAS tests and 611 KVM tests pass.

jmsperu · 2026-05-22T22:21:06Z

Pushed 2150959 — addresses the last outstanding Copilot point (shared detail-key constants).

Detail-map keys were duplicated as private literals in NASBackupProvider and static finals in LibvirtTakeBackupCommandWrapper. A typo/rename in one place silently broke the wire protocol. Moved them onto TakeBackupCommand itself (the class that carries the map between the two sides) so there's a single source of truth.

Sweep of all Copilot threads on this PR:

#	Reviewer point	Status
1	encrypt_backup leaking mount on missing passphrase	fixed — `return 1` (earlier commit)
2	verify_backup leaking mount on failure	fixed — `return 1` (earlier commit)
3	qemu-img convert: cleanup not followed by exit/return	fixed — `return 1` after cleanup
4	`> \"$logFile\"` truncating agent log	fixed — all sites use `>>`
5	verify_backup not handling LUKS	fixed — `--object secret + --image-opts`
6	encrypt step discarding compression	fixed — `-c` preserved via `$compress_flag`
7	Provider: encryption-without-passphrase silent fallback	fixed — throws CloudRuntimeException
8	Missing tests for new detail wirings	fixed — 5 new tests in NASBackupProviderTest
9	Wrapper: same silent-fallback	fixed — returns BackupAnswer false
10	Temp passphrase file leak / perms	fixed — 0600 + UTF-8 + catch+finally cleanup
11	Hard-coded detail-key strings	fixed this commit — moved to TakeBackupCommand
12	`@LogLevel(Off)` on details (secrets in logs)	fixed (earlier commit)
13	setDetails null-safe	fixed (earlier commit)

Mergeable, no conflicts. Ready for re-review @abh1sar / @Copilot.

jmsperu · 2026-05-22T22:34:36Z

Same nudge for this PR — the latest push for the shared-constants refactor is in action_required state on GH Actions and needs a committer (@abh1sar @DaanHoogland) to approve the workflow run before CI executes. Thanks!

jmsperu · 2026-05-26T18:22:44Z

Friendly bump — addressed @copilot-pull-request-reviewer's feedback in 45ce015 (combined Copilot-flagged fixes — cleanup on encrypt/verify failure, compression preserved when encrypting, qemu-img check with LUKS secret, fail-fast when encryption enabled but passphrase missing, sensitive details @loglevel(Off), null-safe setDetails, append-not-truncate to agent.log) and 2150959 (consolidated DETAIL_* keys as constants on TakeBackupCommand).

Just retriggered the flaky CI shard. @DaanHoogland @sureshanaparti @weizhouapache — would appreciate a look when bandwidth allows.

DaanHoogland · 2026-05-27T09:39:44Z

@jmsperu your code looks good, and of course a “but”, can you extract new code into seprate methods where possible?

As @abh1sar explained, testing capacity is an issue. We do require testing for all non-trivial fixes and enhancements though. Please bare with us.

… methods + tests Addresses review feedback (extract methods + add testing): - LibvirtTakeBackupCommandWrapper: extract appendEnhancementFlags() + writePassphraseFile() from execute(); behavior-preserving - NASBackupProvider: extract applyBackupEnhancementDetails() - add LibvirtTakeBackupCommandWrapperTest (5 cases); existing NASBackupProviderTest (14) stays green

jmsperu · 2026-06-20T09:30:02Z

@DaanHoogland Thanks for the review — addressed both points in 0effda26e4:

Method extraction — pulled the inline enhancement logic into dedicated methods:

LibvirtTakeBackupCommandWrapper: extracted appendEnhancementFlags() + writePassphraseFile() (so execute() is a clean dispatch instead of a ~40-line inline block)
NASBackupProvider: extracted applyBackupEnhancementDetails()

(The shell side was already factored — encrypt_backup() / verify_backup() are standalone functions in nasbackup.sh.)

Testing — added LibvirtTakeBackupCommandWrapperTest (5 cases: compression/bandwidth/integrity flag mapping, bandwidth=0 skip, encryption writes the 0600 passphrase file + -e flag, and fail-fast when encryption is enabled without a passphrase). The existing NASBackupProviderTest (14 cases, covering the details map) stays green — confirming the refactor is behavior-preserving.

DaanHoogland · 2026-06-21T14:20:50Z

@blueorangutan package

blueorangutan · 2026-06-21T14:22:06Z

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2026-06-21T15:27:04Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 18329

DaanHoogland · 2026-06-22T06:45:45Z

@blueorangutan test

blueorangutan · 2026-06-22T06:48:04Z

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2026-06-22T21:59:23Z

[SF] Trillian test result (tid-16388)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 51742 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr12898-t16388-kvm-ol8.zip
Smoke tests completed. 149 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

…; drop trailing '(Copilot review)' comment artifacts

Copilot

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (5)

scripts/vm/hypervisor/kvm/nasbackup.sh:461

The -b/--bandwidth flag is accepted without validating that it is a positive integer. If a non-numeric value (or 0/negative) is passed, the later virsh/qemu-img calls will fail in confusing ways (and could treat the value as additional CLI args). Validate the argument at parse time and reject invalid values with a clear error.

 -b|--bandwidth)
BANDWIDTH="$2"
shift
shift
;;

scripts/vm/hypervisor/kvm/nasbackup.sh:387

On mount failure, the script now only prints a generic "Failed to mount" message and redirects the actual mount error output solely into the agent log. This makes failures hard to diagnose from the management server side (stdout/stderr). Capture the mount output and include it in the error message while still appending it to the log.

 if mount -t ${NAS_TYPE} ${NAS_ADDRESS} ${mount_point} $([[ ! -z "${MOUNT_OPTS}" ]] && echo -o ${MOUNT_OPTS}) >> "$logFile" 2>&1; then
log -ne "Successfully mounted ${NAS_TYPE} store"
else
echo "Failed to mount ${NAS_TYPE} store"
exit 1

plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtTakeBackupCommandWrapper.java:112

The temporary passphrase file deletion is currently best-effort but silent (passphraseFile.delete() return value is ignored). If deletion fails, the key file may remain on disk without any indication. Prefer Files.deleteIfExists(...) and log failures so operators can detect and remediate leftover secret files.

 // Clean up passphrase file after backup completes (best-effort).
if (passphraseFile != null && passphraseFile.exists()) {
passphraseFile.delete();
}

scripts/vm/hypervisor/kvm/nasbackup.sh:233

The bandwidth-throttling virsh calls use unquoted $VM and $disk, which can break if either contains unexpected characters/whitespace. Quote these arguments to avoid word-splitting/globbing and keep behavior consistent with the other virsh invocations in this function.

This issue also appears in the following locations of the same file:

line 383
line 457

 for disk in $(virsh -c qemu:///system domblklist $VM --details 2>/dev/null | awk '/disk/{print$3}'); do
virsh -c qemu:///system blockjob $VM $disk --bandwidth "${BANDWIDTH}" 2>/dev/null || true
done

plugins/backup/nas/src/main/java/org/apache/cloudstack/backup/NASBackupProvider.java:116

The config key name "nas.backup.bandwidth.limit.mbps" suggests megabits/sec (Mbps), but the description and the script usage treat the value as MiB/sec. This mismatch is likely to confuse operators and lead to misconfiguration. Consider renaming the key (or changing the documented units and implementing conversion) to make the units unambiguous.

 ConfigKey<Integer> NASBackupBandwidthLimitMbps = new ConfigKey<>("Advanced", Integer.class,
"nas.backup.bandwidth.limit.mbps",
"0",
"Bandwidth limit in MiB/s for backup operations (0 = unlimited).",
true,

+ if (Boolean.TRUE.equals(NASBackupEncryptionEnabled.valueIn(zoneId))) {
+ String passphrase = NASBackupEncryptionPassphrase.valueIn(zoneId);
+ if (passphrase == null || passphrase.isEmpty()) {
+ throw new CloudRuntimeException("NAS backup encryption is enabled but no passphrase is configured (nas.backup.encryption.passphrase)");
+ }


jmsperu · 2026-07-28T07:40:56Z

The failing build shard here (test_lb_secondary_ip, test_list_nics, test_list_pod, …) is unrelated to this backup change — those are load-balancer/networking smoke tests, and the other 24 shards pass. Looks like a flaky/infra failure. Could a committer kick off a re-run of that shard? Thanks.

boring-cyborgBot added component:backup component:kvm labels Mar 26, 2026

DaanHoogland requested review from abh1sar, Copilot and sureshanaparti March 30, 2026 09:17

DaanHoogland added this to the 4.22.1 milestone Mar 30, 2026

Copilotstarted reviewing on behalf of DaanHoogland March 30, 2026 09:18 View session

CopilotAI reviewed Mar 30, 2026
View reviewed changes

sureshanaparti added this to Apache CloudStack 4.22.1 Apr 1, 2026

github-project-automationBot moved this to Todo in Apache CloudStack 4.22.1 Apr 1, 2026

sureshanaparti moved this from Todo to In Progress in Apache CloudStack 4.22.1 Apr 1, 2026

sureshanaparti requested a review from rajujith April 1, 2026 09:53

jmsperu added 2 commits April 2, 2026 00:45

abh1sar requested a review from Copilot April 5, 2026 12:22

Copilotstarted reviewing on behalf of abh1sar April 5, 2026 12:23 View session

sureshanaparti modified the milestones: 4.22.1, 4.22.2, 4.23.0 Apr 24, 2026

github-actionsBot removed the status:has-conflicts label May 15, 2026

ci: retrigger workflow (flaky/stale shards)
8a84d8e

weizhouapache modified the milestones: 4.23.0, 4.24.0 Jul 10, 2026

Damans227 reviewed Jul 21, 2026
View reviewed changes

Comment thread...src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtTakeBackupCommandWrapper.java Outdated
Comment threadscripts/vm/hypervisor/kvm/nasbackup.sh Outdated

address review (apache#12898): reorder java.io imports alphabetically…
30572d2
…; drop trailing '(Copilot review)' comment artifacts

DaanHoogland requested a review from Copilot July 28, 2026 07:11

Copilotstarted reviewing on behalf of DaanHoogland July 28, 2026 07:23 View session

CopilotAI reviewed Jul 28, 2026
View reviewed changes

- }
+ }
+ } else {
+logger.error("Encryption requested for backup but no encryption passphrase was provided");
+returnnewBackupAnswer(command, false, "Encryption requested but encryption_passphrase is missing or empty");

	privateStringmountOptions;
	privateStringmountOptions;
	@LogLevel(LogLevel.Log4jLevel.Off)

Uh oh!

Conversation

jmsperu commented Mar 26, 2026

Summary

Configuration Keys (Zone scope)

Architecture

Files Changed

Notes

Test plan

Uh oh!

codecovBot commented Mar 27, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

CopilotAIMar 30, 2026

Choose a reason for hiding this comment

Uh oh!

CopilotAIMar 30, 2026

Choose a reason for hiding this comment

Uh oh!

CopilotAIMar 30, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

CopilotAIMar 30, 2026

Choose a reason for hiding this comment

Uh oh!

CopilotAIMar 30, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

CopilotAIMar 30, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

sureshanaparti commented Apr 1, 2026

Uh oh!

blueorangutan commented Apr 1, 2026

Uh oh!

blueorangutan commented Apr 1, 2026

Uh oh!

sureshanaparti commented Apr 1, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jmsperu commented Apr 2, 2026

Uh oh!

blueorangutan commented Apr 2, 2026

Uh oh!

blueorangutan commented Apr 2, 2026

Uh oh!

abh1sar commented May 14, 2026

Uh oh!

jmsperu commented May 22, 2026

Uh oh!

jmsperu commented May 22, 2026

Uh oh!

jmsperu commented May 22, 2026

Uh oh!

jmsperu commented May 26, 2026

Uh oh!

DaanHoogland commented May 27, 2026

Uh oh!

jmsperu commented Jun 20, 2026

Uh oh!

DaanHoogland commented Jun 21, 2026

Uh oh!

blueorangutan commented Jun 21, 2026

Uh oh!

blueorangutan commented Jun 21, 2026

Uh oh!

DaanHoogland commented Jun 22, 2026

Uh oh!

blueorangutan commented Jun 22, 2026

Uh oh!

codecovBot commented Mar 27, 2026•
edited
Loading

sureshanaparti commented Apr 1, 2026•
edited
Loading