[NAS Backup] Suppress Errors in Disk Usage Calculation that Caused Backup to Fail. by daviftorres · Pull Request #13424 · apache/cloudstack

daviftorres · 2026-06-15T19:57:40Z

Description

This PR tried to prevent the failure of the job at the statistics section of a backup that has actually succeeded.

Apparently, it also fixes some silent failures I previously reported in #11727

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
Build/CI
Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

daviftorres · 2026-06-15T20:50:13Z

This is the equivalent command for applying the fix:

sed -i 's_du -sb $dest | cut -f1_du -sb $dest 2>/dev/null | cut -f1 || true_g' /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/nasbackup.sh

We haven't confirmed the exact root cause of the du failure yet. As a precaution, we applied this fix to all servers and will monitor backups over the next few days.

So, I am running tests with 2>>/var/log/cloudstack/agent/nasbackup.err so I can see what is the error message.

daviftorres · 2026-06-17T14:08:49Z

Proposed Changes Rationale

backup_size=$(du -sb "$dest"2>/dev/null | cut -f1)||true

NFS issues may cause du command to fail.
A size retrieval failure should not invalidate a successful backup.

timeout 60 umount "$mount_point"2>/dev/null ||true
rmdir "$mount_point"2>/dev/null ||true

Another process may keep the device busy (e.g., parallel backups).
Network issues may cause hangs on NFS.
Cleanup failures should not invalidate a successful backup.

echo -n "$backup_size"

Outputs the size at the end to confirm the script completed past the potentially problematic commands.

daviftorres · 2026-06-17T18:59:55Z

Dear @abh1sar , do you think you can help me with this bug? Regards,

codecov · 2026-06-18T11:27:44Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 18.93%. Comparing base (5ed4894) to head (664b2d5).
⚠️ Report is 42 commits behind head on main.

Additional details and impacted files

@@ Coverage Diff @@## main #13424 +/- ##
============================================
- Coverage 18.94% 18.93% -0.01% - Complexity 18376 18473 +97 
============================================
Files 6192 6221 +29 Lines 556550 560039 +3489 Branches 67954 68285 +331 ============================================
+ Hits 105454 106066 +612 - Misses 439517 442347 +2830 - Partials 11579 11626 +47

Flag	Coverage Δ
uitests	`3.50% <ø> (-0.02%)`	⬇️
unittests	`20.14% <ø> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copilot

Pull request overview

This PR adjusts the KVM NAS backup script’s “statistics/cleanup” section so that failures while computing backup disk usage (and related cleanup commands) don’t cause an otherwise successful backup job to be marked as failed.

Changes:

Capture du output into backup_size and suppress du stderr to avoid failing the script during size calculation.
Add timeout around umount and suppress errors from umount/rmdir.
Emit the computed backup size at the end of backup_running_vm().

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

DaanHoogland

clgtm

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 2 comments.

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 2 comments.

abh1sar

hi @daviftorres
would it be possible to test script changes in your env where it is reproducible?
Some log outputs would be nice.

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

daviftorres · 2026-06-25T13:48:46Z

would it be possible to test script changes in your env where it i

2026-06-25 13:22:42,360 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-1:[]) (logid:e4aca261) Request:Seq 218-281474976710814: { Cmd , MgmtId: 90520736259046, via: 218, Ver: v1, Flags: 100111, [{"org.apache.cloudstack.backup.TakeBackupCommand":{"vmName":"i-84-6493-VM","backupPath":"i-84-6493-VM/2026.06.25.13.22.42","backupRepoType":"nfs","backupRepoAddress":"10.4.2.145:/mnt/VAN3-NAS01-STOR-POOL-A1/VAN3-NAS01-DS-01","wait":"0","bypassHostMaintenance":"false"}}] }
2026-06-25 13:22:42,360 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-1:[]) (logid:e4aca261) Processing command: org.apache.cloudstack.backup.TakeBackupCommand
2026-06-25 13:25:04,120 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-1:[]) (logid:e4aca261) Seq 218-281474976710814: { Ans: , MgmtId: 90520736259046, via: 218, Ver: v1, Flags: 110, [{"org.apache.cloudstack.backup.BackupAnswer":{"size":"(47.67 GB) 51190241222","result":"true","details":"Job type: Completed

Nothing else that I can find in the logs are related to the backup job.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

abh1sar · 2026-06-26T01:50:43Z

@blueorangutan package

blueorangutan · 2026-06-26T01:52:04Z

@abh1sar a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

abh1sar

LGTM

abh1sar · 2026-07-02T14:35:39Z

@blueorangutan package

blueorangutan · 2026-07-02T14:38:05Z

@abh1sar a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2026-07-02T15:25:06Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 18442

abh1sar · 2026-07-02T15:53:07Z

@blueorangutan test

blueorangutan · 2026-07-02T15:54:03Z

@abh1sar a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2026-07-03T07:01:48Z

[SF] Trillian test result (tid-16468)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 51648 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr13424-t16468-kvm-ol8.zip
Smoke tests completed. 152 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

weizhouapache

code lgtm

boring-cyborgBot added the component:kvm label Jun 15, 2026

github-actionsBot mentioned this pull request Jun 15, 2026
[repo-status] Daily Status Report — June 15, 2026 #13425
Closed

daviftorres marked this pull request as ready for review June 16, 2026 15:02

github-actionsBot mentioned this pull request Jun 16, 2026
[repo-status] Daily Status Report — June 16, 2026 #13435
Closed

CopilotAI review requested due to automatic review settings June 18, 2026 20:07

Copilotstarted reviewing on behalf of daviftorres June 18, 2026 20:07 View session

CopilotAI reviewed Jun 18, 2026
View reviewed changes

Comment threadscripts/vm/hypervisor/kvm/nasbackup.sh Outdated

DaanHoogland approved these changes Jun 19, 2026
View reviewed changes

DaanHoogland added the status:needs-testing label Jun 19, 2026

DaanHoogland requested a review from abh1sar June 23, 2026 06:53

CopilotAI review requested due to automatic review settings June 23, 2026 12:20

CopilotAI reviewed Jun 23, 2026
View reviewed changes

Comment threadscripts/vm/hypervisor/kvm/nasbackup.sh Outdated
Comment threadscripts/vm/hypervisor/kvm/nasbackup.sh Outdated

Copilotstarted reviewing on behalf of daviftorres June 23, 2026 12:31 View session

abh1sar reviewed Jun 23, 2026
View reviewed changes

Comment threadscripts/vm/hypervisor/kvm/nasbackup.sh Outdated
Comment threadscripts/vm/hypervisor/kvm/nasbackup.sh Outdated
Comment threadscripts/vm/hypervisor/kvm/nasbackup.sh Outdated

CopilotAI review requested due to automatic review settings June 24, 2026 20:55

Copilotstarted reviewing on behalf of daviftorres June 24, 2026 20:55 View session

CopilotAI reviewed Jun 24, 2026
View reviewed changes

Comment threadscripts/vm/hypervisor/kvm/nasbackup.sh Outdated
Comment threadscripts/vm/hypervisor/kvm/nasbackup.sh Outdated

abh1sar reviewed Jun 25, 2026
View reviewed changes

Comment threadscripts/vm/hypervisor/kvm/nasbackup.sh Outdated
Comment threadscripts/vm/hypervisor/kvm/nasbackup.sh Outdated
Comment threadscripts/vm/hypervisor/kvm/nasbackup.sh Outdated
Comment threadscripts/vm/hypervisor/kvm/nasbackup.sh Outdated

CopilotAI review requested due to automatic review settings June 25, 2026 13:19

Copilotstarted reviewing on behalf of daviftorres June 25, 2026 13:21 View session

CopilotAI reviewed Jun 25, 2026
View reviewed changes

Comment threadscripts/vm/hypervisor/kvm/nasbackup.sh Outdated

CopilotAI review requested due to automatic review settings June 26, 2026 01:49

CopilotAI reviewed Jun 26, 2026

abh1sar reviewed Jul 2, 2026
View reviewed changes

Comment threadscripts/vm/hypervisor/kvm/nasbackup.sh Outdated
Comment threadscripts/vm/hypervisor/kvm/nasbackup.sh Outdated
Comment threadscripts/vm/hypervisor/kvm/nasbackup.sh Outdated

remove extra whitespace
5af17c2

CopilotAI review requested due to automatic review settings July 2, 2026 11:53

remove extra whitespace
830fbac

Copilotstarted reviewing on behalf of abh1sar July 2, 2026 11:53 View session

revert to echo
200c40e

CopilotAI reviewed Jul 2, 2026
View reviewed changes

Comment threadscripts/vm/hypervisor/kvm/nasbackup.sh Outdated

CopilotAI review requested due to automatic review settings July 2, 2026 11:56

Copilotstarted reviewing on behalf of daviftorres July 2, 2026 11:56 View session

CopilotAI reviewed Jul 2, 2026
View reviewed changes

abh1sar reviewed Jul 2, 2026
View reviewed changes

Comment threadscripts/vm/hypervisor/kvm/nasbackup.sh Outdated
Comment threadscripts/vm/hypervisor/kvm/nasbackup.sh Outdated

Update scripts/vm/hypervisor/kvm/nasbackup.sh
cce95d4

CopilotAI review requested due to automatic review settings July 2, 2026 12:12

Update scripts/vm/hypervisor/kvm/nasbackup.sh
664b2d5

Copilotstarted reviewing on behalf of abh1sar July 2, 2026 12:14 View session

CopilotAI reviewed Jul 2, 2026
View reviewed changes

abh1sar approved these changes Jul 2, 2026
View reviewed changes

weizhouapache approved these changes Jul 7, 2026
View reviewed changes

weizhouapache merged commit f9a9451 into apache:mainJul 7, 2026
24 of 27 checks passed

abh1sar mentioned this pull request Jul 10, 2026
CIFS NAS Backups failing "randomly" #12122
Closed

DaanHoogland removed the status:needs-testing label Jul 20, 2026

DaanHoogland linked an issue Jul 20, 2026 that may be closed by this pull request
CIFS NAS Backups failing "randomly" #12122
Closed

This was referenced Jul 20, 2026
Creating a unmount_operation with safety checks for nasbackup.sh #12133
Closed
backport #13424 on 22? #13649
Open

Uh oh!

Conversation

daviftorres commented Jun 15, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

daviftorres commented Jun 15, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

daviftorres commented Jun 17, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed Changes Rationale

Uh oh!

daviftorres commented Jun 17, 2026

Uh oh!

codecovBot commented Jun 18, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

DaanHoogland left a comment

Choose a reason for hiding this comment

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

abh1sar left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

daviftorres commented Jun 25, 2026

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

Uh oh!

abh1sar commented Jun 26, 2026

Uh oh!

blueorangutan commented Jun 26, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

daviftorres commented Jun 15, 2026•
edited
Loading

daviftorres commented Jun 15, 2026•
edited
Loading

daviftorres commented Jun 17, 2026•
edited
Loading

codecovBot commented Jun 18, 2026•
edited
Loading