fix(linstor): verify resource deletion completes; warn if stuck in DELETING by jmsperu · Pull Request #13076 · apache/cloudstack

jmsperu · 2026-04-28T08:31:45Z

Summary

The LINSTOR plugin treats a successful HTTP response from resourceDefinitionDelete as proof the resource is gone and immediately drops the volume from CloudStack's accounting. In practice LINSTOR can return success while the resource lingers in DELETING state — for example when a DRBD peer is unreachable, quorum was lost, or a satellite is down. The plugin has no retry, no verification, and no sweeper. We've found hundreds of stuck DELETING resources accumulated over weeks because nothing surfaces the divergence between the CS view and the LINSTOR view.

What this PR does

Adds two helpers to LinstorUtil:

isResourceDefinitionGone(api, rscName) — existence check via resourceDefinitionList
waitForResourceDefinitionDeleted(api, rscName, timeoutMillis) — polls every 1s until the resource is gone OR timeout elapses; returns true on confirmed-gone, false on timeout

Calls waitForResourceDefinitionDeleted from both delete paths:

LinstorPrimaryDataStoreDriverImpl.deleteResourceDefinition (driver / management-server side)
LinstorStorageAdaptor.deRefOrDeleteResource (host / KVM agent side)

Default timeout 30 seconds (DEFAULT_RD_DELETE_VERIFY_TIMEOUT_MILLIS).

Behaviour

Resource deletes within 30s (the normal case) — no log change, no behaviour change.
Resource still in DELETING after 30s — emits a WARN naming the resource and pointing the operator at linstor resource list. Returns success to the caller (the CS-side accounting has already moved on; throwing here would create a different inconsistency).
Controller transiently unreachable during poll — debug-logs each failed poll, keeps trying until deadline.

What this does NOT do (deferred)

Tier-2 sweeper that periodically re-attempts delete on long-stuck resources is not in this PR. The Tier-1 surface here is the minimum needed to make the problem visible in operator logs. A sweeper can land separately if maintainers want it.

Test plan

CI build + unit tests pass (no public API changes; only behaviour additions in private methods)
Manual: delete a volume on a healthy LINSTOR cluster — no new logs, ~30s shorter than wait would otherwise be (immediate return after first poll confirms gone)
Manual: delete a volume on a cluster with a downed satellite — observe the new WARN line in the agent log and confirm the resource is still in linstor resource list until manually cleared

…stuck The LINSTOR plugin treats a successful HTTP response from resourceDefinitionDelete as proof the resource is gone, then drops the volume from CloudStack's accounting. In practice LINSTOR can return success while the resource lingers in DELETING state — for example when a DRBD peer is unreachable, quorum was lost, or a satellite is down. The plugin had no retry, no verification, and no sweeper. Operators have been finding hundreds of stuck DELETING resources accumulated over weeks because nothing surfaced the divergence between the CS view and the LINSTOR view. This change adds two helpers to LinstorUtil: isResourceDefinitionGone(api, rscName) - quick existence check via resourceDefinitionList waitForResourceDefinitionDeleted(api, rscName, timeoutMillis) - polls every second until the resource is gone OR timeout elapses - returns true on confirmed-gone, false on timeout and calls waitForResourceDefinitionDeleted from both delete sites (driver: LinstorPrimaryDataStoreDriverImpl.deleteResourceDefinition; adaptor: LinstorStorageAdaptor.deRefOrDeleteResource) with a 30s default timeout. On timeout the plugin logs a WARN with the resource name and a hint pointing at `linstor resource list`. We deliberately do NOT throw on timeout: the CS-side accounting has already moved on, and throwing would create a different inconsistency. This is the minimal Tier-1 fix that surfaces the problem in the operator's view. A follow-up could add a periodic sweeper that attempts force-delete on long-stuck DELETING resources.

Copilot

Pull request overview

This PR adds post-delete verification for LINSTOR resource definitions to surface cases where LINSTOR returns success but the resource definition remains stuck in DELETING, improving operational visibility while keeping CloudStack-side deletion semantics unchanged.

Changes:

Add isResourceDefinitionGone and waitForResourceDefinitionDeleted helpers (with a 30s default timeout) to poll for confirmed deletion.
Invoke the delete-verification helper from both management-server and KVM agent delete paths.
Emit WARN logs when the resource definition still exists after the verification timeout.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File	Description
plugins/storage/volume/linstor/src/main/java/org/apache/cloudstack/storage/datastore/util/LinstorUtil.java	Adds delete-verification helpers and a default timeout constant.
plugins/storage/volume/linstor/src/main/java/org/apache/cloudstack/storage/datastore/driver/LinstorPrimaryDataStoreDriverImpl.java	Verifies RD deletion after successful API delete; warns if still present.
plugins/storage/volume/linstor/src/main/java/com/cloud/hypervisor/kvm/storage/LinstorStorageAdaptor.java	Verifies RD deletion after successful API delete in agent path; warns if still present.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-04-28T08:40:29Z

+ public static boolean waitForResourceDefinitionDeleted(DevelopersApi api, String rscName, long timeoutMillis) {
+ final long deadline = System.currentTimeMillis() + timeoutMillis;
+ while (true) {
+ try {
+ if (isResourceDefinitionGone(api, rscName)) {
+ return true;
+ }
+ } catch (ApiException e) {
+ LOGGER.debug("LINSTOR delete-verify poll failed for {}: {}", rscName, e.getMessage());
+ // Keep polling — controller may be transiently unavailable.
+ }
+ if (System.currentTimeMillis() >= deadline) {
+ return false;
+ }
+ try {
+ Thread.sleep(1_000L);
+ } catch (InterruptedException ie) {
+ Thread.currentThread().interrupt();
+ return false;
+ }
+ }


New delete-verification behavior in waitForResourceDefinitionDeleted / isResourceDefinitionGone isn’t covered by the existing LinstorUtilTest suite. Please add unit tests for (a) confirmed-gone case, and (b) timeout/exception paths (ideally without introducing real sleeps) so regressions in the polling logic are caught in CI.

codecov · 2026-04-28T08:44:32Z

Codecov Report

❌ Patch coverage is 0% with 31 lines in your changes missing coverage. Please review.
✅ Project coverage is 18.08%. Comparing base (6f4445c) to head (b7be64f).
⚠️ Report is 28 commits behind head on main.

Files with missing lines	Patch %	Lines
...cloudstack/storage/datastore/util/LinstorUtil.java	0.00%	23 Missing ⚠️
.../hypervisor/kvm/storage/LinstorStorageAdaptor.java	0.00%	4 Missing ⚠️
...tore/driver/LinstorPrimaryDataStoreDriverImpl.java	0.00%	4 Missing ⚠️

Additional details and impacted files

@@ Coverage Diff @@## main #13076 +/- ##
============================================
+ Coverage 18.02% 18.08% +0.05% - Complexity 16621 16719 +98 
============================================
Files 6029 6037 +8 Lines 542184 542611 +427 Branches 66451 66434 -17 ============================================
+ Hits 97740 98120 +380 - Misses 433428 433476 +48 + Partials 11016 11015 -1

Flag	Coverage Δ
uitests	`3.51% <ø> (-0.01%)`	⬇️
unittests	`19.24% <0.00%> (+0.06%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

rp-

if you do the one suggested change, I can ran it against Linstor tests

@rp-

resourceDefinitionList(null, ...) fetched every RD on the cluster and then streamed for a match — repeated once per second from waitForResourceDefinitionDeleted this scales with cluster size. Pass the name as the rscDfns filter so the controller returns an empty list when the RD is gone. Also clarify the timeout-constant Javadoc per copilot/abh1sar review (constant is used from management-server paths too, not just agents). Addresses review comments from @rp- and Copilot on apache#13076.

sureshanaparti · 2026-05-14T15:30:14Z

@blueorangutan package

blueorangutan · 2026-05-14T15:32:04Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2026-05-14T16:21:54Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 17858

DaanHoogland · 2026-05-20T07:26:00Z

@rp- , will you run tests for this one?

rp- · 2026-05-20T10:59:50Z

@rp- , will you run tests for this one?

All fine here, LGTM

DaanHoogland · 2026-05-21T12:49:36Z

@rp- , will you run tests for this one?
All fine here, LGTM

so your “request changes” status is not longer valid?

rp- · 2026-05-21T12:52:54Z

@rp- , will you run tests for this one?
All fine here, LGTM
so your “request changes” status is not longer valid?

sorry, did miss the status

jmsperu requested a review from rp- as a code owner April 28, 2026 08:31

boring-cyborgBot added component:primary-storage component:storage labels Apr 28, 2026

sureshanaparti requested a review from Copilot April 28, 2026 08:35

sureshanaparti added this to the 4.23.0 milestone Apr 28, 2026

Copilotstarted reviewing on behalf of sureshanaparti April 28, 2026 08:35 View session

CopilotAI reviewed Apr 28, 2026
View reviewed changes

jmsperu mentioned this pull request Apr 28, 2026
fix(linstor): surface ambiguous template fallbacks and legacy orphan cleanup #13078
Merged
4 tasks

winterhazel added status:needs-testing status:needs-review labels Apr 30, 2026

rp- requested changes May 1, 2026
View reviewed changes

Comment thread...e/volume/linstor/src/main/java/org/apache/cloudstack/storage/datastore/util/LinstorUtil.java Outdated

rp- approved these changes May 21, 2026
View reviewed changes

DaanHoogland merged commit 850b443 into apache:mainMay 21, 2026
27 of 28 checks passed

Navigation Menu

Uh oh!

fix(linstor): verify resource deletion completes; warn if stuck in DELETING - #13076

fix(linstor): verify resource deletion completes; warn if stuck in DELETING#13076
DaanHoogland merged 2 commits into
apache:mainfrom
jmsperu:feature/linstor-delete-verify

jmsperu commented Apr 28, 2026

Uh oh!

CopilotAI left a comment

Uh oh!

CopilotAIApr 28, 2026

Uh oh!

Uh oh!

Uh oh!

codecovBot commented Apr 28, 2026•
edited
Loading

Uh oh!

rp- left a comment

Uh oh!

Uh oh!

sureshanaparti commented May 14, 2026

Uh oh!

blueorangutan commented May 14, 2026

Uh oh!

blueorangutan commented May 14, 2026

Uh oh!

DaanHoogland commented May 20, 2026

Uh oh!

rp- commented May 20, 2026

Uh oh!

DaanHoogland commented May 21, 2026

Uh oh!

rp- commented May 21, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Uh oh!

Conversation

jmsperu commented Apr 28, 2026

Summary

What this PR does

Behaviour

What this does NOT do (deferred)

Test plan

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

CopilotAIApr 28, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

codecovBot commented Apr 28, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

rp- left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sureshanaparti commented May 14, 2026

Uh oh!

blueorangutan commented May 14, 2026

Uh oh!

blueorangutan commented May 14, 2026

Uh oh!

DaanHoogland commented May 20, 2026

Uh oh!

rp- commented May 20, 2026

Uh oh!

DaanHoogland commented May 21, 2026

Uh oh!

rp- commented May 21, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

codecovBot commented Apr 28, 2026•
edited
Loading