fix(linstor): surface ambiguous template fallbacks and legacy orphan cleanup by jmsperu · Pull Request #13078 · apache/cloudstack

jmsperu · 2026-04-28T21:02:35Z

Summary

Two small visibility improvements to the LINSTOR template-handling path. Both preserve existing behaviour and only add log output that surfaces conditions operators have currently no easy way to detect.

1. `LinstorUtil.findResourceDefinition` — log on ambiguous fallback

When clone-from-template runs, the method scans for a resource whose name starts with the template prefix AND whose aux properties include `_cs-template-for-`. If no exact-property match exists, the method falls back to the first matching resource by name and returns it silently.

In setups with multiple resource groups on the same controller (or with legacy templates cached before the ref-count convention was added), this fallback can return a template that belongs to the wrong resource group, and the clone produces an unexpected result.

This change keeps the fallback behaviour but logs a WARN naming:

the requested resource name and resource group
the fallback resource that was chosen
the actual `_cs-template-for-*` aux properties present on the fallback (so operators can see what RGs do claim it)

2. `LinstorStorageAdaptor.deRefOrDeleteResource` — log on legacy-template orphan cleanup

The ref-count branch already deletes resources that have zero remaining `_cs-template-for-` aux properties. Two conditions reach this branch:

a normal resource (no template-for properties expected) — already commented in code
a legacy template with no template-for properties (predates the ref-count convention)

Currently both look identical in the logs. Operators upgrading from older versions can't tell how many orphan legacy templates were cleaned up.

This change logs an INFO line in the second case, identifying the resource as a legacy template and naming the resource group context. Behaviour unchanged.

Test plan

CI build + unit tests
Manual: deploy a VM from template in a single-RG setup — no new logs
Manual: deploy a VM from template in a multi-RG setup where the template lacks the exact aux property — observe the new WARN
Manual: evict a legacy template (no aux properties) — observe the new INFO line during cleanup

Why three small PRs

Per the suggestion from earlier review, splitting LINSTOR plugin improvements into focused PRs:

This PR (template hygiene)
fix(linstor): verify resource deletion completes; warn if stuck in DELETING #13076 (delete-verification — addresses stuck DELETING accumulation)
fix(linstor): pre-flight check destination is a LINSTOR satellite before live migration #13077 (live-migration pre-flight — catches not-a-LINSTOR-satellite up-front)

All three target the LINSTOR primary-storage plugin and can be reviewed independently.

…e orphans Two small visibility improvements that make existing template behaviour easier to audit, especially after upgrading from versions that predated the ref-count convention. LinstorUtil.findResourceDefinition When no resource has the exact "_cs-template-for-<rscGrpName>" Aux property, the method silently returned the first resource whose name starts with the requested prefix. With multiple resource groups on a single controller, this can route a clone to the wrong template. Now logs a WARN naming the requested rscGrpName, the fallback resource, and the actual aux properties present. Behaviour unchanged: still returns the first match. Operators can act on the warning by setting the property explicitly or removing the unrelated definition. LinstorStorageAdaptor.deRefOrDeleteResource When deleting a resource that has zero `_cs-template-for-` aux properties AND whose name starts with the template-name prefix the caller is acting on, log an INFO line. These are legacy templates cached before the ref-count convention was introduced — they get picked up by the existing "if expectedProps == 0" branch and deleted. Surfacing them lets operators see how many orphans existed at upgrade time and confirm the cleanup happened.

sureshanaparti · 2026-04-29T05:39:51Z

@blueorangutan package

blueorangutan · 2026-04-29T05:42:04Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

Copilot

Pull request overview

This PR improves operator visibility in the LINSTOR template-handling path by adding targeted log lines around (1) ambiguous template selection fallback and (2) legacy/orphan template cleanup, while intending to preserve existing behavior.

Changes:

Add a WARN log in LinstorUtil.findResourceDefinition when falling back to the first name-prefix match due to missing expected _cs-template-for-<rscGrp> aux property.
Add an INFO log in LinstorStorageAdaptor.deRefOrDeleteResource intended to surface deletion of legacy templates that have zero _cs-template-for- aux properties.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
`plugins/storage/volume/linstor/src/main/java/org/apache/cloudstack/storage/datastore/util/LinstorUtil.java`	Adds WARN logging to make ambiguous template fallback selection visible (includes chosen fallback and related aux keys).
`plugins/storage/volume/linstor/src/main/java/com/cloud/hypervisor/kvm/storage/LinstorStorageAdaptor.java`	Adds INFO logging intended to distinguish legacy-template orphan cleanup during deref/delete logic.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-04-29T05:44:03Z

+ // Surface the case where a resource that LOOKS like a template (resource name
+ // starts with the requested prefix) has zero `_cs-template-for-` aux properties
+ // even though we never decremented one — that's a legacy template predating the
+ // ref-count convention. Logging it before deletion lets operators audit how
+ // many such orphans existed at upgrade time.
+ if (expectedProps == 0 && rd.getName().toLowerCase().startsWith(rscName.toLowerCase())) {
+ logger.info("Linstor: deleting resource {} which has no _cs-template-for- aux properties " +
+ "(legacy template predating the ref-count convention, or a stale orphan). " +
+ "Resource group context: {}", rd.getName(), rscGrpName);
+ }


The new INFO log for “legacy template predating the ref-count convention” will currently fire for all non-template resource deletions too. existingRDs is already built via getRDListStartingWith(api, rscName), so rd.getName().toLowerCase().startsWith(rscName.toLowerCase()) is always true; whenever expectedProps == 0 (normal volume/resource with no _cs-template-for- props), this will emit the legacy-template message and add significant log noise/misleading attribution. Tighten the condition so it only matches the intended legacy-template-orphan case (e.g., by using a reliable template discriminator, passing an isTemplate flag from the caller, or checking for a template-specific naming/property convention), and avoid logging for ordinary resource deletions.
Suggested change
// Surface the case where a resource that LOOKS like a template (resource name
// starts with the requested prefix) has zero `_cs-template-for-` aux properties
// even though we never decremented one — that's a legacy template predating the
// ref-count convention. Logging it before deletion lets operators audit how
// many such orphans existed at upgrade time.
if (expectedProps == 0 && rd.getName().toLowerCase().startsWith(rscName.toLowerCase())) {
logger.info("Linstor: deleting resource {} which has no _cs-template-for- aux properties " +
"(legacy template predating the ref-count convention, or a stale orphan). " +
"Resource group context: {}", rd.getName(), rscGrpName);
}

copilot has a point here. the extra .startWith, will always be true, as the existingRD will exactly only contain resources only such resources.

codecov · 2026-04-29T05:48:11Z

Codecov Report

❌ Patch coverage is 0% with 17 lines in your changes missing coverage. Please review.
✅ Project coverage is 18.10%. Comparing base (6f4445c) to head (4ee8d94).
⚠️ Report is 56 commits behind head on main.

Files with missing lines	Patch %	Lines
...cloudstack/storage/datastore/util/LinstorUtil.java	0.00%	11 Missing ⚠️
.../hypervisor/kvm/storage/LinstorStorageAdaptor.java	0.00%	6 Missing ⚠️

Additional details and impacted files

@@ Coverage Diff @@## main #13078 +/- ##
============================================
+ Coverage 18.02% 18.10% +0.07% - Complexity 16621 16750 +129 
============================================
Files 6029 6037 +8 Lines 542184 542946 +762 Branches 66451 66489 +38 ============================================
+ Hits 97740 98304 +564 - Misses 433428 433592 +164 - Partials 11016 11050 +34

Flag	Coverage Δ
uitests	`3.51% <ø> (-0.01%)`	⬇️
unittests	`19.27% <0.00%> (+0.08%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

blueorangutan · 2026-04-29T07:01:41Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 17649

DaanHoogland · 2026-04-29T07:37:20Z

@rp- , will you have a look at this?

rp- · 2026-04-29T14:33:54Z

@rp- , will you have a look at this?

yes

rp-

I would probably remove the unnecessary .startsWith() condition.
I guess I don't mind about the too much logging part.
Functional my tests ran through

rp- · 2026-05-05T14:15:49Z

+ // Surface the case where a resource that LOOKS like a template (resource name
+ // starts with the requested prefix) has zero `_cs-template-for-` aux properties
+ // even though we never decremented one — that's a legacy template predating the
+ // ref-count convention. Logging it before deletion lets operators audit how
+ // many such orphans existed at upgrade time.
+ if (expectedProps == 0 && rd.getName().toLowerCase().startsWith(rscName.toLowerCase())) {
+ logger.info("Linstor: deleting resource {} which has no _cs-template-for- aux properties " +
+ "(legacy template predating the ref-count convention, or a stale orphan). " +
+ "Resource group context: {}", rd.getName(), rscGrpName);
+ }


copilot has a point here. the extra .startWith, will always be true, as the existingRD will exactly only contain resources only such resources.

@rp-

existingRDs already filtered by getRDListStartingWith(rscName), so the .startsWith() guard is always true. Per @rp- review on PR apache#13078.

jmsperu · 2026-05-09T03:43:22Z

@rp- thanks — pushed eff9b09d51 dropping the redundant .startsWith() check. Logging stays as before for the legacy/orphan case.

sureshanaparti · 2026-05-09T06:31:04Z

@blueorangutan package

blueorangutan · 2026-05-09T06:32:04Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2026-05-09T07:27:06Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 17800

rp-

LGTM

sureshanaparti · 2026-05-11T08:33:52Z

@blueorangutan test

blueorangutan · 2026-05-11T08:36:04Z

@sureshanaparti a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2026-05-12T08:55:15Z

[SF] Trillian test result (tid-16066)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 84675 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr13078-t16066-kvm-ol8.zip
Smoke tests completed. 146 look OK, 5 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_vm_backup_create_vm_from_backup	`Failure`	607.86	test_backup_recovery_nas.py
test_vm_backup_lifecycle	`Error`	0.12	test_backup_recovery_nas.py
ContextSuite context=TestSharedFSLifecycle>:setup	`Error`	0.00	test_sharedfs_lifecycle.py
test_10_attachAndDetach_iso	`Failure`	607.15	test_vm_life_cycle.py
test_01_create_vm_snapshots	`Failure`	606.87	test_vm_snapshots.py
test_02_revert_vm_snapshots	`Failure`	600.75	test_vm_snapshots.py
test_03_delete_vm_snapshots	`Failure`	0.04	test_vm_snapshots.py
test_01_create_volume	`Failure`	610.36	test_volumes.py
test_01_root_volume_encryption	`Failure`	716.76	test_volumes.py
test_02_data_volume_encryption	`Failure`	641.91	test_volumes.py
test_03_root_and_data_volume_encryption	`Failure`	663.97	test_volumes.py
test_02_attach_volume	`Failure`	1271.46	test_volumes.py
test_02_attach_volume	`Failure`	1271.48	test_volumes.py
test_03_download_attached_volume	`Failure`	666.67	test_volumes.py
test_04_delete_attached_volume	`Failure`	665.75	test_volumes.py
test_05_detach_volume	`Failure`	753.81	test_volumes.py
test_06_download_detached_volume	`Failure`	846.55	test_volumes.py
test_07_resize_fail	`Failure`	663.69	test_volumes.py
test_08_resize_volume	`Failure`	670.11	test_volumes.py
test_09_delete_detached_volume	`Failure`	664.85	test_volumes.py
test_10_list_volumes	`Failure`	661.43	test_volumes.py
test_11_attach_volume_with_unstarted_vm	`Failure`	770.59	test_volumes.py
test_12_resize_volume_with_only_size_parameter	`Failure`	677.40	test_volumes.py
test_13_migrate_volume_and_change_offering	`Failure`	808.59	test_volumes.py
test_14_delete_volume_delete_protection	`Failure`	663.94	test_volumes.py

blueorangutan · 2026-05-20T07:46:01Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 17922

blueorangutan · 2026-05-20T15:37:35Z

[SF] Trillian Build Failed (tid-16130)

jmsperu · 2026-05-22T19:50:48Z

Hi @sureshanaparti @DaanHoogland — the Trillian build failed on the 2026-05-20 run (tid-16130) after packaging succeeded on el8/el9/el10/debian/suse15, so this looks like a transient infra issue rather than anything in the patch. The previous Trillian run (tid-16066, 2026-05-12) had 146 OK / 5 environmental failures. Could one of you re-trigger with @blueorangutan test when convenient? Approval from @rp- still stands from 2026-05-11. Happy to push a no-op commit if that's easier. Thanks!

One-word refinement — 'first-match selection' -> 'first-match fallback' to align with the local 'fallback' variable name and the surrounding prose. No behavioural change. Force-triggers a fresh CI run on this PR to clear the stale 2026-05-09 component-build failure that's been keeping mergeStateStatus at UNSTABLE.

blueorangutan · 2026-05-22T22:49:20Z

[SF] Trillian test result (tid-16156)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 51440 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr13078-t16156-kvm-ol8.zip
Smoke tests completed. 151 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

blueorangutan · 2026-05-26T09:14:44Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 18036

DaanHoogland · 2026-05-26T14:42:38Z

@blueorangutan test

blueorangutan · 2026-05-26T14:44:04Z

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2026-05-27T06:52:08Z

[SF] Trillian test result (tid-16184)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 55268 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr13078-t16184-kvm-ol8.zip
Smoke tests completed. 149 look OK, 2 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_02_list_cpvm_vm	`Failure`	0.04	test_ssvm.py
test_04_cpvm_internals	`Failure`	0.04	test_ssvm.py
test_10_reboot_cpvm_forced	`Error`	9.32	test_ssvm.py
test_01_redundant_vpc_site2site_vpn	`Failure`	416.51	test_vpc_vpn.py
test_01_redundant_vpc_site2site_vpn	`Error`	416.55	test_vpc_vpn.py

jmsperu requested a review from rp- as a code owner April 28, 2026 21:02

boring-cyborgBot added component:primary-storage component:storage labels Apr 28, 2026

sureshanaparti requested a review from Copilot April 29, 2026 05:39

Copilotstarted reviewing on behalf of sureshanaparti April 29, 2026 05:40 View session

CopilotAI reviewed Apr 29, 2026
View reviewed changes

sureshanaparti added the 3rd-party:linstor label Apr 29, 2026

sureshanaparti added this to the 4.23.0 milestone Apr 29, 2026

winterhazel added status:needs-testing status:needs-review labels Apr 30, 2026

rp- requested changes May 5, 2026
View reviewed changes

fix(linstor): drop redundant startsWith check in deRefOrDeleteResource
eff9b09
existingRDs already filtered by getRDListStartingWith(rscName), so the .startsWith() guard is always true. Per @rp- review on PR apache#13078.

rp- approved these changes May 11, 2026
View reviewed changes

DaanHoogland approved these changes May 27, 2026
View reviewed changes

DaanHoogland merged commit 6786d45 into apache:mainMay 27, 2026
26 of 27 checks passed

This was referenced Jun 1, 2026
[repo-status] Daily Status Report — June 1, 2026 #13317
Closed
[repo-status] Daily Status Report – June 4, 2026 #13350
Closed
[repo-status] Daily Status Report – June 5, 2026 #13360
Closed

Uh oh!

Conversation

jmsperu commented Apr 28, 2026

Summary

1. `LinstorUtil.findResourceDefinition` — log on ambiguous fallback

2. `LinstorStorageAdaptor.deRefOrDeleteResource` — log on legacy-template orphan cleanup

Test plan

Why three small PRs

Uh oh!

sureshanaparti commented Apr 29, 2026

Uh oh!

blueorangutan commented Apr 29, 2026

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

CopilotAIApr 29, 2026

Choose a reason for hiding this comment

Uh oh!

rp-May 5, 2026

Choose a reason for hiding this comment

Uh oh!

codecovBot commented Apr 29, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

blueorangutan commented Apr 29, 2026

Uh oh!

DaanHoogland commented Apr 29, 2026

Uh oh!

rp- commented Apr 29, 2026

Uh oh!

rp- left a comment

Choose a reason for hiding this comment

Uh oh!

rp-May 5, 2026

Choose a reason for hiding this comment

Uh oh!

jmsperu commented May 9, 2026

Uh oh!

sureshanaparti commented May 9, 2026

Uh oh!

blueorangutan commented May 9, 2026

Uh oh!

blueorangutan commented May 9, 2026

Uh oh!

rp- left a comment

Choose a reason for hiding this comment

Uh oh!

sureshanaparti commented May 11, 2026

Uh oh!

blueorangutan commented May 11, 2026

Uh oh!

blueorangutan commented May 12, 2026

Uh oh!

blueorangutan commented May 20, 2026

Uh oh!

blueorangutan commented May 20, 2026

Uh oh!

jmsperu commented May 22, 2026

Uh oh!

blueorangutan commented May 22, 2026

Uh oh!

blueorangutan commented May 26, 2026

Uh oh!

DaanHoogland commented May 26, 2026

Uh oh!

blueorangutan commented May 26, 2026

Uh oh!

blueorangutan commented May 27, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

codecovBot commented Apr 29, 2026•
edited
Loading