Storage pool response improvements by sureshanaparti · Pull Request #10740 · apache/cloudstack

sureshanaparti · 2025-04-16T15:43:33Z

Description

This PR has the following storage pool response improvements (addresses #8731):

adds details parameter to the response
adds capacitybytes parameter to the response, in sync with the capacityiops response parameter and createStoragePool cmd request parameter / db column in storage_pool table. (existing disksizetotal parameter in the storage pool response can be deprecated)
persists the capacityBytes and capacityIops in the storage_pool_details table while creating storage pool as well, for consistency - as these are updated with during update storage pool

capacitybytes in the storage_pool table is updated during the following events:

host connect/re-connect, using ModifyStoragePoolAnswer (from the capacityBytes in StoragePoolInfo)
storage stats collector when the capacityBytes of the pool is not same as the one collected
update storage pool cmd (and this is updated/sync-ed again by storage stats collector)

and update storage pool cmd persists the capacityBytes and capacityIops in the storage_pool_details table, and these values are not updated/synced (so, for consistency, create storage pool also persists the capacityBytes and capacityIops in the storage_pool_details table).

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
build/CI
test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

Tested create storage pool (NFS) with capacitybytes, capacityiops and some details.

Create storage pool (capacitybytes synced with the details from ModifyStoragePoolAnswer, detail parameter doesn't change) =>

(cmk) 🐱 > create storagepool scope=cluster zoneid=f3cc4c59-ceef-4471-aadc-83eea0bada6c podid=0d4a256f-214c-4700-b8f2-ced24959375b clusterid=e3e8fcfd-f9a9-43e5-a569-3965f8097067 name=pri4 provider=DefaultPrimary url=nfs://10.0.32.4/acs/primary/ref-trl-8357-k-Mol8-suresh-anaparti/pri4 capacitybytes=107374182400 capacityiops=10000 tags=pri3 details[0].key1=value1 details[0].key2=value2 details[0].key3=value3
{
"storagepool": {
"capacitybytes": 2197949513728,
"capacityiops": 10000,
"clusterid": "e3e8fcfd-f9a9-43e5-a569-3965f8097067",
"clustername": "p1-c1",
"created": "2025-04-16T19:24:55+0000",
"details": {
"capacityBytes": "107374182400",
"capacityIops": "10000",
"key1": "value1",
"key2": "value2",
"key3": "value3"
},
"disksizeallocated": 0,
"disksizetotal": 2197949513728,
"hasannotations": false,
"hypervisor": "KVM",
"id": "5de042fd-8f29-3b0e-a98d-f351524da85f",
"ipaddress": "10.0.32.4",
"istagarule": false,
"managed": false,
"name": "pri4",
"overprovisionfactor": "2.0",
"path": "/acs/primary/ref-trl-8357-k-Mol8-suresh-anaparti/pri4",
"podid": "0d4a256f-214c-4700-b8f2-ced24959375b",
"podname": "Pod1",
"provider": "DefaultPrimary",
"scope": "CLUSTER",
"state": "Up",
"tags": "pri3",
"type": "NetworkFilesystem",
"zoneid": "f3cc4c59-ceef-4471-aadc-83eea0bada6c",
"zonename": "ref-trl-8357-k-Mol8-suresh-anaparti"
}
}

List storage pool =>

(cmk) 🐱 > list storagepools id=5de042fd-8f29-3b0e-a98d-f351524da85f {
"count": 1,
"storagepool": [
{
"capacitybytes": 2197949513728,
"capacityiops": 10000,
"clusterid": "e3e8fcfd-f9a9-43e5-a569-3965f8097067",
"clustername": "p1-c1",
"created": "2025-04-16T19:24:55+0000",
"details": {
"capacityBytes": "107374182400",
"capacityIops": "10000",
"key1": "value1",
"key2": "value2",
"key3": "value3"
},
"disksizeallocated": 0,
"disksizetotal": 2197949513728,
"hasannotations": false,
"hypervisor": "KVM",
"id": "5de042fd-8f29-3b0e-a98d-f351524da85f",
"ipaddress": "10.0.32.4",
"istagarule": false,
"managed": false,
"name": "pri4",
"overprovisionfactor": "2.0",
"path": "/acs/primary/ref-trl-8357-k-Mol8-suresh-anaparti/pri4",
"podid": "0d4a256f-214c-4700-b8f2-ced24959375b",
"podname": "Pod1",
"provider": "DefaultPrimary",
"scope": "CLUSTER",
"state": "Up",
"storagecapabilities": {
"VOLUME_SNAPSHOT_QUIESCEVM": "false"
},
"tags": "pri3",
"type": "NetworkFilesystem",
"zoneid": "f3cc4c59-ceef-4471-aadc-83eea0bada6c",
"zonename": "ref-trl-8357-k-Mol8-suresh-anaparti"
}
]
}

Update storage pool (capacitybytes and capacityiops, actual and detail parameters updated) =>

(cmk) 🐱 > update storagepool id=5de042fd-8f29-3b0e-a98d-f351524da85f capacitybytes=107374182400 capacityiops=20000
{
"storagepool": {
"capacitybytes": 107374182400,
"capacityiops": 20000,
"clusterid": "e3e8fcfd-f9a9-43e5-a569-3965f8097067",
"clustername": "p1-c1",
"created": "2025-04-16T19:24:55+0000",
"details": {
"capacityBytes": "107374182400",
"capacityIops": "20000",
"key1": "value1",
"key2": "value2",
"key3": "value3"
},
"disksizeallocated": 0,
"disksizetotal": 107374182400,
"disksizeused": 1765468536832,
"hasannotations": false,
"hypervisor": "KVM",
"id": "5de042fd-8f29-3b0e-a98d-f351524da85f",
"ipaddress": "10.0.32.4",
"istagarule": false,
"managed": false,
"name": "pri4",
"overprovisionfactor": "2.0",
"path": "/acs/primary/ref-trl-8357-k-Mol8-suresh-anaparti/pri4",
"podid": "0d4a256f-214c-4700-b8f2-ced24959375b",
"podname": "Pod1",
"provider": "DefaultPrimary",
"scope": "CLUSTER",
"state": "Up",
"tags": "pri3",
"type": "NetworkFilesystem",
"zoneid": "f3cc4c59-ceef-4471-aadc-83eea0bada6c",
"zonename": "ref-trl-8357-k-Mol8-suresh-anaparti"
}
}

List storage pool (capacitybytes doesn't sync-ed with stats collector yet) =>

(cmk) 🐱 > list storagepools id=5de042fd-8f29-3b0e-a98d-f351524da85f {
"count": 1,
"storagepool": [
{
"capacitybytes": 107374182400,
"capacityiops": 20000,
"clusterid": "e3e8fcfd-f9a9-43e5-a569-3965f8097067",
"clustername": "p1-c1",
"created": "2025-04-16T19:24:55+0000",
"details": {
"capacityBytes": "107374182400",
"capacityIops": "20000",
"key1": "value1",
"key2": "value2",
"key3": "value3"
},
"disksizeallocated": 0,
"disksizetotal": 107374182400,
"disksizeused": 1765468536832,
"hasannotations": false,
"hypervisor": "KVM",
"id": "5de042fd-8f29-3b0e-a98d-f351524da85f",
"ipaddress": "10.0.32.4",
"istagarule": false,
"managed": false,
"name": "pri4",
"overprovisionfactor": "2.0",
"path": "/acs/primary/ref-trl-8357-k-Mol8-suresh-anaparti/pri4",
"podid": "0d4a256f-214c-4700-b8f2-ced24959375b",
"podname": "Pod1",
"provider": "DefaultPrimary",
"scope": "CLUSTER",
"state": "Up",
"storagecapabilities": {
"VOLUME_SNAPSHOT_QUIESCEVM": "false"
},
"tags": "pri3",
"type": "NetworkFilesystem",
"zoneid": "f3cc4c59-ceef-4471-aadc-83eea0bada6c",
"zonename": "ref-trl-8357-k-Mol8-suresh-anaparti"
}
]
}

List storage pool (capacitybytes sync-ed with stats collector, detail parameter doesn't change) =>

(cmk) 🐱 > list storagepools id=5de042fd-8f29-3b0e-a98d-f351524da85f {
"count": 1,
"storagepool": [
{
"capacitybytes": 2197949513728,
"capacityiops": 20000,
"clusterid": "e3e8fcfd-f9a9-43e5-a569-3965f8097067",
"clustername": "p1-c1",
"created": "2025-04-16T19:24:55+0000",
"details": {
"capacityBytes": "107374182400",
"capacityIops": "20000",
"key1": "value1",
"key2": "value2",
"key3": "value3"
},
"disksizeallocated": 0,
"disksizetotal": 2197949513728,
"disksizeused": 1765912084480,
"hasannotations": false,
"hypervisor": "KVM",
"id": "5de042fd-8f29-3b0e-a98d-f351524da85f",
"ipaddress": "10.0.32.4",
"istagarule": false,
"managed": false,
"name": "pri4",
"overprovisionfactor": "2.0",
"path": "/acs/primary/ref-trl-8357-k-Mol8-suresh-anaparti/pri4",
"podid": "0d4a256f-214c-4700-b8f2-ced24959375b",
"podname": "Pod1",
"provider": "DefaultPrimary",
"scope": "CLUSTER",
"state": "Up",
"storagecapabilities": {
"VOLUME_SNAPSHOT_QUIESCEVM": "false"
},
"tags": "pri3",
"type": "NetworkFilesystem",
"zoneid": "f3cc4c59-ceef-4471-aadc-83eea0bada6c",
"zonename": "ref-trl-8357-k-Mol8-suresh-anaparti"
}
]
}

DB =>

mysql> SELECT * FROM cloud.storage_pool WHERE name = 'pri4'\G;
*************************** 1. row ***************************
id: 4
name: pri4
uuid: 5de042fd-8f29-3b0e-a98d-f351524da85f
pool_type: NetworkFilesystem
port: 2049
data_center_id: 1
pod_id: 1
cluster_id: 1
used_bytes: 1765912084480
capacity_bytes: 2197949513728
host_address: 10.0.32.4
user_info: NULL
path: /acs/primary/ref-trl-8357-k-Mol8-suresh-anaparti/pri4
created: 2025-04-16 19:24:55
removed: NULL
update_time: 2025-04-16 19:26:47
status: Up
storage_provider_name: DefaultPrimary
scope: CLUSTER
hypervisor: KVM
managed: 0
capacity_iops: 20000
parent: 0
1 row in set (0.00 sec)
mysql> SELECT * FROM cloud.storage_pool_details WHERE pool_id = 4;
+----+---------+---------------+--------------+---------+
| id | pool_id | name | value | display |
+----+---------+---------------+--------------+---------+
| 14 | 4 | key1 | value1 | 1 |
| 15 | 4 | capacityIops | 20000 | 1 |
| 16 | 4 | key2 | value2 | 1 |
| 17 | 4 | key3 | value3 | 1 |
| 18 | 4 | capacityBytes | 107374182400 | 1 |
+----+---------+---------------+--------------+---------+
5 rows in set (0.00 sec)

How did you try to break this feature and the system with this change?

sureshanaparti · 2025-04-16T15:45:57Z

@blueorangutan package

blueorangutan · 2025-04-16T15:48:04Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

codecov · 2025-04-16T15:52:24Z

Codecov Report

❌ Patch coverage is 0% with 33 lines in your changes missing coverage. Please review.
✅ Project coverage is 17.42%. Comparing base (ec533cd) to head (0eeb75b).
⚠️ Report is 32 commits behind head on main.

Files with missing lines	Patch %	Lines
...cycle/CloudStackPrimaryDataStoreLifeCycleImpl.java	0.00%	20 Missing ⚠️
...e/cloudstack/api/response/StoragePoolResponse.java	0.00%	6 Missing ⚠️
...om/cloud/api/query/dao/StoragePoolJoinDaoImpl.java	0.00%	4 Missing ⚠️
.../storage/datastore/db/PrimaryDataStoreDaoImpl.java	0.00%	2 Missing ⚠️
...orage/volume/datastore/PrimaryDataStoreHelper.java	0.00%	1 Missing ⚠️

Additional details and impacted files

@@ Coverage Diff @@## main #10740 +/- ##
============================================
- Coverage 17.42% 17.42% -0.01% 
Complexity 15336 15336 ============================================
Files 5892 5892 Lines 526521 526547 +26 Branches 64293 64300 +7 ============================================
- Hits 91767 91747 -20 - Misses 424401 424448 +47 + Partials 10353 10352 -1

Flag	Coverage Δ
uitests	`3.61% <ø> (ø)`
unittests	`18.47% <0.00%> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

blueorangutan · 2025-04-16T17:05:12Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 13079

sureshanaparti · 2025-04-17T11:24:00Z

@blueorangutan package

blueorangutan · 2025-04-17T11:26:05Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-04-17T12:42:27Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 13094

sureshanaparti · 2025-04-17T13:26:13Z

@blueorangutan test

blueorangutan · 2025-04-17T13:28:04Z

@sureshanaparti a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2025-04-18T03:40:47Z

[SF] Trillian test result (tid-13037)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 48226 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10740-t13037-kvm-ol8.zip
Smoke tests completed. 133 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

DaanHoogland

clgtm but has been lingering for too long and needs an update.

blueorangutan · 2025-09-16T05:06:03Z

@sureshanaparti a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

DaanHoogland · 2025-09-16T08:31:01Z

@blueorangutan test keepEnv

blueorangutan · 2025-09-16T08:32:04Z

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2025-09-16T20:43:54Z

[SF] Trillian test result (tid-14352)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 53094 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10740-t14352-kvm-ol8.zip
Smoke tests completed. 146 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_03_deploy_and_scale_kubernetes_cluster	`Failure`	1.14	test_kubernetes_clusters.py
test_04_autoscale_kubernetes_cluster	`Failure`	26.72	test_kubernetes_clusters.py

blueorangutan · 2025-09-16T23:47:37Z

[SF] Trillian test result (tid-14353)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 51756 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10740-t14353-kvm-ol8.zip
Smoke tests completed. 143 look OK, 4 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_03_deploy_and_scale_kubernetes_cluster	`Failure`	1.15	test_kubernetes_clusters.py
test_04_autoscale_kubernetes_cluster	`Failure`	27.86	test_kubernetes_clusters.py
ContextSuite context=TestVPCRedundancy>:setup	`Error`	0.00	test_vpc_redundant.py
ContextSuite context=TestVPCNics>:setup	`Error`	0.00	test_vpc_router_nics.py
ContextSuite context=TestRVPCSite2SiteVpn>:setup	`Error`	0.00	test_vpc_vpn.py
ContextSuite context=TestVPCSite2SiteVPNMultipleOptions>:setup	`Error`	0.00	test_vpc_vpn.py
ContextSuite context=TestVpcRemoteAccessVpn>:setup	`Error`	0.00	test_vpc_vpn.py
ContextSuite context=TestVpcSite2SiteVpn>:setup	`Error`	0.00	test_vpc_vpn.py

DaanHoogland

lgtm verified in a lab env

github-actions · 2025-09-25T08:09:24Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

… update capacityBytes and capacityIops if applicable while creating storage pool

…th the capacityiops response parameter and createStoragePool cmd request parameter (existing disksizetotal parameter in the storage pool response can be deprecated)

…s table while creating storage pool as well, for consistency - as these are updated with during update storage pool

sureshanaparti · 2025-09-26T15:32:15Z

@blueorangutan package

blueorangutan · 2025-09-26T15:34:04Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-09-26T17:55:00Z

Packaging result [SF]: ✖️ el8 ✖️ el9 ✔️ debian ✖️ suse15. SL-JID 15194

sureshanaparti · 2025-09-26T18:16:18Z

@blueorangutan package

blueorangutan · 2025-09-26T18:18:04Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-09-26T19:36:09Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 15197

Pearl1594

code lgtm

* Return details of the storage pool in the response including url, and update capacityBytes and capacityIops if applicable while creating storage pool * Added capacitybytes parameter to the storage pool response in sync with the capacityiops response parameter and createStoragePool cmd request parameter (existing disksizetotal parameter in the storage pool response can be deprecated) * Don't keep url in details * Persist the capacityBytes and capacityIops in the storage_pool_details table while creating storage pool as well, for consistency - as these are updated with during update storage pool * rebase with main fixes

boring-cyborgBot added component:api component:primary-storage component:storage labels Apr 16, 2025

sureshanaparti linked an issue Apr 16, 2025 that may be closed by this pull request
listStoragePools return all fields #8731
Closed

DaanHoogland added component:upgrade status:needs-testing labels Apr 17, 2025

sureshanaparti added this to the 4.19.3 milestone Apr 17, 2025

sureshanaparti marked this pull request as ready for review April 17, 2025 11:24

sureshanaparti mentioned this pull request Apr 17, 2025
listStoragePools return all fields #8731
Closed

DaanHoogland added this to ACS 4.20.1 May 1, 2025

sureshanaparti moved this to In Progress in ACS 4.20.1 May 5, 2025

sureshanaparti requested review from DaanHoogland, harikrishna-patnala, shwstppr and weizhouapache May 6, 2025 07:08

yadvr requested a review from Pearl1594 May 9, 2025 04:58

DaanHoogland force-pushed the 4.19 branch from 8ef8b46 to 857ccb0Compare May 27, 2025 14:19

sureshanaparti modified the milestones: 4.19.3, 4.19.4 Jun 5, 2025

DaanHoogland reviewed Jul 10, 2025
View reviewed changes

DaanHoogland self-assigned this Sep 16, 2025

DaanHoogland approved these changes Sep 17, 2025
View reviewed changes

sureshanaparti removed the status:needs-testing label Sep 23, 2025

github-actionsBot added the status:has-conflicts label Sep 25, 2025

sureshanapartiand others added 5 commits September 26, 2025 17:05

Return details of the storage pool in the response including url, and…
70889d3
… update capacityBytes and capacityIops if applicable while creating storage pool

Added capacitybytes parameter to the storage pool response in sync wi…
49de5cc
…th the capacityiops response parameter and createStoragePool cmd request parameter (existing disksizetotal parameter in the storage pool response can be deprecated)

Don't keep url in details
a4e61e8

Persist the capacityBytes and capacityIops in the storage_pool_detail…
bd0a9d7
…s table while creating storage pool as well, for consistency - as these are updated with during update storage pool

rebase with main fixes
0eeb75b

sureshanaparti force-pushed the storage_pool_response_improvements branch from 9057537 to 0eeb75bCompare September 26, 2025 11:37

github-actionsBot removed the status:has-conflicts label Sep 26, 2025

Pearl1594 approved these changes Sep 29, 2025
View reviewed changes

harikrishna-patnala merged commit 09b63bc into apache:mainOct 8, 2025
27 of 28 checks passed

github-project-automationBot moved this from In Progress to Done in Apache CloudStack 4.22.0 Oct 8, 2025

harikrishna-patnala deleted the storage_pool_response_improvements branch October 8, 2025 05:50

DaanHoogland removed their assignment Oct 8, 2025

Uh oh!

Conversation

sureshanaparti commented Apr 16, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

sureshanaparti commented Apr 16, 2025

Uh oh!

blueorangutan commented Apr 16, 2025

Uh oh!

codecovBot commented Apr 16, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

blueorangutan commented Apr 16, 2025

Uh oh!

sureshanaparti commented Apr 17, 2025

Uh oh!

blueorangutan commented Apr 17, 2025

Uh oh!

blueorangutan commented Apr 17, 2025

Uh oh!

sureshanaparti commented Apr 17, 2025

Uh oh!

blueorangutan commented Apr 17, 2025

Uh oh!

blueorangutan commented Apr 18, 2025

Uh oh!

DaanHoogland left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

blueorangutan commented Sep 16, 2025

Uh oh!

DaanHoogland commented Sep 16, 2025

Uh oh!

blueorangutan commented Sep 16, 2025

Uh oh!

blueorangutan commented Sep 16, 2025

Uh oh!

blueorangutan commented Sep 16, 2025

Uh oh!

DaanHoogland left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Sep 25, 2025

Uh oh!

sureshanaparti commented Sep 26, 2025

Uh oh!

blueorangutan commented Sep 26, 2025

Uh oh!

blueorangutan commented Sep 26, 2025

Uh oh!

sureshanaparti commented Sep 26, 2025

Uh oh!

blueorangutan commented Sep 26, 2025

Uh oh!

blueorangutan commented Sep 26, 2025

Uh oh!

Pearl1594 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

sureshanaparti commented Apr 16, 2025•
edited
Loading

codecovBot commented Apr 16, 2025•
edited
Loading