Incorrect Disk usage is reported when using Ceph erasure code pool

[kiranchavala]

problem

Incorrect Disk usage is reported when a Ceph storage

versions

ACS 4.22.1
ceph version
ceph version 20.2.2 (0fcffee29411e3a38036764817b6e1afc59741cc) tentacle (stable - RelWithDebInfo)
Ubuntu 24.0.4 kvm host

The steps to reproduce the bug

CloudStack added support for ceph erasure code pool in the 4.21 release

github.com/apache/cloudstack/pull/9808

Steps to reproduce the issue

1. Create a ceph-storage with Erasure code setting

ceph osd erasure-code-profile set cloudstack-ec k=2 m=1 crush-failure-domain=host
ceph osd pool create cloudstack-ec 64 erasure cloudstack-ec
ceph osd pool set cloudstack-ec allow_ec_overwrites true
ceph osd pool application enable cloudstack-ec rbd

ceph osd pool create cloudstack-ec-metadata 32 replicated
ceph osd pool set cloudstack-ec-metadata size 3
rbd pool init cloudstack-ec-metadata

2. Add the storage pool in cloudstack

createStoragePool scope=zone zoneid=<ZONE_ID> hypervisor=KVM \
 name=MyCephECCluster provider=DefaultPrimary \
 url=rbd://cloudstack:<RADOS_SECRET>@192.168.55.11/cloudstack-ec-metadata \
 details[0].key=dataPool details[0].value=cloudstack-ec \
 tags=RBD-EC

3. Check the disk usage from ceph side

root@ceph1:/home/ubuntu# ceph df
--- RAW STORAGE ---
CLASS SIZE AVAIL USED RAW USED %RAW USED
hdd 150 GiB 150 GiB 200 MiB 200 MiB 0.13
TOTAL 150 GiB 150 GiB 200 MiB 200 MiB 0.13

--- POOLS ---
POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL

cloudstack-ec 3 64 8 KiB 1 12 KiB 0 95 GiB
cloudstack-ec-metadata 4 32 7.9 KiB 3 36 KiB 0 47 GiB

4. Check the Disk used size from Cloudstack

5. Deploy a data disk of size lets say 5gb on the ceph storage pool

6. Attach it to a running vm

7. Populate some data on the data disk

dd if=/dev/urandom of=/dev/vdb bs=1M count=2048 status=progress oflag=direct

8. Check the ceph disk usage

root@ceph1:/home/ubuntu# ceph df
--- RAW STORAGE ---
CLASS SIZE AVAIL USED RAW USED %RAW USED
hdd 150 GiB 147 GiB 3.2 GiB 3.2 GiB 2.13
TOTAL 150 GiB 147 GiB 3.2 GiB 3.2 GiB 2.13

--- POOLS ---
POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL
.mgr 1 1 577 KiB 2 1.7 MiB 0 46 GiB
cloudstack 2 64 6.8 KiB 3 32 KiB 0 46 GiB
cloudstack-ec 3 64 2.0 GiB 513 3.0 GiB 2.11 93 GiB
cloudstack-ec-metadata 4 32 9.1 KiB 6 62 KiB 0 46 GiB

9. Check the cloudstack storage used

Cloudstack is reporting the used size from the replicated metadata pool and not the actual erasure code pool

What to do about it?

CloudStack should report the correct disk size used from the erasure -code pool and not the replicated metadata stats

[kiranchavala]

Based on the discussion in

#5500

CloudStack gets the used bytes information from libvirt.

root@Cloudstack-Kvm-Host3:/home/ubuntu# virsh pool-info 15e9a03c-6eb1-3d3c-8cac-5918f0c75f0f
Name: 15e9a03c-6eb1-3d3c-8cac-5918f0c75f0f
UUID: 15e9a03c-6eb1-3d3c-8cac-5918f0c75f0f
State: running
Persistent: no
Autostart: no
Capacity: 149.99 GiB
Allocation: 62.89 KiB
Available: 146.79 GiB

[wido]

Based on the discussion in

#5500

CloudStack gets the used bytes information from libvirt.

root@Cloudstack-Kvm-Host3:/home/ubuntu# virsh pool-info 15e9a03c-6eb1-3d3c-8cac-5918f0c75f0f
Name: 15e9a03c-6eb1-3d3c-8cac-5918f0c75f0f
UUID: 15e9a03c-6eb1-3d3c-8cac-5918f0c75f0f
State: running
Persistent: no
Autostart: no
Capacity: 149.99 GiB
Allocation: 62.89 KiB
Available: 146.79 GiB

That is correct. CloudStack leverages Libvirt for this. If this is broken or wrong in Libvirt, it has to be fixed there. There isn't much CloudStack can do here.

I've set the priority to low as this is outside CloudStack's scope and assigned it to myself.

Will probably close this one soon(ish).

[kiranchavala]

@wido Wondering if we should consider it as an enhancement

Bypassing libvirt entirely and querying Ceph directly via restful api to get the usage stats of a erasure code pool or a ceph storage pool

curl -k -u admin: https://<>:8443/api/pool

The usage reporting in cloudstack could also get affected because of querying the data from libvirt

[wido]

@wido Wondering if we should consider it as an enhancement

Bypassing libvirt entirely and querying Ceph directly via restful api to get the usage stats of a erasure code pool or a ceph storage pool

curl -k -u admin: https://<>:8443/api/pool

The usage reporting in cloudstack could also get affected because of querying the data from libvirt

I doubt if that's a good idea. Technically, yes, it could be done, but I wonder if we can get this to work. Ceph is implemented in the DefaultPrimartStorageDriver and this relies on Libvirt. If we want to do this cleanly we should write a separate Ceph driver for CloudStack, a plugin.

Maybe Libvirt can be improved, getting a patch into Libvirt might be better and more future proof.

[DaanHoogland]

this sounds like we want storage extensions now ;)