Dear Ceph users,I'd need some help with CephFS. Originally, my cluster consisted of 12 hosts with different total raw disk size, ranging from 16 to 32 TB; there were two pools, one 3x replicated metadata and one 6+2 EC data with host failure domain, and which I access via CephFS. In total there were 512 PGs.
Naively, I would expect that the available size of the data pool would be about 75% of the total available space given the EC parameters (6/(6+2) = 75%), but I noticed that it was actually lower, around 66% (as reported by df -h, while for the total raw capacity I used the one reported by the Ceph dashboard). I guessed this could be due to smaller hosts limiting the total amount of available space for the filesystem, so I started a disk upgrade campaign to make the host sizes more even. For example, for a 16 TB host with 8x2TB I replaced two disks with 8 TB ones, for a new raw size of 28 TB. I did this and similar upgrades for other hosts in steps, every time waiting for backfill to finish and checking the new filesystem size before proceeding. And I noticed that the filesystem size reported by df -h always remained at 66% of the total raw capacity, despite host sizes are now all between 24 and 32 TB.
So my guess about the limiting factor for the relative fs size is evidently wrong. I thought that for some reason maybe 66% is the actual limit and also about some possible explanation (e.g. some space being reserved for the metadata pool), and even considered the possibility that the size reported by df -h being meaningless, but I'd need some advice from an expert to sort this out.
Thanks in advance for any help. Nicola
Attachment:
smime.p7s
Description: S/MIME Cryptographic Signature
_______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
