Hi folks,
doing the last cleanup for https://github.com/ceph/ceph/pull/19454
I realized that we still include the space for separate DB volume into
total (and hence treat it as used too) space reported by BlueStore.
It seems we had such a discussion a while ago but unfortunately I don't
recall the results.
See BlueStore::statfs(...)
if (bluefs) {
// include dedicated db, too, if that isn't the shared device.
if (bluefs_shared_bdev != BlueFS::BDEV_DB) {
buf->total += bluefs->get_total(BlueFS::BDEV_DB);
}
I'm not sure if there are any rationales behind that. And have a strong
desire to remove it from 'total' calculation.
Just want to share two options for the new "ceph df' output.
3x OSD config: 10 Gb block device + 1Gb DB device + 1 Gb WAL device.
1) DB space included.
GLOBAL:
SIZE AVAIL USED RAW USED %RAW USED
33 GiB 27 GiB 2.9 GiB 5.9 GiB 18.01
POOLS:
NAME ID STORED OBJECTS USED
%USED MAX AVAIL
cephfs_data_a 1 0 B 0 0
B 0 8.9 GiB
cephfs_metadata_a 2 2.2 KiB 22 384
KiB 0 8.9 GiB
2) DB space isn't included.
GLOBAL:
SIZE AVAIL USED RAW USED %RAW USED
30 GiB 27 GiB 1.9 MiB 3.0 GiB 10.01
POOLS:
NAME ID STORED OBJECTS USED
%USED MAX AVAIL
cephfs_data_a 1 0 B 0 0
B 0 8.9 GiB
cephfs_metadata_a 2 2.2 KiB 22 384
KiB 0 8.9 GiB
So for the first case GLOBAL SIZE includes both block and DB devices
space. RAW USED includes 3GB for separate BlueFS volumes and ~3GB
permanently reserved for BlueFS as slow device as per
bluestore_bluefs_min_free config parameter. Not to mention space
actually allocated for user data.
And AVAIL is equal to SIZE - RAW USED;
For the second option (I'm inclined to) all the numbers lack that DB
device space and hence IMO provide more consistent picture.
Please note that 3x1GB allocated for WAL aren't taken into account in
both cases.
So the question is what variant do we prefer? Are there any reasons to
account for DB space here?
Doesn't it make sense to export BlueFS stats (total/avail) separately
which (along with existing "internal_metadata" and "omap_allocated"
fields) allows to build a perfect and consistent view of DB space usage
if needed?
Thanks,
Igor
