Hi Ali,
> Date: Sat, 05 Oct 2013 09:22:22 +0300
> From: Linux Chips <linux.chips@xxxxxxxxx>
> To: ceph-users@xxxxxxxxxxxxxx
> Subject: ceph uses too much disk space!!
> Message-ID: <524FB01E.3000907@xxxxxxxxx>
> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>
> Hi every one;
> we have a small testing cluster, one node with 4 OSDs of 3TB each. i
> created one RBD image of 4TB. now the cluster is nearly full:
<SNIP>
> /dev/sda 2.8T 2.1T 566G 79% /var/lib/ceph/osd/ceph-0
> /dev/sdb 2.8T 2.4T 316G 89% /var/lib/ceph/osd/ceph-1
> /dev/sdc 2.8T 2.2T 457G 84% /var/lib/ceph/osd/ceph-2
> /dev/sdd 2.8T 2.2T 447G 84% /var/lib/ceph/osd/ceph-3
>
> # ceph osd pool get rbd min_size
> min_size: 1
>
> # ceph osd pool get rbd size
> size: 2
>
>
> 4 disk at 3TB should give me 12TB, and 4TBx2 should be 8TB. that is 66%
> not 80% as the ceph df shows (%RAW).
> where is this space is leaking? how can i fix it?
> or is this normal behavior and this is due to overhead?
> /dev/sda 2.8T 2.1T 566G 79% /var/lib/ceph/osd/ceph-0
> /dev/sdb 2.8T 2.4T 316G 89% /var/lib/ceph/osd/ceph-1
> /dev/sdc 2.8T 2.2T 457G 84% /var/lib/ceph/osd/ceph-2
> /dev/sdd 2.8T 2.2T 447G 84% /var/lib/ceph/osd/ceph-3
>
> # ceph osd pool get rbd min_size
> min_size: 1
>
> # ceph osd pool get rbd size
> size: 2
>
>
> 4 disk at 3TB should give me 12TB, and 4TBx2 should be 8TB. that is 66%
> not 80% as the ceph df shows (%RAW).
> where is this space is leaking? how can i fix it?
> or is this normal behavior and this is due to overhead?
I'm not sure what overhead there might be from Ceph's metadata, but I think you might be basing your calculations on bad assumptions to begin with:
1) define your 3TB OSD drive size properly (probably 3000GB or slightly less)
2) you need to account for OSD filesystem overhead, e.g., format a drive and take a look at usage (probably <95% of raw capacity)
Indeed, if you look at your df output you've got ~2.8TB of total capacity per drive, and if you observed that prior to writing any data you'd see some already in-use by the filesystem. And df is showing just over 2TB per drive used, which makes sense given you've created a 4TiB rbd in a storage pool with a replication factor of 2 (i.e., 2 copies of your rbd are stored across the osds).
Also, keep in mind that Ceph is breaking up your rbd into chunks on the host filesystem/s and probably storing metadata (in extended attributes) for every chunk.
--
Cheers,
~Blairo
Cheers,
~Blairo
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
