Re: Quincy: cephfs "df" used 6x higher than "du"

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Dear All,

Just noticed that ceph osd df shows "Raw Use" of ~360 GiB per OSD, with 65GiB Data stored, see below.

Is the disparity between du and df due to low level OSD data structures (?) consuming a large proportion of space (~300GiB per OSD, 130TB total), compared to the 25TB of actual data?

If so, should we expect the disparity in used space to decrease as we store more data on the cluster?


[root@wilma-s1 ~]# ceph osd df | head -10
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 52 hdd 16.66209 1.00000 17 TiB 365 GiB 67 GiB 2 KiB 960 MiB 16 TiB 2.14 1.01 99 up 53 hdd 16.66209 1.00000 17 TiB 365 GiB 67 GiB 2 KiB 904 MiB 16 TiB 2.14 1.01 99 up 54 hdd 16.66209 1.00000 17 TiB 360 GiB 62 GiB 2 KiB 878 MiB 16 TiB 2.11 1.00 92 up 55 hdd 16.66209 1.00000 17 TiB 362 GiB 64 GiB 2 KiB 838 MiB 16 TiB 2.12 1.00 96 up 56 hdd 16.66209 1.00000 17 TiB 365 GiB 67 GiB 1 KiB 855 MiB 16 TiB 2.14 1.01 99 up 57 hdd 16.66209 1.00000 17 TiB 359 GiB 61 GiB 2 KiB 915 MiB 16 TiB 2.11 0.99 92 up 58 hdd 16.66209 1.00000 17 TiB 361 GiB 63 GiB 2 KiB 853 MiB 16 TiB 2.11 1.00 93 up 59 hdd 16.66209 1.00000 17 TiB 359 GiB 61 GiB 1 KiB 815 MiB 16 TiB 2.11 0.99 91 up 60 hdd 16.66209 1.00000 17 TiB 365 GiB 67 GiB 1 KiB 914 MiB 16 TiB 2.14 1.01 99 up

thanks

Jake

On 20/07/2022 11:52, Jake Grimmett wrote:
Dear All,

We have just built a new cluster using Quincy 17.2.1

After copying ~25TB to the cluster (from a mimic cluster), we see 152 TB used, which is ~6x disparity.

Is this just a ceph accounting error, or is space being wasted?

[root@wilma-s1 ~]# du -sh /cephfs2/users
24T    /cephfs2/users

[root@wilma-s1 ~]# ls -lhd /cephfs2/users
drwxr-xr-x 240 root root 24T Jul 19 12:09 /cephfs2/users

[root@wilma-s1 ~]# df -h /cephfs2/users
Filesystem  Size  Used Avail Use% Mounted on
(SNIP):/    7.1P  152T  6.9P   3% /cephfs2

root@wilma-s1 ~]# ceph df
--- RAW STORAGE ---
CLASS     SIZE    AVAIL     USED  RAW USED  %RAW USED
hdd    7.0 PiB  6.9 PiB  151 TiB   151 TiB       2.10
ssd    2.7 TiB  2.7 TiB   11 GiB    11 GiB       0.38
TOTAL  7.0 PiB  6.9 PiB  151 TiB   151 TiB       2.10

--- POOLS ---
POOL             ID   PGS   STORED  OBJECTS     USED  %USED  MAX AVAIL
.mgr             21    32   90 MiB       24  270 MiB      0    2.2 PiB
mds_ssd          22    32  1.0 GiB   73.69k  3.0 GiB   0.11    881 GiB
ec82pool         23  4096   20 TiB    6.28M   25 TiB   0.38    5.2 PiB
primary_fs_data  24    32      0 B    1.45M      0 B      0    881 GiB


cephfs is using a 8+2 erasure coded data pool (hdd with NVMe db/wal), and a 3x replicated default data pool (primary_fs_data - NVMe)

bluestore_min_alloc_size_hdd is 4096
ceph pool set ec82pool compression_algorithm lz4
ceph osd pool set ec82pool compression_mode aggressive

many thanks for any help

Jake

--
Dr Jake Grimmett
Head Of Scientific Computing
MRC Laboratory of Molecular Biology
Francis Crick Avenue,
Cambridge CB2 0QH, UK.


_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux