Quoting Robert Ruge (robert.ruge@xxxxxxxxxxxxx): > Ceph newbie question. > > I have a disparity between the free space that my cephfs file system > is showing and what ceph df is showing. As you can see below my > cephfs file system says there is 9.5TB free however ceph df says there > is 186TB which with replication size 3 should equate to 62TB free > space. I guess the basic question is how can I get cephfs to see and > use all of the available space? I recently changed my number of pg's > on the cephfs_data pool from 2048 to 4096 and this gave me another 8TB > so do I keep increasing the number of pg's or is there something else > that I am missing? I have only been running ceph for ~6 months so I'm > relatively new to it all and not being able to use all of the space is > just plain bugging me. My guess here is you have a lot of small files in your cephfs, is that right? Do you have HDD or SDD/NVMe? Mohamad Gebai gave a talk about this at Cephalocon 2019: https://static.sched.com/hosted_files/cephalocon2019/d2/cephalocon-2019-mohamad-gebai.pdf for the slides and the recording: https://www.youtube.com/watch?v=26FbUEbiUrw&list=PLrBUGiINAakNCnQUosh63LpHbf84vegNu&index=29&t=0s TL;DR: there is a bluestore_min_alloc_size_ssd which is 16K default for SSD and 64K default for HDD. With lots of small objects this might add up to *a lot* of overhead. You can change that to 4k: bluestore min alloc size ssd = 4096 bluestore min alloc size hdd = 4096 You will have to rebuild _all_ of your OSDs though. Here is another thread about this: http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-February/thread.html#24801 Gr. Stefan -- | BIT BV http://www.bit.nl/ Kamer van Koophandel 09090351 | GPG: 0xD14839C6 +31 318 648 688 / info@xxxxxx _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |