Andras, Unfortunately your attachment didn't come through the list. (It might work if you embed it inline? Not sure.) I don't know if anybody's looked too hard at this before, and without the image I don't know exactly what metric you're using to say something's 320KB in size. Can you explain more? It might help if you dump the objecter_requests from the MDS and share those — it'll display what objects are being written to with what sizes. -Greg On Wed, Dec 8, 2021 at 9:00 AM Andras Sali <sali.andrew@xxxxxxxxx> wrote: > > Hi All, > > We have been observing that if we let our MDS run for some time, the > bandwidth usage of the disks in the metadata pool starts increasing > significantly (whilst IOPS is about constant), even though the number of > clients, the workloads or anything else doesn't change. > > However, after restarting the MDS, the issue goes away for some time and > the same workloads require 1/10th of the metadata disk bandwidth whilst > doing the same IOPS. > > We run our CephFS cluster in a cloud environment where the disk throughput > / bandwidth capacity is quite expensive to increase and we are hitting > bandwidth / throughput limits, even though we still have a lot of IOPS > capacity left. > > We suspect that somehow the journaling of the MDS becomes more extensive > (i.e. larger journal updates for each operation), but we couldn't really > pin down which parameter might affect this. > > I attach a plot of how the Bytes / Operation (throughput in MBps / IOPS) > evolves over time, when we restart the MDS, it drops to around 32kb (even > though the min block size for the metadata pool OSDs is 4kb in our > settings) and then increases over time to around 300kb. > > Any ideas on how to "fix" this and have a significantly lower bandwidth > usage would be really-really appreciated! > > Thank you and kind regards, > > Andras > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |