Hi Eugen, On Mon, 2020-08-24 at 14:26 +0000, Eugen Block wrote: > Hi, > > there have been several threads about this topic [1], most likely > it's > the metadata operation during the cleanup that saturates your disks. > > The recommended settings seem to be: > > [osd] > osd op queue = wpq > osd op queue cut off = high Yeah, I've stumbled upon those settings recently. However, it seems to be the default nowadays... root@cephosd01:~# ceph config get mds.cephosd01 osd_op_queue wpq root@0cephosd01:~# ceph config get mds.cephosd01 osd_op_queue_cut_off high root@cephosd01:~# I do appreciate your input anyway. > This helped us a lot, the number of slow requests has decreased > significantly. > > Regards, > Eugen > > > [1] > https://lists.ceph.io/hyperkitty/list/ceph-users@xxxxxxx/thread/MK672ROJSW3X56PC2KWOK2GX7ENQP2LS/#FF3FMP5EEMOBCXAYB4ZVFIAAN6U4IRS3 > > > Zitat von Momčilo Medić <fedorauser@xxxxxxxxxxxxxxxxx>: > > > Hi friends, > > > > Since deployment of our Ceph cluster we've been plagued by slow > > metadata error. > > Namely, cluster goes into HEALTH_WARN with a message similar to > > this > > one: > > > > 2 MDSs report slow metadata IOs > > 1 MDSs report slow requests > > 1 slow ops, oldest one blocked for 32 sec, daemons [osd.22,osd.4] > > have > > slow ops. > > > > Here is a brief overview of our setup: > > - 7 OSD nodes with 6 OSD drives each > > - three of those are also monitors, managers and MDS > > - there is a single Ceph client (at the moment) > > - there is only CephFS being used (at the moment) > > - metadata for CephFS is on HDD (was on HDD, but we moved it as > > suggested - no improvement) > > > > Our expectation is that this is not a RAM issue as we have 64GiB of > > memory and is never fully utilized. > > > > It might be a CPU problem, as issue happens mostly during high > > loads > > (load of ~12 on a 8-core Intel Xeon Bronze 3106). > > However, the load is present on all OSD nodes, not just MDS ones. > > > > Cluster is used for (mostly nightly) backups and has no critical > > performance requirement. > > Interestingly, significant load across all nodes appears when > > running > > cleanup of outdated backups. > > This boils down to mostly truncating files and some removal, but it > > is > > usually small number of large files. > > > > Bellow you can find an example of "dump_ops_in_flight" output > > during > > the problem (which you may find useful - I couldn't make sense out > > of > > it). > > > > Should we invest into more powerfull CPU hardware (or should we > > move > > MDS roles to more powerful nodes)? > > > > Please let me know if I can share any more information to help > > resolve > > this thing. > > > > Thanks in advance! > > > > Kind regards, > > Momo. > > > > === > > > > { > > "ops": [ > > { > > "description": > > "client_request(client.22661659:706483006 > > create #0x10000002742/a-random-file 2020-08-23T23:09:33.919740+0200 > > caller_uid=117, caller_gid=121{})", > > "initiated_at": "2020-08-23T23:09:33.926509+0200", > > "age": 30.193027896, > > "duration": 30.193083934000001, > > "type_data": { > > "flag_point": "failed to authpin, subtree is being > > exported", > > "reqid": "client.22661659:706483006", > > "op_type": "client_request", > > "client_info": { > > "client": "client.22661659", > > "tid": 706483006 > > }, > > "events": [ > > { > > "time": "2020-08-23T23:09:33.926509+0200", > > "event": "initiated" > > }, > > { > > "time": "2020-08-23T23:09:33.926510+0200", > > "event": "throttled" > > }, > > { > > "time": "2020-08-23T23:09:33.926509+0200", > > "event": "header_read" > > }, > > { > > "time": "2020-08-23T23:09:33.926516+0200", > > "event": "all_read" > > }, > > { > > "time": "2020-08-23T23:09:33.926540+0200", > > "event": "dispatched" > > }, > > { > > "time": "2020-08-23T23:09:33.926595+0200", > > "event": "failed to authpin, subtree is > > being > > exported" > > } > > ] > > } > > } > > ], > > "num_ops": 1 > > } > > _______________________________________________ > > ceph-users mailing list -- ceph-users@xxxxxxx > > To unsubscribe send an email to ceph-users-leave@xxxxxxx > > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |