Yep sure, will try and present some figures at tomorrow?s meeting again. From: Samuel Just [mailto:sjust@xxxxxxxxxx] Sent: 21 February 2017 18:14 To: Nick Fisk <nick at fisk.me.uk> Cc: ceph-users at lists.ceph.com Subject: Re: osd_snap_trim_sleep keeps locks PG during sleep? Ok, I've added explicit support for osd_snap_trim_sleep (same param, new non-blocking implementation) to that branch. Care to take it for a whirl? -Sam On Thu, Feb 9, 2017 at 11:36 AM, Nick Fisk <nick at fisk.me.uk <mailto:nick at fisk.me.uk> > wrote: Building now From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx <mailto:ceph-users-bounces at lists.ceph.com> ] On Behalf Of Samuel Just Sent: 09 February 2017 19:22 To: Nick Fisk <nick at fisk.me.uk <mailto:nick at fisk.me.uk> > Cc: ceph-users at lists.ceph.com <mailto:ceph-users at lists.ceph.com> Subject: Re: osd_snap_trim_sleep keeps locks PG during sleep? Ok, https://github.com/athanatos/ceph/tree/wip-snap-trim-sleep <http://xo4t.mj.am/lnk/AEMAHoCuLsEAAAAAAAAAAFklQUAAADNJBWwAAAAAAACRXwBYnMSxtQOC2XS3Szqb2auk-ebWnwAAlBI/1/h2R0YQ3R0gPNSCn9TMuW8Q/aHR0cHM6Ly9naXRodWIuY29tL2F0aGFuYXRvcy9jZXBoL3RyZWUvd2lwLXNuYXAtdHJpbS1zbGVlcA> (based on master) passed a rados suite. It adds a configurable limit to the number of pgs which can be trimming on any OSD (default: 2). PGs trimming will be in snaptrim state, PGs waiting to trim will be in snaptrim_wait state. I suspect this'll be adequate to throttle the amount of trimming. If not, I can try to add an explicit limit to the rate at which the work items trickle into the queue. Can someone test this branch? Tester beware: this has not merged into master yet and should only be run on a disposable cluster. -Sam On Tue, Feb 7, 2017 at 1:16 PM, Nick Fisk <nick at fisk.me.uk <mailto:nick at fisk.me.uk> > wrote: Yeah it?s probably just the fact that they have more PG?s so they will hold more data and thus serve more IO. As they have a fixed IO limit, they will always hit the limit first and become the bottleneck. The main problem with reducing the filestore queue is that I believe you will start to lose the benefit of having IO?s queued up on the disk, so that the scheduler can re-arrange them to action them in the most efficient manor as the disk head moves across the platters. You might possibly see up to a 20% hit on performance, in exchange for more consistent client latency. From: Steve Taylor [mailto:steve.taylor@xxxxxxxxxxxxxxxx <mailto:steve.taylor at storagecraft.com> ] Sent: 07 February 2017 20:35 To: nick at fisk.me.uk <mailto:nick at fisk.me.uk> ; ceph-users at lists.ceph.com Subject: RE: Re: osd_snap_trim_sleep keeps locks PG during sleep? Thanks, Nick. One other data point that has come up is that nearly all of the blocked requests that are waiting on subops are waiting for OSDs with more PGs than the others. My test cluster has 184 OSDs, 177 of which are 3TB, with 7 4TB OSDs. The cluster is well balanced based on OSD capacity, so those 7 OSDs individually have 33% more PGs than the others and are causing almost all of the blocked requests. It appears that maps updates are generally not blocking long enough to show up as blocked requests. I set the reweight on those 7 OSDs to 0.75 and things are backfilling now. I?ll test some more when the PG counts per OSD are more balanced and see what I get. I?ll also play with the filestore queue. I was telling some of my colleagues yesterday that this looked likely to be related to buffer bloat somewhere. I appreciate the suggestion. _____ <http://xo4t.mj.am/lnk/AEMAHoCuLsEAAAAAAAAAAFklQUAAADNJBWwAAAAAAACRXwBYnMSxtQOC2XS3Szqb2auk-ebWnwAAlBI/2/2T4Xj-_wncGT6Y6LyBEKdw/aHR0cDovL3hvNHQubWouYW0vbG5rL0FEc0FBR1ZFeFk0QUFBQUFBQUFBQUV0ckRjc0FBRE5KQld3QUFBQUFBQUNSWHdCWW1qa3VqWE1Hc2Z2MFFJMklrZHpkTVBIYk93QUFsQkkvMS9HclNQRjU2RnY2VXVUc1JUejFUbnJRL2FIUjBjSE02THk5emRHOXlZV2RsWTNKaFpuUXVZMjl0> Steve Taylor | Senior Software Engineer | <http://xo4t.mj.am/lnk/AEMAHoCuLsEAAAAAAAAAAFklQUAAADNJBWwAAAAAAACRXwBYnMSxtQOC2XS3Szqb2auk-ebWnwAAlBI/3/TFexBfD-LHnCcPturKjU5Q/aHR0cDovL3hvNHQubWouYW0vbG5rL0FEc0FBR1ZFeFk0QUFBQUFBQUFBQUV0ckRjc0FBRE5KQld3QUFBQUFBQUNSWHdCWW1qa3VqWE1Hc2Z2MFFJMklrZHpkTVBIYk93QUFsQkkvMi9IbGVSZWkzWVdEZGljbUN1RG9XeXRBL2FIUjBjSE02THk5emRHOXlZV2RsWTNKaFpuUXVZMjl0> StorageCraft Technology Corporation 380 Data Drive Suite 300 | Draper | Utah | 84020 Office: 801.871.2799 <tel:(801)%20871-2799> | _____ If you are not the intended recipient of this message or received it erroneously, please notify the sender and delete it, together with any attachments, and be advised that any dissemination or copying of this message is prohibited. _____ From: Nick Fisk [mailto:nick@xxxxxxxxxx <mailto:nick at fisk.me.uk> ] Sent: Tuesday, February 7, 2017 10:25 AM To: Steve Taylor <steve.taylor at storagecraft.com <mailto:steve.taylor at storagecraft.com> >; ceph-users at lists.ceph.com <mailto:ceph-users at lists.ceph.com> Subject: RE: Re: osd_snap_trim_sleep keeps locks PG during sleep? Hi Steve, >From what I understand, the issue is not with the queueing in Ceph, which is correctly moving client IO to the front of the queue. The problem lies below what Ceph controls, ie the scheduler and disk layer in Linux. Once the IO?s leave Ceph it?s a bit of a free for all and the client IO?s tend to get lost in large disk queues surrounded by all the snap trim IO?s. The workaround Sam is working on will limit the amount of snap trims that are allowed to run, which I believe will have a similar effect to the sleep parameters in pre-jewel clusters, but without pausing the whole IO thread. Ultimately the solution requires Ceph to be able to control the queuing of IO?s at the lower levels of the kernel. Whether this is via some sort of tagging per IO (currently CFQ is only per thread/process) or some other method, I don?t know. I was speaking to Sage and he thinks the easiest method might be to shrink the filestore queue so that you don?t get buffer bloat at the disk level. You should be able to test this out pretty easily now by changing the parameter, probably around a queue of 5-10 would be about right for spinning disks. It?s a trade off of peak throughput vs queue latency though. Nick From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On Behalf Of Steve Taylor Sent: 07 February 2017 17:01 To: ceph-users at lists.ceph.com <mailto:ceph-users at lists.ceph.com> Subject: Re: osd_snap_trim_sleep keeps locks PG during sleep? As I look at more of these stuck ops, it looks like more of them are actually waiting on subops than on osdmap updates, so maybe there is still some headway to be made with the weighted priority queue settings. I do see OSDs waiting for map updates all the time, but they aren?t blocking things as much as the subops are. Thoughts? _____ Steve Taylor | Senior Software Engineer | StorageCraft Technology Corporation 380 Data Drive Suite 300 | Draper | Utah | 84020 Office: 801.871.2799 <tel:(801)%20871-2799> | _____ If you are not the intended recipient of this message or received it erroneously, please notify the sender and delete it, together with any attachments, and be advised that any dissemination or copying of this message is prohibited. _____ From: Steve Taylor Sent: Tuesday, February 7, 2017 9:13 AM To: 'ceph-users at lists.ceph.com <mailto:ceph-users at lists.ceph.com> ' <ceph-users at lists.ceph.com <mailto:ceph-users at lists.ceph.com> > Subject: Re: osd_snap_trim_sleep keeps locks PG during sleep? Sorry, I lost the previous thread on this. I apologize for the resulting incomplete reply. The issue that we?re having with Jewel, as David Turner mentioned, is that we can?t seem to throttle snap trimming sufficiently to prevent it from blocking I/O requests. On further investigation, I encountered osd_op_pq_max_tokens_per_priority, which should be able to be used in conjunction with ?osd_op_queue = wpq? to govern the availability of queue positions for various operations using costs if I understand correctly. I?m testing with RBDs using 4MB objects, so in order to leave plenty of room in the weighted priority queue for client I/O, I set osd_op_pq_max_tokens_per_priority to 64MB and osd_snap_trim_cost to 32MB+1. I figured this should essentially reserve 32MB in the queue for client I/O operations, which are prioritized higher and therefore shouldn?t get blocked. I still see blocked I/O requests, and when I dump in-flight ops, they show ?op must wait for map.? I assume this means that what?s blocking the I/O requests at this point is all of the osdmap updates caused by snap trimming, and not the actual snap trimming itself starving the ops of op threads. Hammer is able to mitigate this with osd_snap_trim_sleep by directly throttling snap trimming and therefore causing less frequent osdmap updates, but there doesn?t seem to be a good way to accomplish the same thing with Jewel. First of all, am I understanding these settings correctly? If so, are there other settings that could potentially help here, or do we just need something like Sam already mentioned that can sort of reserve threads for client I/O requests? Even then it seems like we might have issues if we can?t also throttle snap trimming. We delete a LOT of RBD snapshots on a daily basis, which we recognize is an extreme use case. Just wondering if there?s something else to try or if we need to start working toward implementing something new ourselves to handle our use case better. _______________________________________________ ceph-users mailing list ceph-users at lists.ceph.com <mailto:ceph-users at lists.ceph.com> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com <http://xo4t.mj.am/lnk/AEMAHoCuLsEAAAAAAAAAAFklQUAAADNJBWwAAAAAAACRXwBYnMSxtQOC2XS3Szqb2auk-ebWnwAAlBI/6/Z086sItvmjXlhBlubwoAQQ/aHR0cDovL2xpc3RzLmNlcGguY29tL2xpc3RpbmZvLmNnaS9jZXBoLXVzZXJzLWNlcGguY29t> -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20170221/23e045fd/attachment.htm> -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 2679 bytes Desc: not available URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20170221/23e045fd/attachment.jpeg> -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 332 bytes Desc: not available URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20170221/23e045fd/attachment-0001.jpeg> -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 332 bytes Desc: not available URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20170221/23e045fd/attachment-0002.jpeg>
 |