I found that I could get much more aggressive with the osd_mclock_cost_per_byte_usec_hdd values if my osd_mclock_max_capacity_iops_hdd numbers were set to sane values. If I used the (very wrong) default calculated values I would start getting slow ops. Setting all hdd in my environment to 300 seemed to calm that down. I'm running the mclock profile of high_client_ops at the moment. I do get better recovery on balanced but I saw client latency go up enough where it wasn't worth the trade-off in my environment at this time. I'm still slowly walking the cost number down looking for the sweet spot. It seems to be very sensitive. A change from .75 to .70 changed my IOPS/BW from 480/1.48GB/s to 515/1.57GB/s. It's easy to overshoot and hit some other bottleneck. --- Paul Mezzanini Platform Engineer III Research Computing Rochester Institute of Technology “End users is a description, not a goal.” ________________________________________ From: Nigel Williams <nigel.williams@xxxxxxxxxxx> Sent: Tuesday, May 23, 2023 12:19 AM To: David Orman Cc: ceph-users@xxxxxxx Subject: Re: Slow recovery on Quincy We're on 17.2.5 and had the default value (5.2), but changing it didn't seem to impact recovery speed: root@rdx-00:/# ceph config get osd osd_mclock_cost_per_byte_usec_hdd 5.200000 root@rdx-00:/# ceph config show osd.0 osd_op_queue mclock_scheduler root@rdx-00:/# ceph config set osd osd_mclock_cost_per_byte_usec_hdd 0.4 root@rdx-00:/# ceph config get osd osd_mclock_cost_per_byte_usec_hdd 0.400000 root@rdx-00:/# We have about 500TB of misplaced objects moving about so we are keen to speed it up; was there another parameter that needed to be changed too? thanks. On Mon, 22 May 2023 at 23:24, David Orman <ormandj@xxxxxxxxxxxx> wrote: > Someone who's got data regarding this should file a bug report, it sounds > like a quick fix for defaults if this holds true. > > On Sat, May 20, 2023, at 00:59, Hector Martin wrote: > > On 17/05/2023 03.07, 胡 玮文 wrote: > >> Hi Sake, > >> > >> We are experiencing the same. I set “osd_mclock_cost_per_byte_usec_hdd” > to 0.1 (default is 2.6) and get about 15 times backfill speed, without > significant affect client IO. This parameter seems calculated wrongly, from > the description 5e-3 should be a reasonable value for HDD (corresponding to > 200MB/s). I noticed this default is originally 5.2, then changed to 2.6 to > increase the recovery speed. So I suspect the original author just convert > the unit wrongly, he may want 5.2e-3 but wrote 5.2 in code. > >> > >> But all this may be not important in the next version. I see the > relevant code is rewritten, and this parameter is now removed. > >> > >> high_recovery_ops profile works very poorly for us. It increase the > average latency of client IO from 50ms to about 1s. > >> > >> Weiwen Hu > >> > > > > Thank you for this, that parameter indeed seems completely wrong > > (assuming it means what it says on the tin). After changing that my > > Quincy cluster is no recovering at a much more reasonable speed. > > > > - Hector > > _______________________________________________ > > ceph-users mailing list -- ceph-users@xxxxxxx > > To unsubscribe send an email to ceph-users-leave@xxxxxxx > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |