Re: Slow recovery on Quincy

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



We're on 17.2.5 and had the default value (5.2), but changing it didn't
seem to impact recovery speed:

root@rdx-00:/# ceph config get osd osd_mclock_cost_per_byte_usec_hdd
5.200000
root@rdx-00:/# ceph config show osd.0 osd_op_queue
mclock_scheduler
root@rdx-00:/# ceph config set osd osd_mclock_cost_per_byte_usec_hdd 0.4
root@rdx-00:/# ceph config get osd osd_mclock_cost_per_byte_usec_hdd
0.400000
root@rdx-00:/#

We have about 500TB of misplaced objects moving about so we are keen to
speed it up; was there another parameter that needed to be changed too?
thanks.


On Mon, 22 May 2023 at 23:24, David Orman <ormandj@xxxxxxxxxxxx> wrote:

> Someone who's got data regarding this should file a bug report, it sounds
> like a quick fix for defaults if this holds true.
>
> On Sat, May 20, 2023, at 00:59, Hector Martin wrote:
> > On 17/05/2023 03.07, 胡 玮文 wrote:
> >> Hi Sake,
> >>
> >> We are experiencing the same. I set “osd_mclock_cost_per_byte_usec_hdd”
> to 0.1 (default is 2.6) and get about 15 times backfill speed, without
> significant affect client IO. This parameter seems calculated wrongly, from
> the description 5e-3 should be a reasonable value for HDD (corresponding to
> 200MB/s). I noticed this default is originally 5.2, then changed to 2.6 to
> increase the recovery speed. So I suspect the original author just convert
> the unit wrongly, he may want 5.2e-3 but wrote 5.2 in code.
> >>
> >> But all this may be not important in the next version. I see the
> relevant code is rewritten, and this parameter is now removed.
> >>
> >> high_recovery_ops profile works very poorly for us. It increase the
> average latency of client IO from 50ms to about 1s.
> >>
> >> Weiwen Hu
> >>
> >
> > Thank you for this, that parameter indeed seems completely wrong
> > (assuming it means what it says on the tin). After changing that my
> > Quincy cluster is no recovering at a much more reasonable speed.
> >
> > - Hector
> > _______________________________________________
> > ceph-users mailing list -- ceph-users@xxxxxxx
> > To unsubscribe send an email to ceph-users-leave@xxxxxxx
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux