Re: Slow recovery and inaccurate recovery figures since Quincy upgrade

[Iain Stott <Iain.Stott@xxxxxxxxxxxxxxx>]

Hi Sridhar,

Thanks for the response, I have added the output you requested below, I have attached the output from the last command in a file as it was rather long. We did try to set high_recovery_ops but it didn't seem to have any visible effect.

root@gb4-li-cephgw-001 ~ # ceph versions
{
    "mon": {
        "ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)": 3
    },
    "mgr": {
        "ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)": 3
    },
    "osd": {
        "ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)": 72
    },
    "mds": {},
    "rgw": {
        "ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)": 3
    },
    "overall": {
        "ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)": 81
    }
}

root@gb4-li-cephgw-001 ~ # ceph config get osd.0 osd_op_queue
mclock_scheduler

root@gb4-li-cephgw-001 ~ # ceph config show osd.0 | grep osd_max_backfills
osd_max_backfills                                3                                                                                                                                    override  (mon[3]),default[10]

root@gb4-li-cephgw-001 ~ # ceph config show osd.0 | grep osd_recovery_max_active
osd_recovery_max_active                          9                                                                                                                                    override  (mon[9]),default[0]
osd_recovery_max_active_hdd                      10                                                                                                                                   default
osd_recovery_max_active_ssd                      20                                                                                                                                   default

Thanks
Iain
________________________________
From: Sridhar Seshasayee <sseshasa@xxxxxxxxxx>
Sent: 03 October 2023 09:07
To: Iain Stott <Iain.Stott@xxxxxxxxxxxxxxx>
Cc: ceph-users@xxxxxxx <ceph-users@xxxxxxx>; dl-osadmins <dl-osadmins@xxxxxxxxxxxxxxx>
Subject: Re:  Slow recovery and inaccurate recovery figures since Quincy upgrade



CAUTION: This email originates from outside THG

________________________________
Hello Iain,


Does anyone have any ideas of what could be the issue here or anywhere we can check what is going on??


You could be hitting the slow backfill/recovery issue with mclock_scheduler.
Could you please provide the output of the following commands?

1. ceph versions
2. ceph config get osd.<id> osd_op_queue
3. ceph config show osd.<id> | grep osd_max_backfills
4. ceph config show osd.<id> | grep osd_recovery_max_active
5. ceph config show-with-defaults osd.<id> | grep osd_mclock where 'id' can be any valid osd id

With the mclock_scheduler enabled and with 17.2.5, it is not possible to override
recovery settings like 'osd_max_backfills' and other recovery related config options.

To improve the recovery rate, you can temporarily switch the mClock profile to 'high_recovery_ops'
on all the OSDs by issuing:

ceph config set osd osd_mclock_profile high_recovery_ops

During recovery with this profile, you may notice a dip in the client ops performance which is expected.
Once the recovery is done, you can switch the mClock profile back to the default 'high_client_ops' profile.

Please note that the upcoming Quincy release will address the slow backfill issues along with other
usability improvements.

-Sridhar

_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx