Hi Sridhar,
Thank you, that explains indeed a few things! :-)
But the underlying problem is that we see iowaits/slowdowns on the clients while rebalancing.
I added some nvme storage and am moving the data off the regular hdd. In earlier releases I could slow this down when there was load on the clients, but now I don’t know how to do this:
pgs: 2880930/5352414 objects misplaced (53.825%)
447 active+remapped+backfilling
290 active+clean
[root@ceph301 ~]# ceph config show osd.20 osd_mclock_profile
high_client_ops
So it is already at the high_client_ops setting.
Thanks again!!
Kenneth
On 5 Aug 2022, at 15:01, Sridhar Seshasayee <sseshasa@xxxxxxxxxx<mailto:sseshasa@xxxxxxxxxx>> wrote:
Hi Kenneth,
Any clues here? Thank you very much!
I assume you are running a Quincy version on your cluster.
If so, from the Quincy release, mClock is set to be the default queuing scheduler. It was
earlier set to the weighted priority queue scheduler (wpq). To maximize the effectiveness
of the mClock algorithm, certain Ceph options are locked to specific values and one of
them is osd_max_backfills which is locked to 1000. Another option which is locked to the
same value is osd_recovery_max_active.
With the mClock scheduler there are a couple of objectives from a user perspective,
1. Eliminate the need to set sleep throttles for various ops.
Therefore, the Ceph config sleep options for various ops have been disabled or in other
words locked to 0 in order to enable the mClock algorithm to work effectively.
2. Use mClock config profiles to hide the complexity of tuning various Ceph options
You can read about mClock profiles and about the config options that are locked here<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.ceph.com%2Fen%2Fquincy%2Frados%2Fconfiguration%2Fmclock-config-ref%2F&data=05%7C01%7CKenneth.Waegeman%40ugent.be%7C9e56dce355d5439fac1308da76e29926%7Cd7811cdeecef496c8f91a1786241b99c%7C1%7C0%7C637953012839327981%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=JDuvw8Bbkfhpy4yzCSaCb%2BawjaXef5yffV6LDKoHgW0%3D&reserved=0>.
Therefore, you can only limit the rate of backfills/recoveries using the mClock profiles.
The 'high_client_ops' profile will allow the lowest recovery rate due to the way the QoS
parameters are set up for the profile.
Hopefully, the above should answer some if not all your questions.
-Sridhar
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
