Re: Reef (18.2): Some PG not scrubbed/deep scrubbed for 1 month

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Considering https://github.com/ceph/ceph/blob/f6edcef6efe209e8947887752bd2b833d0ca13b7/src/osd/OSD.cc#L10086, the OSD:

- always sets and updates its per osd osd_mclock_max_capacity_iops_{hdd,ssd} when the benchmark occurs and its measured iops is below or equal to osd_mclock_iops_capacity_threshold_{hdd,ssd}
but
- doesn't remove osd_mclock_max_capacity_iops_{hdd,ssd} when the measured iops exceeds osd_mclock_iops_capacity_threshold_{hdd,ssd} (500 for HDD and 80.000 for SSD) and the current value for osd_mclock_max_capacity_iops_{hdd,ssd} is set below its default (315 for HDD and 21500 for SSD)

Thus per osd osd_mclock_max_capacity_iops_hdd sometimes being set as low as 0.145327 (as per Michel's post) and never being updated afterwards leading to performance issues.
The idea of a minimum threshold below which osd_mclock_iops_capacity_threshold_{hdd,ssd} should not be set seems relevant.

CC'ing Sridhar to have his thoughts.

Cheers,
Frédéric.

----- Le 22 Mar 24, à 19:37, Kai Stian Olstad ceph+list@xxxxxxxxxx a écrit :

> On Fri, Mar 22, 2024 at 06:51:44PM +0100, Frédéric Nass wrote:
>>
>>> The OSD run bench and update osd_mclock_max_capacity_iops_{hdd,ssd} every time
>>> the OSD is started.
>>> If you check the OSD log you'll see it does the bench.
>
>>Are you sure about the update on every start? Does the update happen only if the
>>benchmark result is < 500 iops?
>
>>Looks like the OSD does not remove any set configuration when the benchmark
>>result is > 500 iops. Otherwise, the extremely low value that Michel reported
>>earlier (less than 1 iops) would have been updated over time.
>>I guess.
> 
> I'm not completely sure, it's a couple a month since I used mclock, have switch
> back to wpq because of a nasty bug in mclock that can freeze cluster I/O.
> 
> It could be because I was testing osd_mclock_force_run_benchmark_on_init.
> The OSD had DB on SSD and data on HDD, so the measured to about 1700 IOPS and
> was ignored because of the 500 limit.
> So only the SSD got the osd_mclock_max_capacity_iops_ssd set.
> 
> --
> Kai Stian Olstad
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux