We’re running Quincy 17.2.7 here,
And we see the iops benchmark performed on osd start:
2024-09-20T09:57:26.265+1000 7facbdc64540 1 osd.196 2879010 maybe_override_max_osd_capacity_for_qos osd bench result - bandwidth (MiB/sec): 4.369 iops: 1118.445 elapsed_sec: 2.682
2024-09-20T09:57:26.265+1000 7facbdc64540 0 log_channel(cluster) log [WRN] : OSD bench result of 1118.445008 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.196. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio) and then override osd_mclock_max_capacity_iops_[hdd|ssd].
Has the accuracy of this benchmark been improved in newer releases?
From: Andrej Filipcic <andrej.filipcic@xxxxxx>
Date: Thursday, 19 September 2024 at 9:20 pm
To: ceph-users@xxxxxxx <ceph-users@xxxxxxx>
Subject: [EXT] Re: mclock scheduler kills clients IOs
External email: Please exercise caution
Hi,
the problem comes from older ceph releases. In our case, hdd iops were
benchmarked in the range of 250 to 4000, which clearly makes no sense.
At osd startup, the benchmark is skipped if that value is already in
ceph config, so these initial benchmark values were never changed. To
reset them, all osd.N osd_mclock_max_capacity_iops_hdd values should be
removed and osds restarted. There is a safety mechanism
(osd_mclock_iops_capacity_threshold_hdd) which prevents for the values
to be overestimated.
Best,
Andrej
On 19. 09. 24 11:33, Daniel Schreiber wrote:
> Hi Denis,
>
> we observed the same behaviour here. The cause was that the number of
> iops discovered at OSD startup was way too high. In our setup the
> rocksdb is on flash.
>
> When I set osd_mclock_max_capacity_iops_hdd to a value that the HDDs
> could handle, the situation was resolved, clients got ther fair share
> of IO.
>
> Hope this helps,
>
> Daniel
>
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
--
_____________________________________________________________
prof. dr. Andrej Filipcic, E-mail: Andrej.Filipcic@xxxxxx
Department of Experimental High Energy Physics - F9
Jozef Stefan Institute, Jamova 39, P.o.Box 3000
SI-1001 Ljubljana, Slovenia
Tel.: +386-1-477-3674 Fax: +386-1-477-3166
-------------------------------------------------------------
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
