Yes. Rotational drives can generally do 100-200IOPS (some outliers, of course). Do you have all forms of caching disabled on your storage controllers/disks? On Tue, Sep 6, 2022 at 11:32 AM Vladimir Brik < vladimir.brik@xxxxxxxxxxxxxxxx> wrote: > Setting osd_mclock_force_run_benchmark_on_init to true and > restarting OSDs fixed the problem of high variability. > > However, osd_mclock_max_capacity_iops_hdd is now over 2000. > That's way too much for spinning disks, isn't it? > > Vlad > > > > On 9/1/22 03:54, Sridhar Seshasayee wrote: > > Hello Vladimir, > > > > I have noticed that our osd_mclock_max_capacity_iops_hdd > > varies widely for OSDs on identical drives in identical > > machines (from ~600 to ~2800). > > > > The IOPS shouldn't vary widely if the drives are of similar > > age and running > > the same workloads. The osd_mclock_max_capacity_iops_hdd is > > determined > > using Ceph's osd bench during osd boot-up. From our testing > > on HDDs, the > > tool shows fairly consistent numbers (+/- a few 10s of IOPS) > > for a given HDD. > > For more details please see: > > > https://docs.ceph.com/en/latest/rados/configuration/mclock-config-ref/#osd-capacity-determination-automated > < > https://docs.ceph.com/en/latest/rados/configuration/mclock-config-ref/#osd-capacity-determination-automated > > > > > > In your case, it would be good to take another look at the > > subset of HDDs > > showing degraded/lower IOPS performance and check if there > > are any > > underlying issues. You could run the osd bench against the > > affected osds as > > described in the link below or your preferred tool to get a > > better understanding: > > > > > https://docs.ceph.com/en/latest/rados/configuration/mclock-config-ref/#benchmarking-test-steps-using-osd-bench > < > https://docs.ceph.com/en/latest/rados/configuration/mclock-config-ref/#benchmarking-test-steps-using-osd-bench > > > > > > Is such great variation a problem? What effect on > > performance does this have? > > > > The mClock profiles use the IOPS capacity of each osd to > > allocate IOPS > > resources to ops like client, recovery, scrubs and so on. > > Therefore, a lower IOPS > > capacity will have an impact on these ops and therefore it > > would make sense to > > check the health of the HDDs that are showing lower than > > expected IOPS numbers. > > -Sridhar > > > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |