I've been watching this thread with great interest because I've also noticed that my benchmarked max capacities seem abnormally high. Out of the box my 18T sas drives report ~2k as their iops capacity. "sdparm --get WCE /dev/sd*" shows that the write cache is on across all my spinners. For some reason, I thought that the OSD daemon would throw the bits to disable the cache. Forcing it disabled (sdparm --set WCE 0 /dev/sdg) brought the IOPS number down to the 900 range. Still high for a spinner but getting closer to reality. Now it matches the 10T sas drives in the same chassis (which also report the cache as on). The 10T drives do not appear to have any IOPS benchmark differences with the cache on or off in the drive. I'm now debating on if making it a persistent change in the drive firmware or via startup scripts is the best course of action. I know these drives will have a second life somewhere else and I don't want future me to have pain with "custom" firmware changes. If I make it part of the startup scripts then any new OSD drives will also get their cache disabled correctly. -paul ________________________________________ From: Sridhar Seshasayee <sseshasa@xxxxxxxxxx> Sent: Wednesday, September 7, 2022 8:58 AM To: Vladimir Brik Cc: David Orman; ceph-users Subject: Re: Wide variation in osd_mclock_max_capacity_iops_hdd Hi Vladimir, Yes, device caching is disabled. The ODSs in question use a > separate DB/WAL device on flash. Could that be the cause of > IOPS over-estimation? > 2000 IOPS definitely looks high for a HDD. The in-built osd bench tool writes to the backing device of the osd to determine the IOPS capacity. With DB/WAL configured, I wouldn't expect a wide variation. To confirm the IOPS measurement, you can run the following commands: $ ceph tell osd.N cache drop followed by, $ ceph tell osd.N bench 12288000 4096 4194304 100 Where N is the osd id. You can run the above sequence a few times to confirm if there are any wide variations and if so, it probably warrants more investigation into why that's the case. Also, may I know the following: - what kind of HDDs are you using in your environment? - Ceph version Is it better for me to manually set > osd_mclock_max_capacity_iops_hdd to a reasonable number or > will things perform OK as long as IOPS are over-estimated > consistently everywhere? > I think it's better to figure out why the IOPS are on the higher side first. -Sridhar <https://www.redhat.com> _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |