Hi Martin, This is a bit of generic recommendation, but I would go down the path of reducing complexity, i.e. first test the drive locally on the OSD node and see if there's anything going on with e.g. drive firmware, cables, HBA, power. Then do fio from another host, and this would incorporate networking. If those look fine, I would do something crazy with Ceph, such as a huge number of PGs, or failure domain of OSD, and just deploy a handful of OSDs to see if you can bring the problem out in the open. I would use a default setup, with no tweaks to scheduler etc. Hopefully, you'll get some error messages in the logs - ceph logs, syslog, dmesg. Maybe at that point it will become more obvious, or at least some messages will come through that will make sense (to you or someone else on the list). In other words, it seems you have to break this a bit more to get proper diagnostics. I know you guys have played with Ceph before, and can do the math of what the IOPS values should be - three clusters all seeing the same problem would most likely indicate a non-default configuration value that is not correct. -- Alex Gorbachev ISS On Mon, Nov 25, 2024 at 9:34 PM Martin Gerhard Loschwitz < martin.loschwitz@xxxxxxxxxxxxx> wrote: > Folks, > > I am getting somewhat desperate debugging multiple setups here within the > same environment. Three clusters, two SSD-only, one HDD-only, and what they > all have in common is abysmal 4k IOPS performance when measuring with > „rados bench“. Abysmal means: In an All-SSD cluster I will get roughly 400 > IOPS over more than 250 devices. I’ve know SAS-SSDs are not ideal, but 250 > looks a bit on the low side of things to me. > > In the second cluster, also All-SSD based, I get roughly 120 4k IOPS. And > the HDD-only cluster delivers 60 4k IOPS. The latter both with > substantially fewer devices, granted. But even with 20 HDDs, 68 4k IOPS > seems like a very bad value to me. > > I’ve tried to rule out everything I know of: BIOS misconfigurations, HBA > problems, networking trouble (I am seeing comparably bad values with a > size=1 pool) and so further and so on. But to no avail. Has anybody dealt > with something similar on Dell hardware or in general? What could cause > such extremely bad benchmark results? > > I measure with rados bench and qd=1 at 4k block size. „ceph tell osd > bench“ with 4k blocks yields 30k+ IOPS for every single device in the big > cluster, and all that leads to is 400 IOPS in total when writing to it? > Even with no replication in place? That looks a bit off, doesn't it? Any > help will be greatly appreciated, thank you very much in advance. Even a > pointer to the right direction would be held in high esteem right now. > Thank you very much in advance! > > Best regards > Martin > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |