Hi, There are many talks and presentations out there about Ceph's performance. Ceph is great when it comes to parallel I/O, large queue depths and many applications sending I/O towards Ceph. One thing where Ceph isn't the fastest are 4k blocks written at Queue Depth 1. Some applications benefit very much from high performance/low latency I/O at qd=1, for example Single Threaded applications which are writing small files inside a VM running on RBD. With some tuning you can get to a ~700us latency for a 4k write with qd=1 (Replication, size=3) I benchmark this using fio: $ fio --ioengine=librbd --bs=4k --iodepth=1 --direct=1 .. .. .. .. 700us latency means the result will be about ~1500 IOps (1000 / 0.7) When comparing this to let's say a BSD machine running ZFS that's on the low side. With ZFS+NVMe you'll be able to reach about somewhere between 7.000 and 10.000 IOps, the latency is simply much lower. My benchmarking / test setup for this: - Ceph Nautilus/Octopus (doesn't make a big difference) - 3x SuperMicro 1U with: - AMD Epyc 7302P 16-core CPU - 128GB DDR4 - 10x Samsung PM983 3,84TB - 10Gbit Base-T networking Things to configure/tune: - C-State pinning to 1 - CPU governer to performance - Turn off all logging in Ceph (debug_osd, debug_ms, debug_bluestore=0) Higher clock speeds (New AMD Epyc coming in March!) help to reduce the latency and going towards 25Gbit/100Gbit might help as well. These are however only very small increments and might help to reduce the latency by another 15% or so. It doesn't bring us anywhere near the 10k IOps other applications can do. And I totally understand that replication over a TCP/IP network takes time and thus increases latency. The Crimson project [0] is aiming to lower the latency with many things like DPDK and SPDK, but this is far from finished and production ready. In the meantime, am I overseeing some things here? Can we reduce the latency further of the current OSDs? Reaching a ~500us latency would already be great! Thanks, Wido [0]: https://docs.ceph.com/en/latest/dev/crimson/crimson/ _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |