Quoting Robert Sander (r.sander@xxxxxxxxxxxxxxxxxxx): > Hi, > > we have a small cluster at a customer's site with three nodes and 4 SSD-OSDs > each. > Connected with 10G the system is supposed to perform well. > > rados bench shows ~450MB/s write and ~950MB/s read speeds with 4MB objects > but only 20MB/s write and 95MB/s read with 4KB objects. > > This is a little bit disappointing as the 4K performance is also seen in KVM > VMs using RBD. > > Is there anything we can do to improve performance with small objects / > block sizes? Josh gave a talk about this: https://static.sched.com/hosted_files/cephalocon2019/10/Optimizing%20Small%20Ceph%20Clusters.pdf TL;DR: - For small clusters use relatively more PGs than for large clusters - Make sure your cluster is well balanced, and this script might be useful: https://github.com/JoshSalomon/Cephalocon-2019/blob/master/pool_pgs_osd.sh Josh is also tuning the objecter_* attributes (if you have plenty of CPU/Memory): objecter_inflight_ops = 5120 objecter_inflight_op_bytes = 524288000 (512 * 1,024,000) ## You can multiply / divide both with the same factor Some more tuning tips in the presentation by Wido/Piotr that might be useful: https://static.sched.com/hosted_files/cephalocon2019/d6/ceph%20on%20nvme%20barcelona%202019.pdf Gr. Stefan -- | BIT BV http://www.bit.nl/ Kamer van Koophandel 09090351 | GPG: 0xD14839C6 +31 318 648 688 / info@xxxxxx _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |