20 MB/s at 4K blocks is ~5000 iops, that's 1250 IOPS per SSD (assuming replica 3).
What we usually check in scenarios like these:
* SSD model? Lots of cheap SSDs simply can't handle more than that
* Get some proper statistics such as OSD latencies, disk IO utilization, etc. A benchmark without detailed performance data doesn't really help to debug such a problem
Paul
--
Paul Emmerich
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90
Paul Emmerich
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90
On Fri, May 24, 2019 at 1:23 PM Robert Sander <r.sander@xxxxxxxxxxxxxxxxxxx> wrote:
Hi,
we have a small cluster at a customer's site with three nodes and 4
SSD-OSDs each.
Connected with 10G the system is supposed to perform well.
rados bench shows ~450MB/s write and ~950MB/s read speeds with 4MB
objects but only 20MB/s write and 95MB/s read with 4KB objects.
This is a little bit disappointing as the 4K performance is also seen in
KVM VMs using RBD.
Is there anything we can do to improve performance with small objects /
block sizes?
Jumbo frames have already been enabled.
4MB objects write:
Total time run: 30.218930
Total writes made: 3391
Write size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 448.858
Stddev Bandwidth: 63.5044
Max bandwidth (MB/sec): 552
Min bandwidth (MB/sec): 320
Average IOPS: 112
Stddev IOPS: 15
Max IOPS: 138
Min IOPS: 80
Average Latency(s): 0.142475
Stddev Latency(s): 0.0990132
Max latency(s): 0.814715
Min latency(s): 0.0308732
4MB objects rand read:
Total time run: 30.169312
Total reads made: 7223
Read size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 957.662
Average IOPS: 239
Stddev IOPS: 23
Max IOPS: 272
Min IOPS: 175
Average Latency(s): 0.0653696
Max latency(s): 0.517275
Min latency(s): 0.00201978
4K objects write:
Total time run: 30.002628
Total writes made: 165404
Write size: 4096
Object size: 4096
Bandwidth (MB/sec): 21.5351
Stddev Bandwidth: 2.0575
Max bandwidth (MB/sec): 22.4727
Min bandwidth (MB/sec): 11.0508
Average IOPS: 5512
Stddev IOPS: 526
Max IOPS: 5753
Min IOPS: 2829
Average Latency(s): 0.00290095
Stddev Latency(s): 0.0015036
Max latency(s): 0.0778454
Min latency(s): 0.00174262
4K objects read:
Total time run: 30.000538
Total reads made: 1064610
Read size: 4096
Object size: 4096
Bandwidth (MB/sec): 138.619
Average IOPS: 35486
Stddev IOPS: 3776
Max IOPS: 42208
Min IOPS: 26264
Average Latency(s): 0.000443905
Max latency(s): 0.0123462
Min latency(s): 0.000123081
Regards
--
Robert Sander
Heinlein Support GmbH
Linux: Akademie - Support - Hosting
http://www.heinlein-support.de
Tel: 030-405051-43
Fax: 030-405051-19
Zwangsangaben lt. §35a GmbHG:
HRB 93818 B / Amtsgericht Berlin-Charlottenburg,
Geschäftsführer: Peer Heinlein -- Sitz: Berlin
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
