Hi there, i'm new to ceph and just got my first cluster running. Now i'd like to know if the performance we get is expectable. Is there a website with benchmark results somewhere where i could have a look to compare with our HW and our results? This are the results: rados bench single threaded: # rados bench 10 write --rbd-cache=false -t 1 Object size: 4194304 Bandwidth (MB/sec): 53.7186 Stddev Bandwidth: 3.86437 Max bandwidth (MB/sec): 60 Min bandwidth (MB/sec): 48 Average IOPS: 13 Stddev IOPS: 0.966092 Average Latency(s): 0.0744599 Stddev Latency(s): 0.00911778 nearly maxing out one (idle) client with 28 threads # rados bench 10 write --rbd-cache=false -t 28 Bandwidth (MB/sec): 850.451 Stddev Bandwidth: 40.6699 Max bandwidth (MB/sec): 904 Min bandwidth (MB/sec): 748 Average IOPS: 212 Stddev IOPS: 10.1675 Average Latency(s): 0.131309 Stddev Latency(s): 0.0318489 four concurrent benchmarks on four clients each with 24 threads: Bandwidth (MB/sec): 396 376 381 389 Stddev Bandwidth: 30 25 22 22 Max bandwidth (MB/sec): 440 420 416 428 Min bandwidth (MB/sec): 352 348 344 364 Average IOPS: 99 94 95 97 Stddev IOPS: 7.5 6.3 5.6 5.6 Average Latency(s): 0.24 0.25 0.25 0.24 Stddev Latency(s): 0.12 0.15 0.15 0.14 summing up: write mode ~1500 MB/sec Bandwidth ~385 IOPS ~0.25s Latency rand mode: ~3500 MB/sec ~920 IOPS ~0.154s Latency Maybe someone could judge our numbers. I am actually very satisfied with the values. The (mostly idle) cluster is build from these components: * 10GB frontend network, bonding two connections to mon-, mds- and osd-nodes ** no bonding to clients * 25GB backend network, bonding two connections to osd-nodes cluster: * 3x mon, 2x Intel(R) Xeon(R) Bronze 3104 CPU @ 1.70GHz, 64GB RAM * 3x mds, 1x Intel(R) Xeon(R) Gold 5115 CPU @ 2.40GHz, 128MB RAM * 7x OSD-nodes, 2x Intel(R) Xeon(R) Silver 4112 CPU @ 2.60GHz, 96GB RAM ** 4x 6TB SAS HDD HGST HUS726T6TAL5204 (5x on two nodes, max. 6x per chassis for later growth) ** 2x 800GB SAS SSD WDC WUSTM3280ASS200 => SW-RAID1 => LVM ~116 GiB per OSD for DB and WAL erasure encoded pool: (made for CephFS) * plugin=clay k=5 m=2 d=6 crush-failure-domain=host Thanks and best regards Lars _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |