I just ran your test on a cluster with 5 hosts 2x Intel 6130, 12x 860 Evo 2TB SSD per host (6 per SAS3008), 2x bonded 10GB NIC, 2x Arista switches.
Pool with 3x replication
rados bench -p scbench -b 4096 10 write --no-cleanup
hints = 1
Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 10 seconds or 0 objects
Object prefix: benchmark_data_dc1-kube-01_3458991
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 0 0 0 0 0 - 0
1 16 5090 5074 19.7774 19.8203 0.00312568 0.00315352
2 16 10441 10425 20.3276 20.9023 0.00332591 0.00307105
3 16 15548 15532 20.201 19.9492 0.00337573 0.00309134
4 16 20906 20890 20.3826 20.9297 0.00282902 0.00306437
5 16 26107 26091 20.3686 20.3164 0.00269844 0.00306698
6 16 31246 31230 20.3187 20.0742 0.00339814 0.00307462
7 16 36372 36356 20.2753 20.0234 0.00286653 0.0030813
8 16 41470 41454 20.2293 19.9141 0.00272051 0.00308839
9 16 46815 46799 20.3011 20.8789 0.00284063 0.00307738
Total time run: 10.0035
Total writes made: 51918
Write size: 4096
Object size: 4096
Bandwidth (MB/sec): 20.2734
Stddev Bandwidth: 0.464082
Max bandwidth (MB/sec): 20.9297
Min bandwidth (MB/sec): 19.8203
Average IOPS: 5189
Stddev IOPS: 118
Max IOPS: 5358
Min IOPS: 5074
Average Latency(s): 0.00308195
Stddev Latency(s): 0.00142825
Max latency(s): 0.0267947
Min latency(s): 0.00217364
rados bench -p scbench 10 rand
hints = 1
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 0 0 0 0 0 - 0
1 15 39691 39676 154.95 154.984 0.00027022 0.000395993
2 16 83701 83685 163.416 171.91 0.000318949 0.000375363
3 15 129218 129203 168.199 177.805 0.000300898 0.000364647
4 15 173733 173718 169.617 173.887 0.000311723 0.00036156
5 15 216073 216058 168.769 165.391 0.000407594 0.000363371
6 16 260381 260365 169.483 173.074 0.000323371 0.000361829
7 15 306838 306823 171.193 181.477 0.000284247 0.000358199
8 15 353675 353660 172.661 182.957 0.000338128 0.000355139
9 15 399221 399206 173.243 177.914 0.000422527 0.00035393
Total time run: 10.0003
Total reads made: 446353
Read size: 4096
Object size: 4096
Bandwidth (MB/sec): 174.351
Average IOPS: 44633
Stddev IOPS: 2220
Max IOPS: 46837
Min IOPS: 39676
Average Latency(s): 0.000351679
Max latency(s): 0.00530195
Min latency(s): 0.000135292
On Thu, Feb 7, 2019 at 2:17 AM <jesper@xxxxxxxx> wrote:
Hi List
We are in the process of moving to the next usecase for our ceph cluster
(Bulk, cheap, slow, erasurecoded, cephfs) storage was the first - and
that works fine.
We're currently on luminous / bluestore, if upgrading is deemed to
change what we're seeing then please let us know.
We have 6 OSD hosts, each with a S4510 of 1TB with 1 SSD in each. Connected
through a H700 MegaRaid Perc BBWC, EachDiskRaid0 - and scheduler set to
deadline, nomerges = 1, rotational = 0.
Each disk "should" give approximately 36K IOPS random write and the double
random read.
Pool is setup with a 3x replicaiton. We would like a "scaleout" setup of
well performing SSD block devices - potentially to host databases and
things like that. I ready through this nice document [0], I know the
HW are radically different from mine, but I still think I'm in the
very low end of what 6 x S4510 should be capable of doing.
Since it is IOPS i care about I have lowered block size to 4096 -- 4M
blocksize nicely saturates the NIC's in both directions.
$ sudo rados bench -p scbench -b 4096 10 write --no-cleanup
hints = 1
Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for
up to 10 seconds or 0 objects
Object prefix: benchmark_data_torsk2_11207
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 0 0 0 0 0 - 0
1 16 5857 5841 22.8155 22.8164 0.00238437 0.00273434
2 15 11768 11753 22.9533 23.0938 0.0028559 0.00271944
3 16 17264 17248 22.4564 21.4648 0.00246666 0.00278101
4 16 22857 22841 22.3037 21.8477 0.002716 0.00280023
5 16 28462 28446 22.2213 21.8945 0.00220186 0.002811
6 16 34216 34200 22.2635 22.4766 0.00234315 0.00280552
7 16 39616 39600 22.0962 21.0938 0.00290661 0.00282718
8 16 45510 45494 22.2118 23.0234 0.0033541 0.00281253
9 16 50995 50979 22.1243 21.4258 0.00267282 0.00282371
10 16 56745 56729 22.1577 22.4609 0.00252583 0.0028193
Total time run: 10.002668
Total writes made: 56745
Write size: 4096
Object size: 4096
Bandwidth (MB/sec): 22.1601
Stddev Bandwidth: 0.712297
Max bandwidth (MB/sec): 23.0938
Min bandwidth (MB/sec): 21.0938
Average IOPS: 5672
Stddev IOPS: 182
Max IOPS: 5912
Min IOPS: 5400
Average Latency(s): 0.00281953
Stddev Latency(s): 0.00190771
Max latency(s): 0.0834767
Min latency(s): 0.00120945
Min latency is fine -- but Max latency of 83ms ?
Average IOPS @ 5672 ?
$ sudo rados bench -p scbench 10 rand
hints = 1
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 0 0 0 0 0 - 0
1 15 23329 23314 91.0537 91.0703 0.000349856 0.000679074
2 16 48555 48539 94.7884 98.5352 0.000499159 0.000652067
3 16 76193 76177 99.1747 107.961 0.000443877 0.000622775
4 15 103923 103908 101.459 108.324 0.000678589 0.000609182
5 15 132720 132705 103.663 112.488 0.000741734 0.000595998
6 15 161811 161796 105.323 113.637 0.000333166 0.000586323
7 15 190196 190181 106.115 110.879 0.000612227 0.000582014
8 15 221155 221140 107.966 120.934 0.000471219 0.000571944
9 16 251143 251127 108.984 117.137 0.000267528 0.000566659
Total time run: 10.000640
Total reads made: 282097
Read size: 4096
Object size: 4096
Bandwidth (MB/sec): 110.187
Average IOPS: 28207
Stddev IOPS: 2357
Max IOPS: 30959
Min IOPS: 23314
Average Latency(s): 0.000560402
Max latency(s): 0.109804
Min latency(s): 0.000212671
This is also quite far from expected. I have 12GB of memory on the OSD
daemon for caching on each host - close to idle cluster - thus 50GB+ for
caching with a working set of < 6GB .. this should - in this case
not really be bound by the underlying SSD. But if it were:
IOPS/disk * num disks / replication => 95K * 6 / 3 => 190K or 6x off?
No measureable service time in iostat when running tests, thus I have
come to the conclusion that it has to be either client side, the
network path, or the OSD-daemon that deliveres the increasing latency /
decreased IOPS.
Is there any suggestions on how to get more insigths in that?
Has anyone replicated close to the number Micron are reporting on NVMe?
Thanks a log.
[0]
https://www.micron.com/-/media/client/global/documents/products/other-documents/micron_9200_max_ceph_12,-d-,2,-d-,8_luminous_bluestore_reference_architecture.pdf?la=en
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
