> Some more questions please:
> How many OSDs have you been using in your second email tests for 1gbit
> [1]
> and 10gbit [2] ethernet? Or to be precise, what is your cluster for
When I was testing with 1gbit ethernet I had 11 osds on 4 servers, but this already showed saturated 1Gbit links. Now on the 10gbit ethernet DAC it is with 30 hdd's or so. Keep in mind that the default rados bench is using 16 threads.
If I do the 1 thread I will get something like yours[1], if I do the same on the ssd pool, I get this[2]. And if I remove the 3x times replication on the ssd pool, this[3], and the 16 threads ssd pool with 3x on[4]
Side note is that I did not fully tune my cluster on performance, I have still processors doing frequency/powerstate switching. Have slower hdd sata drives combined with faster sas. But this fits my use case.
What I have should not be of interest to you. You have to determine what you need, and describe your use case, then there are quite a few good people here that can advice you how to realize that, or tell you it is not possible with ceph ;)
[@~]# rados bench -t 1 -p rbd 10 write
hints = 1
Maintaining 1 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects
Object prefix: benchmark_data_c01_3768767
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 0 0 0 0 0 - 0
1 1 5 4 15.9973 16 0.278477 0.240159
2 1 10 9 17.9973 20 0.162663 0.219858
3 1 17 16 21.33 28 0.21535 0.181435
4 1 26 25 24.9965 36 0.154064 0.158931
5 1 33 32 25.5966 28 0.119773 0.153031
6 1 42 41 27.3295 36 0.064895 0.144242
7 1 50 49 27.9962 32 0.192591 0.142036
8 1 59 58 28.9961 36 0.108623 0.137699
9 1 69 68 30.2183 40 0.0684741 0.132143
10 1 78 77 30.796 36 0.118075 0.12872
Total time run: 10.1903
Total writes made: 79
Write size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 31.01
Stddev Bandwidth: 7.78603
Max bandwidth (MB/sec): 40
Min bandwidth (MB/sec): 16
Average IOPS: 7
Stddev IOPS: 1.94651
Max IOPS: 10
Min IOPS: 4
Average Latency(s): 0.128988
Stddev Latency(s): 0.0571245
Max latency(s): 0.385165
Min latency(s): 0.0608502
Cleaning up (deleting benchmark objects)
Removed 79 objects
Clean up completed and total clean up time :2.49933
[2]
[@~]# rados bench -t 1 -p rbd.ssd 10 write
hints = 1
Maintaining 1 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects
Object prefix: benchmark_data_c01_3769249
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 0 0 0 0 0 - 0
1 1 39 38 151.992 152 0.0318137 0.0258572
2 1 80 79 157.985 164 0.0239471 0.0250284
3 1 122 121 161.315 168 0.0240444 0.0247604
4 1 163 162 161.981 164 0.0270316 0.024625
5 1 204 203 162.38 164 0.0235799 0.0245714
6 1 246 245 163.313 168 0.0296698 0.0244574
7 1 286 285 162.836 160 0.0232353 0.0245383
8 1 326 325 162.479 160 0.0236261 0.0245476
9 1 367 366 162.646 164 0.0249223 0.0245132
10 1 408 407 162.779 164 0.0229952 0.0245034
Total time run: 10.0277
Total writes made: 409
Write size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 163.149
Stddev Bandwidth: 4.63801
Max bandwidth (MB/sec): 168
Min bandwidth (MB/sec): 152
Average IOPS: 40
Stddev IOPS: 1.1595
Max IOPS: 42
Min IOPS: 38
Average Latency(s): 0.0245153
Stddev Latency(s): 0.00212425
Max latency(s): 0.0343171
Min latency(s): 0.0202639
Cleaning up (deleting benchmark objects)
Removed 409 objects
Clean up completed and total clean up time :0.521216
[3]
[@~]# rados bench -t 1 -p rbd.ssd.r1 10 write
hints = 1
Maintaining 1 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects
Object prefix: benchmark_data_c01_3769477
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 0 0 0 0 0 - 0
1 1 62 61 243.98 244 0.0149218 0.0162273
2 1 130 129 257.972 272 0.0144574 0.0154287
3 1 198 197 262.639 272 0.0144917 0.0151589
4 1 266 265 264.973 272 0.0156794 0.0150565
5 1 333 332 265.572 268 0.0149153 0.0150315
6 1 401 400 266.636 272 0.0143737 0.014966
7 1 469 468 267.399 272 0.0155345 0.0149459
8 1 536 535 267.471 268 0.0171765 0.0149397
9 1 604 603 267.971 272 0.0168833 0.0149184
10 1 672 671 268.37 272 0.0145986 0.0148998
Total time run: 10.0309
Total writes made: 673
Write size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 268.371
Stddev Bandwidth: 8.73308
Max bandwidth (MB/sec): 272
Min bandwidth (MB/sec): 244
Average IOPS: 67
Stddev IOPS: 2.18327
Max IOPS: 68
Min IOPS: 61
Average Latency(s): 0.014903
Stddev Latency(s): 0.00209157
Max latency(s): 0.0434543
Min latency(s): 0.0111273
Cleaning up (deleting benchmark objects)
Removed 673 objects
Clean up completed and total clean up time :0.69776
[4]
[@~]# rados bench -p rbd.ssd 10 write
hints = 1
Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects
Object prefix: benchmark_data_c01_3771109
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 0 0 0 0 0 - 0
1 16 198 182 727.843 728 0.0647907 0.0820398
2 16 400 384 767.86 808 0.0977614 0.0812242
3 16 603 587 782.536 812 0.0322522 0.0801139
4 16 801 785 784.878 792 0.0621462 0.0804142
5 16 996 980 783.883 780 0.0854195 0.0807731
6 16 1202 1186 790.546 824 0.041 0.0805795
7 16 1408 1392 795.307 824 0.122898 0.0798532
8 16 1608 1592 795.88 800 0.0382256 0.0802024
9 16 1791 1775 788.773 732 0.0480604 0.0807028
10 16 1997 1981 792.286 824 0.0581529 0.080433
Total time run: 10.07
Total writes made: 1997
Write size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 793.249
Stddev Bandwidth: 35.9481
Max bandwidth (MB/sec): 824
Min bandwidth (MB/sec): 728
Average IOPS: 198
Stddev IOPS: 8.98703
Max IOPS: 206
Min IOPS: 182
Average Latency(s): 0.0805776
Stddev Latency(s): 0.0339439
Max latency(s): 0.255658
Min latency(s): 0.0212272
Cleaning up (deleting benchmark objects)
Removed 1997 objects
Clean up completed and total clean up time :0.179425
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
