Hi Marc,
Thanks for sharing. How many nodes/OSDs?, I get the following tail for
the same command and size=3 (3 nodes, 4 OSD each):
# rados bench -p bench 60 write
hints = 1
Maintaining 16 concurrent writes of 4194304 bytes to objects of size
4194304 for up to 60 seconds or 0 objects
Object prefix: benchmark_data_recev08_56750
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg
lat(s)
0 0 0 0 0 0 - 0
1 16 216 200 799.957 800 0.0614726 0.0767338
2 16 423 407 813.942 828 0.134841 0.0763884
3 16 630 614 818.604 828 0.0665102 0.0768481
4 16 844 828 827.934 856 0.0501812 0.0765091
5 16 1056 1040 831.932 848 0.0433767 0.0762525
6 16 1256 1240 826.597 800 0.101928 0.076811
7 16 1477 1461 834.785 884 0.0607711 0.0762284
8 16 1686 1670 834.927 836 0.0324832 0.0759334
9 16 1896 1880 835.482 840 0.0618377 0.0762244
10 16 2101 2085 833.925 820 0.108434 0.0762006
11 16 2307 2291 833.016 824 0.0544224 0.0765177
12 16 2522 2506 835.258 860 0.0340822 0.0762951
13 16 2730 2714 835.002 832 0.16751 0.076386
14 16 2943 2927 836.211 852 0.0498373 0.0763445
15 16 3151 3135 835.925 832 0.0616787 0.0763538
16 16 3354 3338 834.425 812 0.0789788 0.0765443
17 16 3557 3541 833.101 812 0.0604666 0.076477
18 16 3770 3754 834.146 852 0.0434525 0.0764539
19 16 3979 3963 834.239 836 0.0415418 0.0765218
2022-07-07T12:20:38.242682+0200 min lat: 0.0218974 max lat: 0.273712 avg
lat: 0.0765015
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg
lat(s)
20 16 4189 4173 834.523 840 0.0391263 0.0765015
21 16 4399 4383 834.778 840 0.0458377 0.0764423
22 16 4602 4586 833.739 812 0.104963 0.0766156
23 16 4820 4804 835.399 872 0.0315176 0.0764862
24 16 5034 5018 836.254 856 0.0647235 0.0764235
25 16 5241 5225 835.921 828 0.0378583 0.0764432
26 16 5452 5436 836.228 844 0.162137 0.0763584
27 16 5673 5657 837.994 884 0.0527343 0.0761778
28 16 5875 5859 836.92 808 0.0802883 0.0763725
29 16 6083 6067 836.748 832 0.035094 0.0763403
30 16 6284 6268 835.653 804 0.145906 0.076455
31 16 6489 6473 835.146 820 0.147326 0.0765264
32 16 6705 6689 836.045 864 0.0538247 0.0764059
33 16 6924 6908 837.253 876 0.0494262 0.0763401
34 16 7135 7119 837.449 844 0.0464947 0.0763056
35 16 7346 7330 837.634 844 0.0557179 0.0762938
36 16 7559 7543 838.03 852 0.0400139 0.0762645
37 16 7773 7757 838.514 856 0.0613892 0.0762203
38 16 7976 7960 837.814 812 0.1253 0.076288
39 16 8180 8164 837.252 816 0.0868544 0.0763577
2022-07-07T12:20:58.244706+0200 min lat: 0.0218974 max lat: 0.287681 avg
lat: 0.0763664
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg
lat(s)
40 16 8386 8370 836.919 824 0.127506 0.0763664
41 16 8595 8579 836.894 836 0.0468506 0.0763887
42 16 8799 8783 836.394 816 0.0466729 0.0764293
43 16 9017 9001 837.22 872 0.0674629 0.0763611
44 16 9222 9206 836.827 820 0.0979357 0.0763808
45 16 9429 9413 836.629 828 0.0337436 0.076392
46 16 9641 9625 836.874 848 0.0809087 0.076402
47 16 9851 9835 836.939 840 0.0624201 0.0763971
48 16 10066 10050 837.418 860 0.106171 0.076356
49 16 10279 10263 837.714 852 0.0554064 0.0763406
50 16 10479 10463 836.958 800 0.0436087 0.0763697
51 16 10687 10671 836.859 832 0.113212 0.076415
52 16 10904 10888 837.456 868 0.17198 0.0763662
53 16 11099 11083 836.371 780 0.189178 0.0764542
54 16 11317 11301 837.029 872 0.0412388 0.0763851
55 16 11531 11515 837.372 856 0.0534224 0.0763493
56 16 11741 11725 837.417 840 0.0820318 0.0763512
57 16 11956 11940 837.811 860 0.0565221 0.0763373
58 16 12169 12153 838.055 852 0.0427414 0.076283
59 16 12372 12356 837.612 812 0.0888336 0.0763543
2022-07-07T12:21:18.246792+0200 min lat: 0.0218974 max lat: 0.287681 avg
lat: 0.07633
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg
lat(s)
60 15 12582 12567 837.717 844 0.0773648 0.07633
Total time run: 60.0575
Total writes made: 12582
Write size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 837.997
Stddev Bandwidth: 22.859
Max bandwidth (MB/sec): 884
Min bandwidth (MB/sec): 780
Average IOPS: 209
Stddev IOPS: 5.71476
Max IOPS: 221
Min IOPS: 195
Average Latency(s): 0.0763377
Stddev Latency(s): 0.0384322
Max latency(s): 0.287681
Min latency(s): 0.0218974
Cleaning up (deleting benchmark objects)
Removed 12582 objects
Clean up completed and total clean up time :1.0554
El 6/7/22 a las 22:35, Marc escribió:
This is from rbd ssd pool 3x replication (sata ssd drives, 2.2 cpu's are on balanced not optimized, nautilus)
[@~]# rados bench -p rbd.ssd 60 write
hints = 1
Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 60 seconds or 0 objects
Object prefix: benchmark_data_c01_1487351
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 0 0 0 0 0 - 0
1 16 145 129 515.975 516 0.0720097 0.117034
2 16 250 234 467.954 420 0.0332527 0.122599
3 16 353 337 449.286 412 0.0254232 0.13569
4 16 447 431 430.95 376 0.0308228 0.140396
5 16 550 534 427.149 412 0.0489281 0.147189
6 16 656 640 426.616 424 0.0365499 0.147231
7 16 739 723 413.094 332 0.0261517 0.149667
8 16 837 821 410.451 392 0.0242721 0.150902
9 16 934 918 407.952 388 0.0700446 0.155572
10 16 1040 1024 409.55 424 0.0546874 0.155168
11 16 1135 1119 406.86 380 0.0246981 0.154823
12 16 1228 1212 403.951 372 0.0281447 0.156045
13 16 1329 1313 403.951 404 0.0577759 0.157578
14 16 1424 1408 402.237 380 0.15696 0.157949
15 16 1528 1512 403.152 416 0.0756755 0.158072
16 16 1618 1602 400.451 360 0.0596802 0.158514
17 16 1710 1694 398.539 368 0.0515091 0.15869
18 16 1795 1779 395.284 340 0.218853 0.161703
19 16 1890 1874 394.477 380 0.0435594 0.161365
2022-07-06 22:19:01.255264 min lat: 0.0219673 max lat: 0.49047 avg lat: 0.159777
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
20 16 1998 1982 396.35 432 0.0638213 0.159777
21 16 2083 2067 393.663 340 0.0627098 0.161907
22 16 2191 2175 395.403 432 0.19454 0.161806
23 16 2273 2257 392.47 328 0.138672 0.162491
24 16 2367 2351 391.782 376 0.0281001 0.161778
25 16 2464 2448 391.629 388 0.0220856 0.161989
26 16 2562 2546 391.642 392 0.0309591 0.16242
27 16 2657 2641 391.209 380 0.343609 0.163424
28 16 2743 2727 389.521 344 0.145933 0.163728
29 16 2844 2828 390.019 404 0.0241817 0.163001
30 16 2935 2919 389.15 364 0.427961 0.164419
31 16 3014 2998 386.788 316 0.0307399 0.16469
32 16 3101 3085 385.574 348 0.0506665 0.165257
33 16 3174 3158 382.738 292 0.0260261 0.165897
34 16 3275 3259 383.361 404 0.0596022 0.166677
35 16 3399 3383 386.578 496 0.0445334 0.165292
36 16 3495 3479 386.504 384 0.0253431 0.164562
37 16 3604 3588 387.841 436 0.102069 0.164639
38 16 3689 3673 386.58 340 0.0282991 0.16442
39 16 3765 3749 384.461 304 0.0988234 0.166138
2022-07-06 22:19:21.258166 min lat: 0.0208629 max lat: 0.49047 avg lat: 0.165966
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
40 16 3849 3833 383.248 336 0.025483 0.165966
41 16 3936 3920 382.386 348 0.445226 0.167291
42 16 4021 4005 381.376 340 0.0527353 0.167655
43 16 4090 4074 378.924 276 0.101183 0.168101
44 16 4179 4163 378.402 356 0.0787725 0.168783
45 16 4282 4266 379.147 412 0.0804984 0.168064
46 16 4369 4353 378.469 348 0.0977855 0.168786
47 16 4469 4453 378.926 400 0.0804289 0.168483
48 16 4541 4525 377.031 288 0.195228 0.169554
49 16 4627 4611 376.356 344 0.19558 0.169988
50 16 4722 4706 376.428 380 0.0712425 0.169813
51 16 4801 4785 375.242 316 0.203578 0.169939
52 16 4900 4884 375.64 396 0.344731 0.170263
53 16 4993 4977 375.571 372 0.0691528 0.169995
54 16 5096 5080 376.244 412 0.0515177 0.169545
55 15 5183 5168 375.803 352 0.0509726 0.170167
56 16 5262 5246 374.663 312 0.0427843 0.170383
57 16 5355 5339 374.615 372 0.060925 0.170397
58 16 5445 5429 374.362 360 0.0669999 0.170316
59 16 5533 5517 373.983 352 0.0881168 0.170904
2022-07-06 22:19:41.260960 min lat: 0.0208629 max lat: 0.538331 avg lat: 0.170964
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
60 16 5623 5607 373.749 360 0.10889 0.170964
Total time run: 60.1307
Total writes made: 5623
Write size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 374.052
Stddev Bandwidth: 45.1854
Max bandwidth (MB/sec): 516
Min bandwidth (MB/sec): 276
Average IOPS: 93
Stddev IOPS: 11.2964
Max IOPS: 129
Min IOPS: 69
Average Latency(s): 0.171096
Stddev Latency(s): 0.131978
Max latency(s): 0.538331
Min latency(s): 0.0208629
Cleaning up (deleting benchmark objects)
Removed 5623 objects
Clean up completed and total clean up time :0.645015
Eneko Lacunza
Zuzendari teknikoa | Director técnico
Binovo IT Human Project
Tel. +34 943 569 206 |https://www.binovo.es
Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun
https://www.youtube.com/user/CANALBINOVO
https://www.linkedin.com/company/37269706/
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
