Hi,
TL;DR: In my Ceph clusters I replaced all OSDs from HDDs of several brands and models with Samsung 860 Pro SSDs and used
the opportunity to switch from filestore to bluestore. Now I'm seeing blocked ops in Ceph and file system freezes inside
VMs. Any suggestions?
I have two Proxmox clusters for virtualization which use Ceph on HDDs as backend storage for VMs. About half a year ago
I had to increase the pool size and used the occasion to switch from filestore to bluestore. That was when trouble
started. Both clusters showed blocked ops that caused freezes inside VMs which needed a reboot to function properly
again. I wasn't able to identify the cause of the blocking ops but I blamed the low performance of the HDDs. It was also
the time when patches for Spectre/Meltdown were released. Kernel 4.13.x didn't show the behavior while kernel 4.15.x
did. After several weeks of debugging the workaround was to go back to filestore.
Today I replace all HDDs with brand new Samsung 860 Pro SSDs and switched to bluestore again (on one cluster). And… the
blocked ops reappeared. I am out of ideas about the cause.
Any idea why bluestore is so much more demanding on the storage devices compared to filestore?
Before switching back to filestore do you have any suggestions for debugging? Anything special to check for in the network?
The clusters are both connected via 10GbE (MTU 9000) and are only lightly loaded (15 VMs on the first, 6 VMs on the
second). Each host has 3 SSDs and 64GB memory.
"rados bench" gives decent results for 4M block size but 4K block size triggers blocked ops (and only finishes after I
restart the OSD with the blocked ops). Results below.
Thanks,
Uwe
Results from "rados bench" runs with 4K block size when the cluster didn't block:
root@px-hotel-cluster:~# rados bench -p scbench 60 write -b 4K -t 16 --no-cleanup
hints = 1
Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 60 seconds or 0 objects
Object prefix: benchmark_data_px-hotel-cluster_3814550
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 0 0 0 0 0 - 0
1 16 2338 2322 9.06888 9.07031 0.0068972 0.0068597
2 16 4631 4615 9.01238 8.95703 0.0076618 0.00692027
3 16 6936 6920 9.00928 9.00391 0.0066511 0.00692966
4 16 9173 9157 8.94133 8.73828 0.00416256 0.00698071
5 16 11535 11519 8.99821 9.22656 0.00799875 0.00693842
6 16 13892 13876 9.03287 9.20703 0.00688782 0.00691459
7 15 16173 16158 9.01578 8.91406 0.00791589 0.00692736
8 16 18406 18390 8.97854 8.71875 0.00745151 0.00695723
9 16 20681 20665 8.96822 8.88672 0.0072881 0.00696475
10 16 23037 23021 8.99163 9.20312 0.00728763 0.0069473
11 16 24261 24245 8.60882 4.78125 0.00502342 0.00725673
12 16 25420 25404 8.26863 4.52734 0.00443917 0.00750865
13 16 27347 27331 8.21154 7.52734 0.00670819 0.00760455
14 16 28750 28734 8.01642 5.48047 0.00617038 0.00779322
15 16 30222 30206 7.8653 5.75 0.00700398 0.00794209
16 16 32180 32164 7.8517 7.64844 0.00704785 0.0079573
17 16 34527 34511 7.92907 9.16797 0.00582831 0.00788017
18 15 36969 36954 8.01868 9.54297 0.00635168 0.00779228
19 16 39059 39043 8.02609 8.16016 0.00622597 0.00778436
2019-02-26 21:55:41.623245 min lat: 0.00337595 max lat: 0.431158 avg lat: 0.00779143
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
20 16 41079 41063 8.01928 7.89062 0.00649895 0.00779143
21 16 43076 43060 8.00878 7.80078 0.00726145 0.00780128
22 16 45433 45417 8.06321 9.20703 0.00455727 0.00774944
23 16 47763 47747 8.10832 9.10156 0.00582818 0.00770599
24 16 50079 50063 8.14738 9.04688 0.0051125 0.00766894
25 16 52477 52461 8.19614 9.36719 0.00537575 0.00762343
26 16 54895 54879 8.24415 9.44531 0.00573134 0.00757909
27 16 57276 57260 8.28325 9.30078 0.00576683 0.00754383
28 16 59487 59471 8.29585 8.63672 0.00651535 0.00753232
29 16 61948 61932 8.34125 9.61328 0.00499461 0.00749048
30 16 64289 64273 8.36799 9.14453 0.00735917 0.00746708
31 16 66645 66629 8.3949 9.20312 0.00644432 0.00744233
32 16 68926 68910 8.41098 8.91016 0.00545702 0.0074289
33 16 71257 71241 8.432 9.10547 0.00505016 0.00741037
34 16 73668 73652 8.46097 9.41797 0.00432774 0.00738525
35 16 76088 76072 8.48929 9.45312 0.00514377 0.00736052
36 16 78340 78324 8.49781 8.79688 0.00623798 0.00735318
37 16 79702 79686 8.41192 5.32031 0.00586925 0.00742791
38 16 82048 82032 8.43169 9.16406 0.00631341 0.00741101
39 16 84574 84558 8.46848 9.86719 0.00457312 0.0073788
2019-02-26 21:56:01.625257 min lat: 0.00337595 max lat: 0.431158 avg lat: 0.00735446
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
40 16 87027 87011 8.4963 9.58203 0.00483967 0.00735446
41 16 89363 89347 8.5116 9.125 0.00689418 0.00734112
42 16 91532 91516 8.51066 8.47266 0.00589121 0.00734207
43 16 94011 93995 8.53791 9.68359 0.00687577 0.00731853
44 16 96510 96494 8.56571 9.76172 0.00554477 0.00729513
45 16 98825 98809 8.57629 9.04297 0.00659778 0.00728593
46 16 101096 101080 8.58269 8.87109 0.00645024 0.00728041
47 16 103334 103318 8.58606 8.74219 0.00533782 0.00727755
48 16 105796 105780 8.60752 9.61719 0.00481175 0.00725942
49 16 108209 108193 8.6242 9.42578 0.00665163 0.00724555
50 16 110557 110541 8.63514 9.17188 0.00514146 0.0072366
51 16 112799 112783 8.63753 8.75781 0.00542164 0.00723403
52 16 115145 115129 8.64764 9.16406 0.0078739 0.0072261
53 16 117599 117583 8.66532 9.58594 0.00636182 0.00721115
54 16 119951 119935 8.67497 9.1875 0.00902746 0.00720322
55 16 122231 122215 8.67916 8.90625 0.00769823 0.00719975
56 16 124473 124457 8.68055 8.75781 0.00455414 0.00719853
57 16 126845 126829 8.6908 9.26562 0.00440477 0.00719023
58 16 129172 129156 8.69766 9.08984 0.014724 0.00718433
59 16 130642 130626 8.64756 5.74219 0.0044018 0.0072242
2019-02-26 21:56:21.627245 min lat: 0.00337595 max lat: 0.431158 avg lat: 0.00722993
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
60 13 132775 132762 8.64248 8.34375 0.0074357 0.00722993
Total time run: 60.006703
Total writes made: 132775
Write size: 4096
Object size: 4096
Bandwidth (MB/sec): 8.64324
Stddev Bandwidth: 1.24153
Max bandwidth (MB/sec): 9.86719
Min bandwidth (MB/sec): 4.52734
Average IOPS: 2212
Stddev IOPS: 317
Max IOPS: 2526
Min IOPS: 1159
Average Latency(s): 0.00723007
Stddev Latency(s): 0.00965629
Max latency(s): 0.431158
Min latency(s): 0.00337595
root@px-hotel-cluster:~# rados bench -p scbench 60 seq -t 16 --no-cleanup
hints = 1
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 0 0 0 0 0 - 0
1 15 25546 25531 99.7129 99.7305 0.00038738 0.000617037
2 15 51245 51230 100.043 100.387 0.000679383 0.000614879
3 15 77097 77082 100.353 100.984 0.000649252 0.000613069
4 16 102482 102466 100.051 99.1562 0.000665031 0.000615048
5 16 127877 127861 99.8788 99.1992 0.0003609 0.000616109
Total time run: 5.189528
Total reads made: 132775
Read size: 4096
Object size: 4096
Bandwidth (MB/sec): 99.9421
Average IOPS: 25585
Stddev IOPS: 201
Max IOPS: 25852
Min IOPS: 25384
Average Latency(s): 0.000615755
Max latency(s): 0.00516975
Min latency(s): 0.000300567
root@px-hotel-cluster:~# rados bench -p scbench 60 rand -t 16 --no-cleanup
hints = 1
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 0 0 0 0 0 - 0
1 16 29609 29593 115.579 115.598 0.000356415 0.0005319
2 16 58715 58699 114.63 113.695 0.000685151 0.000536463
3 15 88167 88152 114.765 115.051 0.000372257 0.000535853
4 15 117488 117473 114.705 114.535 0.00047013 0.000536184
5 15 146694 146679 114.579 114.086 0.00037833 0.000536823
6 16 176406 176390 114.824 116.059 0.000659678 0.000535637
7 15 205080 205065 114.421 112.012 0.00028265 0.000537569
8 15 235437 235422 114.939 118.582 0.000438279 0.000535161
9 15 264395 264380 114.736 113.117 0.000619314 0.000536149
10 15 293705 293690 114.71 114.492 0.00050762 0.000536255
11 15 322550 322535 114.524 112.676 0.00116935 0.000537162
12 15 350283 350268 114.007 108.332 0.000482695 0.000539653
13 15 380550 380535 114.331 118.23 0.000637042 0.000538136
14 15 409882 409867 114.348 114.578 0.000340766 0.000538077
15 15 439190 439175 114.357 114.484 0.000901077 0.000538053
16 16 468430 468414 114.347 114.215 0.000526489 0.000538119
17 15 498201 498186 114.461 116.297 0.00039584 0.000537558
18 15 527448 527433 114.449 114.246 0.000561086 0.000537584
19 16 557524 557508 114.608 117.48 0.000459305 0.000536828
2019-02-26 21:57:29.158335 min lat: 0.000193689 max lat: 0.00471296 avg lat: 0.000536229
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
20 15 587514 587499 114.734 117.152 0.000298343 0.000536229
21 16 615970 615954 114.563 111.152 0.000617898 0.000537056
22 16 646448 646432 114.767 119.055 0.000365562 0.000536103
23 15 676857 676842 114.941 118.789 0.00053347 0.000535283
24 16 707108 707092 115.075 118.164 0.000489048 0.00053466
25 15 735298 735283 114.876 110.121 0.000879961 0.000535606
26 16 763299 763283 114.664 109.375 0.000893751 0.00053662
27 15 792847 792832 114.692 115.426 0.000289513 0.000536496
28 15 822380 822365 114.716 115.363 0.000363185 0.000536393
29 15 851806 851791 114.723 114.945 0.000400721 0.00053637
30 15 882240 882225 114.862 118.883 0.000323953 0.000535728
31 16 912697 912681 114.994 118.969 0.000584203 0.00053511
32 16 942922 942906 115.09 118.066 0.000237542 0.000534669
33 15 972480 972465 115.101 115.465 0.000304352 0.000534625
34 15 1002004 1001989 115.107 115.328 0.000417205 0.000534598
35 15 1031656 1031641 115.127 115.828 0.000731773 0.000534508
36 15 1061794 1061779 115.199 117.727 0.000399053 0.000534167
37 15 1090982 1090967 115.167 114.016 0.000273898 0.000534303
38 15 1120932 1120917 115.215 116.992 0.000563786 0.000534085
39 16 1151319 1151303 115.304 118.695 0.000584587 0.000533673
2019-02-26 21:57:49.160177 min lat: 0.000189357 max lat: 0.00551016 avg lat: 0.000533845
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
40 15 1180422 1180407 115.263 113.688 0.000266425 0.000533845
41 16 1209042 1209026 115.178 111.793 0.000877275 0.000534238
42 16 1238819 1238803 115.205 116.316 0.000268862 0.000534114
43 15 1268209 1268194 115.195 114.809 0.000478607 0.000534166
44 15 1298456 1298441 115.262 118.152 0.000371342 0.000533858
45 15 1327611 1327596 115.232 113.887 0.000881212 0.000534
46 15 1356842 1356827 115.209 114.184 0.00046055 0.000534112
47 15 1386848 1386833 115.251 117.211 0.000775375 0.000533913
48 15 1415035 1415020 115.143 110.105 0.000364161 0.00053442
49 15 1444927 1444912 115.176 116.766 0.000374783 0.00053427
50 15 1475316 1475301 115.247 118.707 0.000564861 0.000533942
51 16 1504352 1504336 115.211 113.418 0.00068548 0.000534105
52 15 1533646 1533631 115.196 114.434 0.000288343 0.000534174
53 15 1564054 1564039 115.263 118.781 0.000587993 0.00053386
54 15 1593366 1593351 115.249 114.5 0.000436205 0.00053392
55 15 1622167 1622152 115.199 112.504 0.000601619 0.000534158
56 15 1652256 1652241 115.24 117.535 0.00067666 0.000533963
57 15 1681711 1681696 115.237 115.059 0.000723098 0.000533968
58 15 1710686 1710671 115.201 113.184 0.000970663 0.000534127
59 15 1740906 1740891 115.249 118.047 0.000314217 0.000533908
Total time run: 60.000464
Total reads made: 1770108
Read size: 4096
Object size: 4096
Bandwidth (MB/sec): 115.241
Average IOPS: 29501
Stddev IOPS: 666
Max IOPS: 30478
Min IOPS: 27733
Average Latency(s): 0.000533953
Max latency(s): 0.00551016
Min latency(s): 0.000189357
And results with 4M block size:
root@px-hotel-cluster:~# ceph osd pool create scbench 100 100
pool 'scbench' created
root@px-hotel-cluster:~# rados bench -p scbench 60 write -b 4M -t 16 --no-cleanup
hints = 1
Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 60 seconds or 0 objects
Object prefix: benchmark_data_px-hotel-cluster_3803595
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 0 0 0 0 0 - 0
1 16 121 105 419.981 420 0.0684363 0.14782
2 16 231 215 429.951 440 0.0402411 0.141534
3 16 365 349 465.275 536 0.0435463 0.134897
4 16 470 454 453.944 420 0.0554416 0.135084
5 16 568 552 441.544 392 0.233723 0.140098
6 16 684 668 445.276 464 0.0357724 0.140952
7 16 823 807 461.085 556 0.062224 0.138544
8 16 926 910 454.942 412 0.263991 0.139367
9 16 1044 1028 456.829 472 0.146897 0.138908
10 16 1153 1137 454.741 436 0.0995426 0.13898
11 16 1278 1262 458.85 500 0.0397514 0.138972
12 16 1380 1364 454.609 408 0.0697221 0.138761
13 16 1511 1495 459.942 524 0.090388 0.138845
14 16 1621 1605 458.514 440 0.100218 0.138448
15 16 1748 1732 461.809 508 0.0357433 0.138012
16 16 1858 1842 460.443 440 0.0764942 0.138516
17 16 1961 1945 457.591 412 0.209572 0.139726
18 16 2064 2048 455.055 412 0.104463 0.139988
19 16 2170 2154 453.418 424 0.165126 0.140337
2019-02-26 21:24:44.759567 min lat: 0.0227191 max lat: 0.318478 avg lat: 0.14015
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
20 16 2293 2277 455.344 492 0.0785952 0.14015
21 16 2411 2395 456.134 472 0.137855 0.140047
22 16 2527 2511 456.489 464 0.081168 0.13934
23 16 2650 2634 458.03 492 0.127442 0.139296
24 16 2765 2749 458.11 460 0.0591567 0.139316
25 16 2870 2854 456.582 420 0.0341181 0.139131
26 16 2996 2980 458.403 504 0.0631039 0.13931
27 16 3110 3094 458.312 456 0.0528259 0.139285
28 16 3217 3201 457.227 428 0.057046 0.139622
29 16 3334 3318 457.596 468 0.0444768 0.139484
30 16 3452 3436 458.074 472 0.167902 0.139238
31 16 3556 3540 456.715 416 0.0392211 0.13946
32 16 3669 3653 456.566 452 0.116063 0.139371
33 16 3789 3773 457.274 480 0.10848 0.139689
34 16 3900 3884 456.881 444 0.0312001 0.139837
35 16 4005 3989 455.826 420 0.0999828 0.139968
36 16 4126 4110 456.607 484 0.0748115 0.139842
37 16 4233 4217 455.832 428 0.171641 0.140061
38 16 4333 4317 454.361 400 0.0546583 0.140427
39 16 4455 4439 455.222 488 0.109672 0.140391
2019-02-26 21:25:04.762404 min lat: 0.0219402 max lat: 0.453859 avg lat: 0.140093
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
40 16 4568 4552 455.14 452 0.0341032 0.140093
41 16 4691 4675 456.037 492 0.0622903 0.140183
42 16 4810 4794 456.511 476 0.0368881 0.139874
43 16 4930 4914 457.056 480 0.136802 0.139648
44 16 5043 5027 456.939 452 0.0941255 0.139592
45 16 5168 5152 457.895 500 0.0859091 0.139689
46 16 5275 5259 457.243 428 0.0553456 0.139781
47 16 5384 5368 456.79 436 0.0559706 0.139848
48 16 5494 5478 456.439 440 0.0504884 0.140001
49 16 5608 5592 456.429 456 0.0936318 0.139992
50 16 5736 5720 457.539 512 0.172677 0.139731
51 16 5865 5849 458.683 516 0.0606197 0.139352
52 16 5987 5971 459.246 488 0.0602065 0.139042
53 16 6096 6080 458.806 436 0.0967087 0.139395
54 16 6212 6196 458.901 464 0.0863546 0.139228
55 16 6330 6314 459.138 472 0.143207 0.139147
56 16 6439 6423 458.724 436 0.160516 0.139281
57 16 6544 6528 458.044 420 0.0770203 0.139479
58 16 6657 6641 457.938 452 0.0462132 0.139585
59 16 6769 6753 457.769 448 0.0890968 0.139535
2019-02-26 21:25:24.765186 min lat: 0.0219402 max lat: 0.453859 avg lat: 0.139601
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
60 16 6885 6869 457.872 464 0.0683147 0.139601
Total time run: 60.064041
Total writes made: 6885
Write size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 458.511
Stddev Bandwidth: 35.849
Max bandwidth (MB/sec): 556
Min bandwidth (MB/sec): 392
Average IOPS: 114
Stddev IOPS: 8
Max IOPS: 139
Min IOPS: 98
Average Latency(s): 0.139576
Stddev Latency(s): 0.0662593
Max latency(s): 0.453859
Min latency(s): 0.0219402
root@px-hotel-cluster:~# rados bench -p scbench 60 seq -t 16 --no-cleanup
hints = 1
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 0 0 0 0 0 - 0
1 16 384 368 1471.73 1472 0.01866 0.0415297
2 16 696 680 1359.77 1248 0.0917004 0.0459489
3 15 1048 1033 1376.92 1412 0.0166296 0.0452437
4 16 1384 1368 1367.65 1340 0.0140827 0.0458988
5 16 1775 1759 1406.88 1564 0.0225302 0.044743
6 16 2128 2112 1407.71 1412 0.0298933 0.0448714
7 16 2527 2511 1434.58 1596 0.020457 0.0439445
8 15 2912 2897 1448.2 1544 0.0159675 0.0435627
9 16 3313 3297 1465.05 1600 0.0186496 0.0429805
10 16 3624 3608 1442.93 1244 0.0366258 0.0435085
11 15 3942 3927 1427.7 1276 0.0179256 0.0442724
12 16 4302 4286 1428.38 1436 0.0415135 0.0442627
13 16 4714 4698 1445.25 1648 0.0256314 0.0436973
14 16 5049 5033 1437.73 1340 0.0183511 0.0438648
15 16 5397 5381 1434.67 1392 0.0467047 0.0439667
16 16 5791 5775 1443.49 1576 0.103247 0.0437964
17 16 6169 6153 1447.51 1512 0.0411054 0.0436606
18 16 6574 6558 1457.08 1620 0.0331255 0.0433989
Total time run: 18.849876
Total reads made: 6885
Read size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 1461.02
Average IOPS: 365
Stddev IOPS: 33
Max IOPS: 412
Min IOPS: 311
Average Latency(s): 0.043257
Max latency(s): 0.420441
Min latency(s): 0.0106036
root@px-hotel-cluster:~# rados bench -p scbench 60 rand -t 16 --no-cleanup
hints = 1
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 0 0 0 0 0 - 0
1 16 470 454 1815.73 1816 0.0512056 0.0339588
2 16 933 917 1833.73 1852 0.012993 0.0339007
3 16 1384 1368 1823.74 1804 0.0229874 0.0341975
4 16 1824 1808 1807.74 1760 0.0315442 0.0346609
5 16 2293 2277 1821.34 1876 0.018928 0.0343011
6 16 2734 2718 1811.73 1764 0.0172553 0.0345841
7 16 3188 3172 1812.31 1816 0.0526328 0.0346511
8 16 3648 3632 1815.74 1840 0.064372 0.0346222
9 16 4104 4088 1816.63 1824 0.0580501 0.0346048
10 16 4556 4540 1815.75 1808 0.0203061 0.0346393
11 15 5034 5019 1824.84 1916 0.0323724 0.034475
12 16 5502 5486 1828.42 1868 0.0418273 0.0343747
13 16 5961 5945 1828.98 1836 0.0910453 0.0343914
14 16 6423 6407 1830.32 1848 0.0229763 0.0343634
15 16 6889 6873 1832.56 1864 0.0106228 0.0342979
16 15 7365 7350 1837.26 1908 0.00837461 0.0342174
17 15 7847 7832 1842.58 1928 0.0247365 0.0341596
18 15 8323 8308 1845.98 1904 0.00197862 0.0340899
19 15 8760 8745 1840.82 1748 0.0185352 0.0341643
2019-02-26 21:26:56.730479 min lat: 0.00169505 max lat: 0.257525 avg lat: 0.0341765
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
20 16 9224 9208 1841.37 1852 0.0283496 0.0341765
21 16 9683 9667 1841.09 1836 0.0476514 0.0341884
22 16 10149 10133 1842.13 1864 0.0382981 0.0341645
23 16 10601 10585 1840.63 1808 0.0399852 0.0341945
24 15 11054 11039 1839.6 1816 0.0224427 0.0342213
25 16 11515 11499 1839.6 1840 0.020452 0.0342026
26 16 11995 11979 1842.68 1920 0.0703125 0.034164
27 16 12464 12448 1843.91 1876 0.0693259 0.0341387
28 15 12928 12913 1844.48 1860 0.0346587 0.0341312
29 16 13392 13376 1844.73 1852 0.0227206 0.0341126
30 16 13843 13827 1843.36 1804 0.0640094 0.0341546
31 15 14296 14281 1842.47 1816 0.0945452 0.0341648
32 15 14757 14742 1842.51 1844 0.0189832 0.034163
33 16 15196 15180 1839.76 1752 0.0608944 0.0342256
34 16 15632 15616 1836.94 1744 0.0829651 0.0342715
35 15 16087 16072 1836.56 1824 0.0292455 0.0342801
36 15 16549 16534 1836.87 1848 0.0245028 0.034283
37 16 16988 16972 1834.57 1752 0.0799359 0.0343315
38 15 17414 17399 1831.24 1708 0.0173303 0.0343958
39 16 17871 17855 1831.04 1824 0.0402483 0.0343933
2019-02-26 21:27:16.733211 min lat: 0.0016877 max lat: 0.257525 avg lat: 0.034381
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
40 16 18330 18314 1831.16 1836 0.114651 0.034381
41 15 18777 18762 1830.2 1792 0.0231119 0.0344147
42 15 19256 19241 1832.24 1916 0.0281483 0.0343706
43 16 19678 19662 1828.78 1684 0.0112887 0.0343865
44 16 20116 20100 1827.03 1752 0.0388422 0.0344744
45 16 20571 20555 1826.87 1820 0.0457656 0.0344798
46 16 21039 21023 1827.85 1872 0.0351042 0.0344607
47 15 21477 21462 1826.32 1756 0.0206584 0.034491
48 15 21916 21901 1824.85 1756 0.00505084 0.0345191
49 16 22378 22362 1825.24 1844 0.0492922 0.0345123
50 16 22817 22801 1823.84 1756 0.0874689 0.0345343
51 16 23266 23250 1823.29 1796 0.0554869 0.0345511
52 16 23723 23707 1823.38 1828 0.046502 0.0345468
53 16 24194 24178 1824.52 1884 0.0449133 0.0345267
54 16 24673 24657 1826.21 1916 0.0242969 0.0344924
55 16 25133 25117 1826.46 1840 0.117927 0.0344925
56 16 25576 25560 1825.48 1772 0.0308638 0.0345085
57 16 26033 26017 1825.52 1828 0.0229086 0.0345071
58 15 26502 26487 1826.46 1880 0.0366591 0.0345002
59 16 26952 26936 1825.94 1796 0.0328505 0.0345056
2019-02-26 21:27:36.735624 min lat: 0.0016877 max lat: 0.257525 avg lat: 0.0344872
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
60 16 27414 27398 1826.3 1848 0.0153289 0.0344872
Total time run: 60.050631
Total reads made: 27414
Read size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 1826.06
Average IOPS: 456
Stddev IOPS: 13
Max IOPS: 482
Min IOPS: 421
Average Latency(s): 0.0345121
Max latency(s): 0.257525
Min latency(s): 0.0016877
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
