Rados bench before deleting OSD's and recreating them + syncing with fragmentation 0.89 > T1 - wr,4M > Total time run 60.0405 > Total writes made 9997 > Write size 4194304 > Object size 4194304 > Bandwidth (MB/sec) 666,017 > Stddev Bandwidth 24.1108 > Max bandwidth (MB/sec) 744 > Min bandwidth (MB/sec) 604 > Average IOPS 166 > Stddev IOPS 6.02769 > Max IOPS 186 > Min IOPS 151 > Average Latency(s) 0.0960791 > Stddev Latency(s) 0.0182781 > Max latency(s) 0.190993 > Min latency(s) 0.0284014 T2 = ro,seq,4M > Total time run 250.486 > Total reads made 9997 > Read size 4194304 > Object size 4194304 > Bandwidth (MB/sec) 1596,41 > Average IOPS 399 > Stddev IOPS 252.166 > Max IOPS 446 > Min IOPS 350 > Average Latency(s) 0.0395391 > Max latency(s) 0.187176 > Min latency(s) 0.0056981 T3 = ro,rand,4M > Total time run 600.463 > Total reads made 23.947 > Read size 4194304 > Object size 4194304 > Bandwidth (MB/sec) 1595,24 > Average IOPS 398 > Stddev IOPS 261.614 > Max IOPS 446 > Min IOPS 341 > Average Latency(s) 0.0395782 > Max latency(s) 0.17207 > Min latency(s) 0.00326339 Rados bench after recreating and sync with fragmentation 0.1 T1 > Total time run 60.0143 > Total writes made 30868 > Write size 4194304 > Object size 4194304 > Bandwidth (MB/sec) 2057,38 > Stddev Bandwidth 121.141 > Max bandwidth (MB/sec) 2208 > Min bandwidth (MB/sec) 1472 > Average IOPS 514 > Stddev IOPS 30.2852 > Max IOPS 552 > Min IOPS 368 > Average Latency(s) 0.0310978 > Stddev Latency(s) 0.0120903 > Max latency(s) 0.127144 > Min latency(s) 0.00719787 T2 > Total time run 51.9554 > Total reads made 30868 > Read size 4194304 > Object size 4194304 > Bandwidth (MB/sec) 2376,5 > Average IOPS 594 > Stddev IOPS 27.1142 > Max IOPS 641 > Min IOPS 543 > Average Latency(s) 0.026446 > Max latency(s) 0.120386 > Min latency(s) 0.00436071 T3 > Total time run 60.0455 > Total reads made 33853 > Read size 4194304 > Object size 4194304 > Bandwidth (MB/sec) 2255,16 > Average IOPS 563 > Stddev IOPS 23.7633 > Max IOPS 616 > Min IOPS 500 > Average Latency(s) 0.0278983 > Max latency(s) 0.13513 > Min latency(s) 0.00267677 > To me this looks like normal sequential write performance to an ssd. This is normal write for these OSD's when they are not fragmented and already have data , osd bench: > Server disk osd MB/s IOPS > s01 2TB osd.7 2.587 616 > s01 2TB osd.8 2.566 611 > s02 2TB osd.2 2.611 622 > s02 2TB osd.6 2.555 609 > s03 2TB osd.0 2.519 600 > s03 2TB osd.1 2.552 608 > s01 4TB osd.3 3.319 791 > s01 4TB osd.10 4.054 966 > s02 4TB osd.4 3.884 926 > s02 4TB osd.11 3.931 937 > s03 4TB osd.5 3.797 905 > s03 4TB osd.9 3.701 882 and this is when it is fragmented , first 3 are on 2TB drives OSD's and second 3 on 4TB MB/s IOPS > 455 108 > 449 107 > 533 127 > 846 201 > 825 196 > 795 189 > I am curious what makes you think this is related to the 'fragmentation_rating' I did hundreds of fio tests , rados , osd benches etc and recreated OSD's many times with different number of OSD's , PG's etc and only thing that is constant is this fragmentation that happens after few days of light use and all performance in all tests follows it. _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |