> Where is the rados bench before and after your problem? Rados bench before deleting OSD's and recreating them + syncing with fragmentation 0.89 T1 - wr,4M T2 = ro,seq,4M T3 = ro,rand,4M > Total time run 60.0405 Total time run 250.486 Total time run > 600.463 > Total writes made 9997 Total reads made 9997 Total reads made 23.947 > Write size 4194304 Read size 4194304 Read size 4194304 > Object size 4194304 Object size 4194304 Object size 4194304 > Bandwidth (MB/sec) 666,017 Bandwidth (MB/sec) 1596,41 Bandwidth (MB/sec) > 1595,24 > Stddev Bandwidth 24.1108 Average IOPS 399 Average IOPS 398 > Max bandwidth (MB/sec) 744 Stddev IOPS 252.166 Stddev IOPS 261.614 > Min bandwidth (MB/sec) 604 Max IOPS 446 Max IOPS 446 > Average IOPS 166 Min IOPS 350 Min IOPS 341 > Stddev IOPS 6.02769 Average Latency(s) 0.0395391 Average > Latency(s) 0.0395782 > Max IOPS 186 Max latency(s) 0.187176 Max latency(s) > 0.17207 > Min IOPS 151 Min latency(s) 0.0056981 Min latency(s) > 0.00326339 > Average Latency(s) 0.0960791 > Stddev Latency(s) 0.0182781 > Max latency(s) 0.190993 > Min latency(s) 0.0284014 Rados bench after recreating and sync with fragmentation 0.1 T1 T2 T3 > Total time run 60.0143 Total time run 51.9554 Total time > run 60.0455 > Total writes made 30868 Total reads made 30868 Total reads made > 33853 > Write size 4194304 Read size 4194304 Read size > 4194304 > Object size 4194304 Object size 4194304 Object size > 4194304 > Bandwidth (MB/sec) 2057,38 Bandwidth (MB/sec) 2376,5 Bandwidth (MB/sec) > 2255,16 > Stddev Bandwidth 121.141 Average IOPS 594 Average IOPS 563 > Max bandwidth (MB/sec) 2208 Stddev IOPS 27.1142 Stddev IOPS > 23.7633 > Min bandwidth (MB/sec) 1472 Max IOPS 641 Max IOPS > 616 > Average IOPS 514 Min IOPS 543 Min IOPS > 500 > Stddev IOPS 30.2852 Average Latency(s) 0.026446 Average > Latency(s) 0.0278983 > Max IOPS 552 Max latency(s) 0.120386 Max latency(s) > 0.13513 > Min IOPS 368 Min latency(s) 0.00436071 Min > latency(s) 0.00267677 > Average Latency(s) 0.0310978 > Stddev Latency(s) 0.0120903 > Max latency(s) 0.127144 > Min latency(s) 0.00719787 > > To me this looks like normal sequential write performance to an ssd. This is normal write for these OSD's when they are not fragmented and already have data , osd bench: Server disk osd MB/s IOPS s01 2TB osd.7 2.587 616 s01 2TB osd.8 2.566 611 s02 2TB osd.2 2.611 622 s02 2TB osd.6 2.555 609 s03 2TB osd.0 2.519 600 s03 2TB osd.1 2.552 608 s01 4TB osd.3 3.319 791 s01 4TB osd.10 4.054 966 s02 4TB osd.4 3.884 926 s02 4TB osd.11 3.931 937 s03 4TB osd.5 3.797 905 s03 4TB osd.9 3.701 882 > > and this is when it is fragmented , first 3 are on 2TB drives OSD's and second 3 on 4TB MB/s IOPS 455 108 449 107 533 127 846 201 825 196 795 189 > > > I am curious what makes you think this is related to the 'fragmentation_rating' I did hundreds of fio tests , rados , osd benches etc and recreated OSD's many times with different number of OSD's , PG's etc and only thing that is constant is this fragmentation that happens after few days of light use and all performance in all tests follows it. -- Pozdrav , Glumac Aleksandar-Saša _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |