Re: *****SPAM***** 3 node CEPH PVE hyper-converged cluster serious fragmentation and performance loss in matter of days.

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



> Where is the rados bench before and after your problem?

Rados bench before deleting OSD's and recreating them + syncing with
fragmentation 0.89

  T1 - wr,4M   T2 = ro,seq,4M   T3 = ro,rand,4M
> Total time run          60.0405 Total time run 250.486 Total time run
> 600.463
> Total writes made       9997 Total reads made 9997 Total reads made 23.947
> Write size              4194304 Read size 4194304 Read size 4194304
> Object size             4194304 Object size 4194304 Object size 4194304
> Bandwidth (MB/sec) 666,017 Bandwidth (MB/sec) 1596,41 Bandwidth (MB/sec)
> 1595,24
> Stddev Bandwidth        24.1108 Average IOPS 399 Average IOPS 398
> Max bandwidth (MB/sec) 744 Stddev IOPS 252.166 Stddev IOPS 261.614
> Min bandwidth (MB/sec) 604 Max IOPS 446 Max IOPS 446
> Average IOPS            166 Min IOPS 350 Min IOPS 341
> Stddev IOPS             6.02769 Average Latency(s)    0.0395391 Average
> Latency(s)    0.0395782
> Max IOPS                186 Max latency(s)        0.187176 Max latency(s)
> 0.17207
> Min IOPS                151 Min latency(s)        0.0056981 Min latency(s)
> 0.00326339
> Average Latency(s)      0.0960791
> Stddev Latency(s)       0.0182781
> Max latency(s)          0.190993
> Min latency(s)          0.0284014

Rados bench after recreating and sync with fragmentation 0.1

T1   T2   T3
> Total time run          60.0143 Total time run        51.9554 Total time
> run        60.0455
> Total writes made       30868 Total reads made      30868 Total reads made
> 33853
> Write size              4194304 Read size             4194304 Read size
> 4194304
> Object size             4194304 Object size           4194304 Object size
> 4194304
> Bandwidth (MB/sec) 2057,38 Bandwidth (MB/sec) 2376,5 Bandwidth (MB/sec)
> 2255,16
> Stddev Bandwidth 121.141 Average IOPS 594 Average IOPS 563
> Max bandwidth (MB/sec)  2208 Stddev IOPS           27.1142 Stddev IOPS
> 23.7633
> Min bandwidth (MB/sec)  1472 Max IOPS              641 Max IOPS
> 616
> Average IOPS            514 Min IOPS              543 Min IOPS
> 500
> Stddev IOPS             30.2852 Average Latency(s)    0.026446 Average
> Latency(s)    0.0278983
> Max IOPS                552 Max latency(s)        0.120386 Max latency(s)
> 0.13513
> Min IOPS                368 Min latency(s)        0.00436071 Min
> latency(s)        0.00267677
> Average Latency(s)      0.0310978
> Stddev Latency(s)       0.0120903
> Max latency(s)          0.127144
> Min latency(s)          0.00719787
>



> To me this looks like normal sequential write performance to an ssd.

This is normal write for these OSD's when they are not fragmented and
already have data , osd bench:

Server disk osd MB/s IOPS
s01 2TB osd.7 2.587 616
s01 2TB osd.8 2.566 611
s02 2TB osd.2 2.611 622
s02 2TB osd.6 2.555 609
s03 2TB osd.0 2.519 600
s03 2TB osd.1 2.552 608

s01 4TB osd.3 3.319 791
s01 4TB osd.10 4.054 966
s02 4TB osd.4 3.884 926
s02 4TB osd.11 3.931 937
s03 4TB osd.5 3.797 905
s03 4TB osd.9 3.701 882
>
>


and this is when it is fragmented , first 3 are on 2TB drives OSD's and
second 3 on 4TB

MB/s IOPS
455 108
449 107
533 127
846 201
825 196
795 189
>
>


> I am curious what makes you think this is related to the
'fragmentation_rating'

I did hundreds of fio tests , rados , osd benches etc and recreated OSD's
many times with different number of OSD's , PG's etc and only thing that is
constant is this fragmentation that happens after few days of light use and
all performance in all tests follows it.



-- 

Pozdrav ,
Glumac Aleksandar-Saša
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux