Re: RBD performance slowly degrades :-(

[Irek Fasikhov <malmyzh@xxxxxxxxx>]

Hi.

Read this thread here:

https://www.mail-archive.com/ceph-users@xxxxxxxxxxxxxx/msg17360.html

С уважением, Фасихов Ирек Нургаязович

Моб.: +79229045757

2015-08-12 14:52 GMT+03:00 Pieter Koorts <pieter.koorts@xxxxxx>:

Hi

Something that's been bugging me for a while is I am trying to diagnose iowait time within KVM guests. Guests doing reads or writes tend do about 50% to 90% iowait but the host itself is only doing about 1% to 2% iowait. So the result is the guests are extremely slow.

I currently run 3x hosts each with a single SSD and single HDD OSD in cache-teir writeback mode. Although the SSD (Samsung 850 EVO 120GB) is not a great one it should at least perform reasonably compared to a hard disk and doing some direct SSD tests I get approximately 100MB/s write and 200MB/s read on each SSD.

When I run rados bench though, the benchmark starts with a not great but okay speed and as the benchmark progresses it just gets slower and slower till it's worse than a USB hard drive. The SSD cache pool is 120GB in size (360GB RAW) and in use at about 90GB. I have tried tuning the XFS mount options as well but it has had little effect.

Understandably the server spec is not great but I don't expect performance to be that bad.

OSD config:

[osd]

osd crush update on start = false

osd mount options xfs = "rw,noatime,inode64,logbsize=256k,delaylog,allocsize=4M"

Servers spec:

Dual Quad Core XEON E5410 and 32GB RAM in each server

10GBE @ 10G speed with 8000byte Jumbo Frames.

Rados bench result: (starts at 50MB/s average and plummets down to 11MB/s)

sudo rados bench -p rbd 50 write --no-cleanup -t 1

 Maintaining 1 concurrent writes of 4194304 bytes for up to 50 seconds or 0 objects

 Object prefix: benchmark_data_osc-mgmt-1_10007

   sec Cur ops   started  finished  avg MB/s  cur MB/s  last lat   avg lat

     0       0         0         0         0         0         -         0

     1       1        14        13   51.9906        52 0.0671911  0.074661

     2       1        27        26   51.9908        52 0.0631836 0.0751152

     3       1        37        36   47.9921        40 0.0691167 0.0802425

     4       1        51        50   49.9922        56 0.0816432 0.0795869

     5       1        56        55   43.9934        20  0.208393  0.088523

     6       1        61        60    39.994        20  0.241164 0.0999179

     7       1        64        63   35.9934        12  0.239001  0.106577

     8       1        66        65   32.4942         8  0.214354  0.122767

     9       1        72        71     31.55        24  0.132588  0.125438

    10       1        77        76   30.3948        20  0.256474  0.128548

    11       1        79        78   28.3589         8  0.183564  0.138354

    12       1        82        81   26.9956        12  0.345809  0.145523

    13       1        85        84    25.842        12  0.373247  0.151291

    14       1        86        85   24.2819         4  0.950586  0.160694

    15       1        86        85   22.6632         0         -  0.160694

    16       1        90        89   22.2466         8  0.204714  0.178352

    17       1        94        93   21.8791        16  0.282236  0.180571

    18       1        98        97   21.5524        16  0.262566  0.183742

    19       1       101       100   21.0495        12  0.357659  0.187477

    20       1       104       103    20.597        12  0.369327  0.192479

    21       1       105       104   19.8066         4  0.373233  0.194217

    22       1       105       104   18.9064         0         -  0.194217

    23       1       106       105   18.2582         2   2.35078  0.214756

    24       1       107       106   17.6642         4  0.680246  0.219147

    25       1       109       108   17.2776         8  0.677688  0.229222

    26       1       113       112   17.2283        16   0.29171  0.230487

    27       1       117       116   17.1828        16  0.255915  0.231101

    28       1       120       119   16.9976        12  0.412411  0.235122

    29       1       120       119   16.4115         0         -  0.235122

    30       1       120       119   15.8645         0         -  0.235122

    31       1       120       119   15.3527         0         -  0.235122

    32       1       122       121   15.1229         2  0.319309  0.262822

    33       1       124       123   14.9071         8  0.344094  0.266201

    34       1       127       126   14.8215        12   0.33534  0.267913

    35       1       129       128   14.6266         8  0.355403  0.269241

    36       1       132       131   14.5536        12  0.581528  0.274327

    37       1       132       131   14.1603         0         -  0.274327

    38       1       133       132   13.8929         2   1.43621   0.28313

    39       1       134       133   13.6392         4  0.894817  0.287729

    40       1       134       133   13.2982         0         -  0.287729

    41       1       135       134   13.0714         2   1.87878  0.299602

    42       1       138       137   13.0459        12  0.309637  0.304601

    43       1       140       139   12.9285         8  0.302935  0.304491

    44       1       141       140   12.7256         4    1.5538  0.313415

    45       1       142       141   12.5317         4  0.352417  0.313691

    46       1       145       144   12.5201        12  0.322063  0.317458

    47       1       145       144   12.2537         0         -  0.317458

    48       1       145       144   11.9984         0         -  0.317458

    49       1       145       144   11.7536         0         -  0.317458

    50       1       146       145   11.5985         1   3.79816  0.341463

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com