Re: tgt / rbd performance

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi,

 

Can you post the commands you ran for both benchmarks, without knowing the block size, write pattern and queue depth, it’s hard to determine where the bottleneck might be.

 

I can see OSD sde has a very high service time, which could be a sign of a problem, does it always show up high during tests, or is this just a one off spike?

 

The service times as a whole also look slightly high, I’m assuming there is a write back cache on the HP controller?

 

It might also be worth checking the max queue depths of both target and initiator to make sure you are not hitting any limits.

 

Nick

 

From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On Behalf Of ano nym
Sent: 11 December 2014 17:40
To: ceph-users@xxxxxxxxxxxxxx
Subject: tgt / rbd performance

 

 

there is a ceph pool on a hp dl360g5 with 25 sas 10k (sda-sdy) on a msa70 which gives me about 600 MB/s continous write speed with rados write bench. tgt on the server with rbd backend uses this pool. mounting local(host) with iscsiadm, sdz is the virtual iscsi device. As you can see, sdz max out with 100%util at ~55MB/s when writing to it. 

 

I know that tgt-rbd is more a proof-of-concept then production-ready. 

 

Anyway, is someone using it and/or are there any hints to speed it up?

 

avg-cpu:  %user   %nice %system %iowait  %steal   %idle

          18.31    0.00   26.01   53.28    0.00    2.40

 

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util

sda               0.00     5.00    0.00   35.00     0.00  8316.50   475.23     2.36   68.74    0.00   68.74  15.11  52.90

sdb               0.00     0.00    0.00   24.00     0.00  3718.50   309.88     0.87   49.54    0.00   49.54  14.96  35.90

sdc               0.00     9.00    0.00   64.00     0.00 19739.00   616.84     3.33   51.12    0.00   51.12   9.20  58.90

sdd               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00

sde               0.00     0.00    0.00    2.00     0.00   520.00   520.00     0.66   86.00    0.00   86.00  78.00  15.60

sdf               0.00     8.00    0.00  103.00     0.00 34638.00   672.58     7.45   78.69    0.00   78.69   9.46  97.40

sdg               0.00     3.00    0.00   74.00     0.00 25472.00   688.43     4.47   60.50    0.00   60.50   9.62  71.20

sdh               0.00     0.00    0.00   17.00     0.00  5760.00   677.65     0.62   45.65    0.00   45.65  14.29  24.30

sdi               0.00     3.00    0.00   43.00     0.00 14724.00   684.84     2.92   67.91    0.00   67.91  12.98  55.80

sdj               0.00     4.00    1.00   45.00     4.00 14152.00   615.48     3.18   58.52   19.00   59.40  13.59  62.50

sdk               0.00     6.00    0.00   75.00     0.00 29047.50   774.60     6.35   61.08    0.00   61.08  10.23  76.70

sdm               0.00     3.00    0.00   48.00     0.00 16488.00   687.00     1.75   36.40    0.00   36.40   9.08  43.60

sdl               0.00     1.00    0.00   46.00     0.00 17412.00   757.04     4.94  108.63    0.00  108.63   9.48  43.60

sdn               0.00     0.00    0.00   51.00     0.00 17692.00   693.80     4.49   88.10    0.00   88.10   9.90  50.50

sdo               0.00     8.00    0.00   55.00     0.00 13574.00   493.60     3.44   62.89    0.00   62.89   8.55  47.00

sdp               0.00     0.00    0.00   40.00     0.00 14488.00   724.40     2.82   69.72    0.00   69.72  16.88  67.50

sdq               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00

sdr               0.00     1.00    0.00   32.00     0.00  8856.00   553.50     0.78   24.22    0.00   24.22   9.53  30.50

sds               0.00     7.00    1.00   51.00     4.00 16132.50   620.63     3.14   58.98   79.00   58.59  16.48  85.70

sdt               0.00     2.00    0.00   50.00     0.00 19040.00   761.60     3.56   71.12    0.00   71.12   9.54  47.70

sdu               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00

sdw               0.00     0.00    0.00   16.00     0.00  6388.00   798.50     0.95   77.62    0.00   77.62  11.56  18.50

sdx               0.00     1.00    0.00   28.00     0.00  9840.00   702.86     1.40   49.86    0.00   49.86  12.43  34.80

sdy               0.00     2.00    0.00   54.00     0.00 20168.00   746.96     5.64   72.65    0.00   72.65   9.72  52.50

sdv               0.00     0.00    0.00   15.00     0.00  4300.00   573.33     0.49   33.07    0.00   33.07  19.00  28.50

 

sdz               0.00     0.00    0.00  115.00     0.00 57468.00   999.44   143.50  736.48    0.00  736.48   8.70 100.00

 

b.r. Michael


_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux