Re: Performance question

[Alan Johnson <alanj@xxxxxxxxxxxxxx>]

Or separate the journals as this will bring the workload down on the spinners to 3Xrather than 6X

 

From: Marek Dohojda [mailto:mdohojda@xxxxxxxxxxxxxxxxxxx]
Sent: Tuesday, November 24, 2015 1:24 PM
To: Nick Fisk
Cc: Alan Johnson; ceph-users@xxxxxxxxxxxxxx
Subject: Re: [ceph-users] Performance question

 

Crad I think you are 100% correct:

 

rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util

 

 0.00   369.00   33.00 1405.00   132.00 135656.00   188.86     5.61    4.02   21.94    3.60   0.70 100.00

 

I was kinda wondering that this maybe the case, which is why I was wondering if I should be doing too much in terms of troubleshooting.

 

So basically what you are saying I need to wait for new version?

 

 

Thank you very much everybody!

 

 

On Tue, Nov 24, 2015 at 9:35 AM, Nick Fisk <nick@xxxxxxxxxx> wrote:

You haven’t stated what size replication you are running. Keep in mind that with a replication factor of 3, you will be writing 6x the amount of data down to disks than what the benchmark says (3x replication x2 for data+journal write).

 

You might actually be near the hardware maximums. What does iostat looks like whilst you are running rados bench, are the disks getting maxed out?

 

From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On Behalf Of Marek Dohojda
Sent: 24 November 2015 16:27
To: Alan Johnson <alanj@xxxxxxxxxxxxxx>

Cc: ceph-users@xxxxxxxxxxxxxx
Subject: Re: [ceph-users] Performance question

 

7 total servers, 20 GIG pipe between servers, both reads and writes.  The network itself has plenty of pipe left, it is averaging 40Mbits/s 

 

Rados Bench SAS 30 writes

 Total time run:         30.591927

Total writes made:      386

Write size:             4194304

Bandwidth (MB/sec):     50.471 

 

Stddev Bandwidth:       48.1052

Max bandwidth (MB/sec): 160

Min bandwidth (MB/sec): 0

Average Latency:        1.25908

Stddev Latency:         2.62018

Max latency:            21.2809

Min latency:            0.029227

 

Rados Bench SSD writes

 Total time run:         20.425192

Total writes made:      1405

Write size:             4194304

Bandwidth (MB/sec):     275.150 

 

Stddev Bandwidth:       122.565

Max bandwidth (MB/sec): 576

Min bandwidth (MB/sec): 0

Average Latency:        0.231803

Stddev Latency:         0.190978

Max latency:            0.981022

Min latency:            0.0265421

 

 

As you can see SSD is better but not as much as I would expect SSD to be. 

 

 

 

On Tue, Nov 24, 2015 at 9:10 AM, Alan Johnson <alanj@xxxxxxxxxxxxxx> wrote:

Hard to know without more config details such as no of servers, network  – GigE or !0 GigE, also not sure how you are measuring, (reads or writes) you could try RADOS bench as a baseline, I would expect more performance with 7 X 10K spinners journaled to SSDs. The fact that SSDs did not perform much better may mean to a bottleneck elsewhere – network perhaps?

From: Marek Dohojda [mailto:mdohojda@xxxxxxxxxxxxxxxxxxx]
Sent: Tuesday, November 24, 2015 10:37 AM
To: Alan Johnson
Cc: Haomai Wang; ceph-users@xxxxxxxxxxxxxx

Subject: Re: [ceph-users] Performance question

 

Yeah they are, that is one thing I was planning on changing, What I am really interested at the moment, is vague expected performance.  I mean is 100MB around normal, very low, or "could be better"?

 

On Tue, Nov 24, 2015 at 8:02 AM, Alan Johnson <alanj@xxxxxxxxxxxxxx> wrote:

Are the journals on the same device – it might be better to use the SSDs for journaling since you are not getting better performance with SSDs?

 

From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On Behalf Of Marek Dohojda
Sent: Monday, November 23, 2015 10:24 PM
To: Haomai Wang
Cc: ceph-users@xxxxxxxxxxxxxx
Subject: Re: [ceph-users] Performance question

 

 Sorry I should have specified SAS is the 100 MB :) , but to be honest SSD isn't much faster.

 

On Mon, Nov 23, 2015 at 7:38 PM, Haomai Wang <haomaiwang@xxxxxxxxx> wrote:

On Tue, Nov 24, 2015 at 10:35 AM, Marek Dohojda
<mdohojda@xxxxxxxxxxxxxxxxxxx> wrote:
> No SSD and SAS are in two separate pools.
>
> On Mon, Nov 23, 2015 at 7:30 PM, Haomai Wang <haomaiwang@xxxxxxxxx> wrote:
>>
>> On Tue, Nov 24, 2015 at 10:23 AM, Marek Dohojda
>> <mdohojda@xxxxxxxxxxxxxxxxxxx> wrote:
>> > I have a Hammer Ceph cluster on 7 nodes with total 14 OSDs.  7 of which
>> > are
>> > SSD and 7 of which are SAS 10K drives.  I get typically about 100MB IO
>> > rates
>> > on this cluster.

So which pool you get with 100 MB?

>>
>> You mixed up sas and ssd in one pool?
>>
>> >
>> > I have a simple question.  Is 100MB within my configuration what I
>> > should
>> > expect, or should it be higher? I am not sure if I should be looking for
>> > issues, or just accept what I have.
>> >
>> > _______________________________________________
>> > ceph-users mailing list
>> > ceph-users@xxxxxxxxxxxxxx

>> > http://xo4t.mj.am/link/xo4t/rsxjit1/1/NlEqhua2rOHxmXdiOCL_wA/aHR0cDovL2xpc3RzLmNlcGguY29tL2xpc3RpbmZvLmNnaS9jZXBoLXVzZXJzLWNlcGguY29t
>> >
>>
>>
>>
>> --
>> Best Regards,
>>
>> Wheat
>
>

--
Best Regards,

Wheat

 

 

 

 

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com