help to tune ceph

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi to all,

I need your help to understand how to tune my ceph configuration to achieve some performance result.

my installation is built as follow:

5 server with 16gb ram and 8 core
5 client (same machine)

Each computer is connected with a 1gb/s ethernet to the same switch.
The device storage export 20 luns with 7.2TB of capacity.

Every server is connected to a device storace with a FC connection at 4Gb/s  and export 2 osds.

I always has a healthy ceph cluster up and running before start any test.

I have tried 3 different configuration of storage to achieve the best result.

The test for the performance is always the same: 5 dd from /dev/zero to a file the dimension of the dd is 25GB, each dd is run from a different client concurrently and the result file is stored intoceph posix mounted folder.

1) first configuration 
every luns is been partitioned with data and journal, every /var/lib/caph/osd/[osd folder]/journal  was linked to the partition journal (on the same lun).

In this configuration I saw that the san storage was stormed by a lot of iops and the performance was very slow, I investigated and I discovered that a possible reason could be the journal of every osd,
http://pastebin.com/GdkngHTd

so i decide to change the configuration as follow.

2) second 

Every server use 4 luns one for data and one for journal, changing the link of journal dir into the var/lib/caph/osd/[osd fold]/journal. Obviously I have flushed and recreated journal before start the cluster again.

again this configuration was not optimal and the performance was a little better from the previous experiment but again very far from the nominal possibility of the storage... i haven't ssds where to put the journal... so I decided to change the configuration again.
http://pastebin.com/VhmAt7Sf

3) third
Every server use a ram disk for journal.

I thought that since it is a test configuration I could use a ram disk that is the fastest device that I have and put there the journal... so i did. I launch again the same test... but the performance is still the same...

http://pastebin.com/tdAdGGnM


What am I doing wrong?

Could anyone help me to understand what I can do to achieve the best performance from this device?

ps. this device with the gpfs filesystem reached 500/600 MB/s in write and 600/700MB/s in read with ceph only 250/300 in almost all configurations...

pps. i know that with 5 servers and 5 client I will never reach the same performance but I will expect to saturate the ethernet bandwidth.

thanks a lot 
Matteo


ceph.conf: http://pastebin.com/ZQsA1i7A
ceph -w during last configuration test: http://pastebin.com/zzJ6atj1
ceph-osd tree: http://pastebin.com/UxzqmSN8


speed during test inboud from server: http://pastebin.com/d8FT8UZE
speed during test outbound from client: http://pastebin.com/2W1DWaPP







---
Matteo Favaro
mail: matteo.favaro (at) cnaf.infn.it
alias: mfavaro (at) cnaf.infn.it

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20140502/1c91cedc/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 1756 bytes
Desc: not available
URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20140502/1c91cedc/attachment.bin>


[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux