Hi to all, I need your help to understand how to tune my ceph configuration to achieve some performance result. my installation is built as follow: 5 server with 16gb ram and 8 core 5 client (same machine) Each computer is connected with a 1gb/s ethernet to the same switch. The device storage export 20 luns with 7.2TB of capacity. Every server is connected to a device storace with a FC connection at 4Gb/s and export 2 osds. I always has a healthy ceph cluster up and running before start any test. I have tried 3 different configuration of storage to achieve the best result. The test for the performance is always the same: 5 dd from /dev/zero to a file the dimension of the dd is 25GB, each dd is run from a different client concurrently and the result file is stored intoceph posix mounted folder. 1) first configuration every luns is been partitioned with data and journal, every /var/lib/caph/osd/[osd folder]/journal was linked to the partition journal (on the same lun). In this configuration I saw that the san storage was stormed by a lot of iops and the performance was very slow, I investigated and I discovered that a possible reason could be the journal of every osd, http://pastebin.com/GdkngHTd so i decide to change the configuration as follow. 2) second Every server use 4 luns one for data and one for journal, changing the link of journal dir into the var/lib/caph/osd/[osd fold]/journal. Obviously I have flushed and recreated journal before start the cluster again. again this configuration was not optimal and the performance was a little better from the previous experiment but again very far from the nominal possibility of the storage... i haven't ssds where to put the journal... so I decided to change the configuration again. http://pastebin.com/VhmAt7Sf 3) third Every server use a ram disk for journal. I thought that since it is a test configuration I could use a ram disk that is the fastest device that I have and put there the journal... so i did. I launch again the same test... but the performance is still the same... http://pastebin.com/tdAdGGnM What am I doing wrong? Could anyone help me to understand what I can do to achieve the best performance from this device? ps. this device with the gpfs filesystem reached 500/600 MB/s in write and 600/700MB/s in read with ceph only 250/300 in almost all configurations... pps. i know that with 5 servers and 5 client I will never reach the same performance but I will expect to saturate the ethernet bandwidth. thanks a lot Matteo ceph.conf: http://pastebin.com/ZQsA1i7A ceph -w during last configuration test: http://pastebin.com/zzJ6atj1 ceph-osd tree: http://pastebin.com/UxzqmSN8 speed during test inboud from server: http://pastebin.com/d8FT8UZE speed during test outbound from client: http://pastebin.com/2W1DWaPP --- Matteo Favaro mail: matteo.favaro (at) cnaf.infn.it alias: mfavaro (at) cnaf.infn.it -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20140502/1c91cedc/attachment.htm> -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 1756 bytes Desc: not available URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20140502/1c91cedc/attachment.bin>
 |