Ricardo, Thought to share my testing results. I've been using IPoIB with ceph for quite some time now. I've got QDR osd/mon/client servers to serve rbd images to kvm hypervisor. I've done some performance testing using both rados and guest vm benchmarks while running the last three stable versions of ceph. My conclusion was that ceph itself needs to mature and/or be optimised in order to utilise the capabilities of the infiniband link. In my experience, I was not able to reach the limits of the network speeds reported to me by the network performance monitoring tools. I was struggling to push data throughput beyond 1.5GB/s while using between 2 and 64 concurrent tests. This was the case when the benchmark data was using the same data over and over again and the data was cached on the osd servers and was coming directly from server's ram without any access to the osds themselves. My ipoib network performance tests were showing on average 2.5-3GB/s with peaks reaching 3.3GB/s over ipoib. It would be nice to see how ceph is performing over rdma ))). Having said this, perhaps my test gear is somewhat limited or my ceph optimisation was not done correctly. I had 2 osd servers with 8 osds each and three clients running guest vms and rados benchmarks. None of the benchmarks were able to fully utilise the server resources. my osd servers were running on about 50% utilisation during the tests. So, I had to conclude that unless you are running a large cluster with some specific data sets that utilise multithreading you will probably not need to have an infiniband link. A single thread performance for the cold data will be limited to about 1/2 of the speed of a single osd device. So, if your osds are running 150MB/s do not expect to have a single thread faster than 70-80MB/s. On the other hand, if you utilise high performance gear, like cache cards capable of achieving speeds of over gigabytes per second, perhaps infiniband link might be of use. Not sure if the ceph-osd process is capable of "spitting" out this amount of data though. You might be having a CPU bottleneck. Andrei ----- Original Message ----- From: "Sage Weil" <sweil@xxxxxxxxxx> To: "Riccardo Murri" <riccardo.murri at uzh.ch> Cc: ceph-users at lists.ceph.com Sent: Tuesday, 22 July, 2014 9:42:56 PM Subject: Re: Ceph and Infiniband On Tue, 22 Jul 2014, Riccardo Murri wrote: > Hello, > > a few questions on Ceph's current support for Infiniband > > (A) Can Ceph use Infiniband's native protocol stack, or must it use > IP-over-IB? Google finds a couple of entries in the Ceph wiki related > to native IB support (see [1], [2]), but none of them seems finished > and there is no timeline. > > [1]: https://wiki.ceph.com/Planning/Blueprints/Emperor/msgr%3A_implement_infiniband_support_via_rsockets > [2]: http://wiki.ceph.com/Planning/Blueprints/Giant/Accelio_RDMA_Messenger This is work in progress. We hope to get basic support into the tree in the next couple of months. > (B) Can we connect to the same Ceph cluster from Infiniband *and* > Ethernet? Some clients do only have Ethernet and will not be > upgraded, some others would have QDR Infiniband -- we would like both > sets to access the same storage cluster. This is further out. Very early refactoring to make this work in wip-addr. > (C) I found this old thread about Ceph's performance on 10GbE and > Infiniband: are the issues reported there still current? > > http://comments.gmane.org/gmane.comp.file-systems.ceph.devel/6816 No idea! :) sage > > > Thanks for any hint! > > Riccardo > > -- > Riccardo Murri > http://www.s3it.uzh.ch/about/team/ > > S3IT: Services and Support for Science IT > University of Zurich > Winterthurerstrasse 190, CH-8057 Z?rich (Switzerland) > Tel: +41 44 635 4222 > Fax: +41 44 635 6888 > _______________________________________________ > ceph-users mailing list > ceph-users at lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ ceph-users mailing list ceph-users at lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20140723/47bde040/attachment.htm>
 |