On Jan 17, 2013, at 8:37 AM, Mark Nelson <mark.nelson@xxxxxxxxxxx> wrote: > On 01/17/2013 07:32 AM, Joseph Glanville wrote: >> On 17 January 2013 20:46, Gandalf Corvotempesta >> <gandalf.corvotempesta@xxxxxxxxx> wrote: >>> 2013/1/16 Mark Nelson<mark.nelson@xxxxxxxxxxx>: >> >>> I don't know if I have to use a single two port IB card (switch >>> redundancy and no card redundancy) or >>> I have to use two single port cards. (or a single one port IB?) >> >> On the topic of IB.. >> >> But slightly off-topic all the same.. I would love to attempt getting >> Ceph running on rsockets if I could find the time (alas we don't run >> Ceph). >> rsockets is a fully userland implementation of BSD sockets over RDMA, >> supporting fork and all the usual goodies, in theory unless you are >> using the kernel RBD module (of the kernel FS module etc) you should >> be able to run it on rsockets and enjoy a considerable performance >> increase. >> >> rsockets is available in the librdmacm git up on Open Fabrics and dev >> + support happens on the linux-rdma list. >> > > There's been some talk about rsockets on the list before. I think there > are a couple of different folks that have tried (succeeded?) in getting > it working. barring that, it sounds like if you tune interrupt affinity > settings and various other bits you can get IPoIB up into the 2GB/s+ > range which while not RDMA speed, is at least better than 10GbE. IB DDR should get you close to 2 GB/s with IPoIB. I have gotten our IB QDR PCI-E Gen. 2 up to 2.8 GB/s measured via netperf with lots of tuning. Since it uses the traditional socket stack through the kernel, CPU usage will be as high (or higher if QDR) than 10GbE. I would be interested in seeing if rsockets helps ceph. I have questions though. By default, rsockets still has to copy data in and out. It has extensions for zero-copy, but they do not work with non-blocking sockets. Does ceph use non-blocking sockets? How many simultaneous connections does it support before falling over into the non-rsockets path? If rsockets has a ceiling on connections and falls over to the non-socket path, can the application determine if it is safe to use the zero-copy extensions for a specific connection or do they fail gracefully? Scott-- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
