Hi, I was wondering if there was any work in progress to get Ceph to run over RDMA transports? Our HPC clusters here, where we would like to run Ceph, all use DDR or QDR IB interconnects, and we think there are latency and bandwidth improvements to be had that would improve Ceph performance. On the server side I've been looking at SimpleMessenger and SimpleMessenger::Pipe, and the threading model they use. It seems like it would be pretty straightforward to have a RDMAPipe that works like Pipe, but holds an IB queue pair rather than a socket, and uses the same threading model as Pipe. An RDMAMessenger could hold a collection of RDMAPipe objects, but otherwise work much the same way as SimpleMessenger does now. Some recent commits for the OSD code introduced separate messengers for clients and cluster members, so it seems a natural way forward to introduce an option for RDMA support between daemons, followed by RDMA support between clients and daemons. Comments? Opinions? Also, FWIW quite a lot of work happened in 2008 for NFS-RDMA, and some of the issues they had to deal with might be pertinent. This presentation: http://openfabrics.org/archives/spring2008sonoma/Tuesday/NFS-RDMA-OFA-08.pdf has some interesting information on memory registration issues at page 20. It was mentioned in this email: http://www.spinics.net/lists/linux-nfs/msg03035.html, which is part of an interesting thread on some of the issues involved. Thanks -- Jim -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
