On Thu, Jan 17, 2013 at 7:00 PM, Atchley, Scott <atchleyes@xxxxxxxx> wrote: > On Jan 17, 2013, at 9:48 AM, Gandalf Corvotempesta <gandalf.corvotempesta@xxxxxxxxx> wrote: > >> 2013/1/17 Atchley, Scott <atchleyes@xxxxxxxx>: >>> IB DDR should get you close to 2 GB/s with IPoIB. I have gotten our IB QDR PCI-E Gen. 2 up to 2.8 GB/s measured via netperf with lots of tuning. Since it uses the traditional socket stack through the kernel, CPU usage will be as high (or higher if QDR) than 10GbE. >> >> Which kind of tuning? Do you have a paper about this? > > No, I followed the Mellanox tuning guide and modified their interrupt affinity scripts. Did you tried to bind interrupts only to core to which QPI link belongs in reality and measure difference with spread-over-all-cores binding? > >> But, actually, is possible to use ceph with IPoIB in a stable way or >> is this experimental ? > > IPoIB appears as a traditional Ethernet device to Linux and can be used as such. Not exactly, this summer kernel added additional driver for fully featured L2(ib ethernet driver), before that it was quite painful to do any possible failover using ipoib. > >> I don't know if i support for rsocket that is experimental/untested >> and IPoIB is a stable workaroud or what else. > > IPoIB is much more used and pretty stable, while rsockets is new with limited testing. That said, more people using it will help Sean improve it. > > Ideally, we would like support for zero-copy and reduced CPU usage (via OS-bypass) and with more interconnects than just InfiniBand. :-) > >> And is a dual controller needed on each OSD node? Ceph is able to >> handle OSD network failures? This is really important to know. It >> change the whole network topology. > > I will let others answer this. > > Scott-- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
