On Jan 17, 2013, at 10:07 AM, Andrey Korolyov <andrey@xxxxxxx> wrote: > On Thu, Jan 17, 2013 at 7:00 PM, Atchley, Scott <atchleyes@xxxxxxxx> wrote: >> On Jan 17, 2013, at 9:48 AM, Gandalf Corvotempesta <gandalf.corvotempesta@xxxxxxxxx> wrote: >> >>> 2013/1/17 Atchley, Scott <atchleyes@xxxxxxxx>: >>>> IB DDR should get you close to 2 GB/s with IPoIB. I have gotten our IB QDR PCI-E Gen. 2 up to 2.8 GB/s measured via netperf with lots of tuning. Since it uses the traditional socket stack through the kernel, CPU usage will be as high (or higher if QDR) than 10GbE. >>> >>> Which kind of tuning? Do you have a paper about this? >> >> No, I followed the Mellanox tuning guide and modified their interrupt affinity scripts. > > Did you tried to bind interrupts only to core to which QPI link > belongs in reality and measure difference with spread-over-all-cores > binding? This is the modified part. I bound the mlx4-async handler to core 0 and the mlx4-ib-1-0 handle to core 1 for our machines. >>> But, actually, is possible to use ceph with IPoIB in a stable way or >>> is this experimental ? >> >> IPoIB appears as a traditional Ethernet device to Linux and can be used as such. > > Not exactly, this summer kernel added additional driver for fully > featured L2(ib ethernet driver), before that it was quite painful to > do any possible failover using ipoib. I assume it is now an EoIB driver. Does it replace the IPoIB driver? >>> I don't know if i support for rsocket that is experimental/untested >>> and IPoIB is a stable workaroud or what else. >> >> IPoIB is much more used and pretty stable, while rsockets is new with limited testing. That said, more people using it will help Sean improve it. >> >> Ideally, we would like support for zero-copy and reduced CPU usage (via OS-bypass) and with more interconnects than just InfiniBand. :-) >> >>> And is a dual controller needed on each OSD node? Ceph is able to >>> handle OSD network failures? This is really important to know. It >>> change the whole network topology. >> >> I will let others answer this. >> >> Scott-- >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >> the body of a message to majordomo@xxxxxxxxxxxxxxx >> More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
