On 06/03/2012 02:56 PM, Mark Nelson wrote: > On 6/3/12 3:10 AM, Stefan Priebe wrote: >> Hi List, >> >> has anybody already tried CEPH over Infiniband 40GB? >> >> Stefan >> -- >> To unsubscribe from this list: send the line "unsubscribe >> ceph-devel" in >> the body of a message to majordomo@xxxxxxxxxxxxxxx >> More majordomo info at http://vger.kernel.org/majordomo-info.html > > Hi Stefan, > > A couple of folks have done DDR IB. For now you are limited to > ipoib though. If you have the hardware available I'd be really > curious what kind of throughput/latencies you see. > Hehe. Good luck with that. We've tried on 10GigE with _disastrous_ results. Up to the point where 1GigE was actually _faster_. So far we've uncovered two issues: - intel_idle was/is seriously broken (we've tried on 3.0-stable, so might've been fixed by now) - osd-server is calling 'fsync' on each and every write request. Does wonders for performance ... Cheers, Hannes -- Dr. Hannes Reinecke zSeries & Storage hare@xxxxxxx +49 911 74053 688 SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg) -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
