Hi Hannes, our production environment is running on 10GB infrastructure. We had a lot of troubles till we got to where we are today. We use Intel X520 D2 cards on our OSD´s and nexus switch infrastructure. All other cards we where testing failed horrible. Some of the problems we encountered have been: - page allocation failures in the ixgbe driver --> fixed in upstream - problems with jumbo frames, we had to disable tso, gro, lro -- > this is the most obscure thing - various tuning via sysctl in the net.tcp and net.ipv4 area --> this was also the outcome of stefan´s benchmarking odysee. But after all this we a quite happy actully and are only limited by the speed of the drives (2TB SATA). The fsync is a fdatasync in fact which is available in newer glibc. If you dont use btrfs (we use xfs) you need to use a recent glibc with fdatasync support. On Mon, Jun 4, 2012 at 10:02 AM, Hannes Reinecke <hare@xxxxxxx> wrote: hope this helps Greetings Stefan > On 06/04/2012 09:53 AM, Stefan Priebe - Profihost AG wrote: >> Am 04.06.2012 09:39, schrieb Hannes Reinecke: >>> On 06/04/2012 09:26 AM, Stefan Priebe - Profihost AG wrote: >>>> Am 04.06.2012 08:22, schrieb Hannes Reinecke: >>>>> Hehe. >>>>> Good luck with that. >>>>> >>>>> We've tried on 10GigE with _disastrous_ results. >>>>> Up to the point where 1GigE was actually _faster_. >>>> >>>> So you mean you've tried 10GBE or 10GB ipoib with Infiniband? >> >> Could you please answer this question too? Thx. >> > This was plain 10GigE, ie TCP/IP. Not infiniband, I'm afraid. > > However, given that our problems have not been related to the actual > transport I'd be very much surprised if they would not occur on > Infiniband. > > And I would _definitely_ like to hear if someone managed to get any > decent speed (notably write speed) on fast interconnects. > There's always a chance we've messed things up and were just > measuring our crap setup ... > > Cheers, > > Hannes > -- > Dr. Hannes Reinecke zSeries & Storage > hare@xxxxxxx +49 911 74053 688 > SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg > GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg) > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html -- Stefan Majer -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
