Jake Grimmett wrote:
On Friday 14 March 2008 10:43, Kay Diederichs wrote:
Jake Grimmett schrieb:
If I could ask question about 10Gbit ethernet....
We have a 70 node cluster built on Centos 5.1, using NFS to mount user
home areas. At the moment the network is a bottleneck, and it's going to
get worse as we add another 112 CPUs in the form of two blade servers.
To help things breath better, we are considering building three new NFS
servers, and connecting these and the blade servers directly via 10Gbit
to a new core switch, the older nodes will stay with 1Gbit ethernet, but
be given new switches uplinked via 10Gbit to the core switch.
Before I spend a great deal of money, can I ask if anyone here has
experience of 10GBit? Dell, HP, Supermicro, IBM (etc..) seem to be
pushing the NetXen PCIe cards, so I guess these drivers work...? As to
media, although CX4 seems cheaper than optical, I hear the cabling is
nasty. And is the magical fairy going to fix 10Gbit over cat6A anytime
soon?
any thoughts appreciated.
Jake
Jake,
please note that the latest RedHat/CentOS-5.1 kernels have NFS
performance issues - if these are (partly) responsible for the
bottleneck then you may want to install a kernel which fixes the issue.
Check out RedHat bugzilla 321111.
HTH,
Kay
_______________________________________________
CentOS mailing list
CentOS@xxxxxxxxxx
http://lists.centos.org/mailman/listinfo/centos
Many thanks for your answers :)
I will certainly talk to the beowulf crowd, and I think your right that our
bottleneck's are *partly* caused by NFS issues.
I'm probably not going to feed 10Gb to the individual blades, as I have few
MPI sers, though it's an option with IBM and HP blades. However IBM, and Dell
offer a 10Gbit XFP uplink to the blade servers internal switch, and this has
to be worthwhile with 56 CPUs on the other side of it.
I'm most concerned about whether anyone has tried the Netxen or Chelsio 10Gbit
NICs on Centos 5.1; I see drivers in /lib/modules for these...
Also - do people have good / bad experiences of CX4 cabling? As an economical
short range solution (<15M) it seems ideal for a server room, but I have a
sales rep who is trying to scare me off, but he is biased as the 10Gb SR XFP
transceivers are very expensive (~£820)...
Many thanks
Jake
_______________________________________________
CentOS mailing list
CentOS@xxxxxxxxxx
http://lists.centos.org/mailman/listinfo/centos
CX4 cabling is ok if you like playing with garden hoses. It refuses to
go into cable arms and hasn't got much in the way of flexibility.
The cards are cheaper than fibre however the cables are expensive.
Personally I think that fibre is the way forward with this.
I have used NetXen cards under RH and they seem fine.
_______________________________________________
CentOS mailing list
CentOS@xxxxxxxxxx
http://lists.centos.org/mailman/listinfo/centos
