On 05/06/2013 09:18 AM, Jens Kristian Søgaard wrote:
Hi,
I'm just supposing, i'm also evaluating the same network topology
like you.
I have tested my current setup for a while with just triple-bonded GbE,
and haven't found the need for more features. I might be wrong ofcourse :-)
There is a lowcost. 12x 10GbaseT switch from Netgear.
I was looking at the Netgear XS708E which is very low cost compared to
what the prices were 6 months ago when I looked at it last.
I'm really not up to speed on the capabilities of cheap 10GbE switches.
I'm sure other people will have comments about what features are
worthwhile, etc. Just from a raw performance perspective, you might
want to make sure that the switch can handle lots of randomly
distributed traffic between all of the ports well. I'd expect that it
shouldn't be too terrible, but who knows. I expect most IB switches you
run into should deal with this kind of pattern competently enough for
Ceph workloads to do OK. On the front end portion of the network you'll
always have client<->server communication, so the pattern there will be
less all-to-all than the backend traffic (more 1st half <-> 2nd half).
On really big deployments, static routing becomes a pretty big issue.
Dynamic routing, or at least well optimized routes can make a huge
difference. ORNL has done some work in this area for their Lustre IB
networks and I think LLNL is investigating it as well.
Mark
What I don't know is how those prices compare to IB pricing.
on cluster network and IB to public network). Do you think will be
possible to use all ceph services (RBD, RGW, CephFS, QEMU, ...) via
SDP/rsockets ?
I haven't tried SDP/rsocket, so I don't know. From what I have
experienced in the past, these things doesn't seem to be completely
"drop in and forget about it" - there's always something that is not
completely compatible with ordinary TCP/IP sockets.
On ebay there are many IB switches sold at 1000-1500$ with 24 or 36
DDR ports. If used with SDP/rsockets you should archieve more or less
18Gbps
That sounds cheap yes, but I assume that is used equipment?
I was looking at costs for new equipment.
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
