Just a couple of points. 1. I know you said 10G was not an option, but I would really push for it. You can pick up Dell 10G-T switches (N4032) for not a lot more than a 48 port 1G switch. They make a lot more difference than just 10x the bandwidth. With Ceph latency is critical. As its 10G-T, you can use the existing 1GB Nic's until with the new switches until you can upgrade the Nic's. 2. If you still only want 1GB, maybe something like the HP 2920's? 3. RR load balancing is probably not working well for you, due to out of order TCP packets. As the amount of traffic increases with RR, packets arrive slightly in the wrong order, which causes them to be retransmitted. This eventually snowballs and all sorts of nasty stuff starts happening. 4. As Ceph has lots of connections on lots of IP's and port's, LACP or the Linux ALB mode should work really well to balance connections. 5. Read #1 again, if you are buying a new switch anyway, don't throw good money after bad. > -----Original Message----- > From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On Behalf Of > Christian Balzer > Sent: 01 June 2016 10:16 > To: ceph-users@xxxxxxxx > Subject: Re: Best Network Switches for Redundancy > > > Hello, > > On Wed, 1 Jun 2016 11:03:16 +0200 David Riedl wrote: > > > > > > So 3 servers are the entirety of your Ceph storage nodes, right? > > Exactly. + 3 Openstack Compute Nodes > > > > > > > Have you been able to determine what causes the drops? > > > My first guess would be that this bonding is simply not compatible > > > with what the switches can do/expect. > > > > > Yeah, something like that. load balancing round robin kinda works, but > > it's a 'server side' bonding protocol. The switches don't know > > anything about that particular configuration. > > > LACP isn't round-robin, but it does distribute things in fashion and > > > given the fact that it actually works you should try it. > > > > > > To be more specific, LACP distribution is based on "sessions", so if > > > you have enough variety in there you will get something that's good > > > enough. A single session however will not be faster than an > > > individual link, IIRC. > > > > > What do you mean by 'variety'? Do you mean I/O? > > > Variety as in what those sessions (hashes) are based on. > Usually IP addresses. > So if you were to send just data over one specific TCP session from > 10.0.0.1 to 10.0.0.2 it would go over one of your interfaces only, not both. > > So on some my servers with LACP I see noticeable differences between > interface usage, especially when sending traffic is just to one other node > usually. > On others with a sufficiently large number of connections to various hosts, it > approaches uniform utilization. > > > > > > > > > > Why a single switch and thus a SPoF? > > > Or are you planning to get 2 switches and plan for more clients and > > > Ceph nodes down the road? > > Sorry I wasn't more clear. Yes, 2 48 port switches. And yes, I am > > planning to add more Ceph nodes. The backend network also runs on only > > one failover Gigabit interface right now and I'm planning to utilize > > the > > 2 remaining interfaces as well. > > > Then mLAG, mc-lag, vlag, clag is for you. > > Also consider a flat network consisting of 4 mc-LAGed interfaces instead of a > private cluster and client network. > At 4Gb/s total your local storage is still going to be most likely faster than you > network bandwidth. > > > > > > > If I were in your shoes, I'd look at 2 switches running MC-LAG (in > > > any of the happy variations there are) > > > https://en.wikipedia.org/wiki/MC-LAG > > > > > > And since you're on a budget, something like the Cumulus based > > > offerings (Penguin computing, etc). > > Thanks, I'll look into it. Never heard of that protocol before. > > > It's just LACP over multiple switches, giving you full redundancy AND > bandwidth. > > Christian > -- > Christian Balzer Network/Systems Engineer > chibi@xxxxxxx Global OnLine Japan/Rakuten Communications > http://www.gol.com/ > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |