Hello Adrian, On Thu, 2 Jun 2016 00:53:41 +0000 Adrian Saul wrote: > > I am currently running our Ceph POC environment using dual Nexus 9372TX > 10G-T switches, each OSD host has two connections to each switch and > they are formed into a single 4 link VPC (MC-LAG), which is bonded under > LACP on the host side. > > What I have noticed is that the various hashing policies for LACP do not > guarantee you will make full use of all the links. I tried various > policies and from what I could see the normal L3+L4 IP and port hashing > generally worked as good as anything else, but if you have lots of > similar connections it doesn't seem to hash across all the links and say > 2 will be heavily used while not much is hashed onto the other links. > This might have just been because it was a fairly small pool of IPs and > fairly similar port numbers that just happened to keep hashing to the > same links (I ended up going to the point of tcpdumping traffic and > scripting a calculation of what link it should use, it just happened to > be so consistent). > > For two links it should be quite good - it seemed to balance across that > quite well, but with 4 links it seemed to really prefer 2 in my case. > Just for the record, did you also change the LACP policies on the switches? >From what I gather, having fancy pants L3+4 hashing on the Linux side will not fix imbalances by itself, the switches need to be configured likewise. Christian > > > -----Original Message----- > > From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On Behalf > > Of David Riedl > > Sent: Thursday, 2 June 2016 2:12 AM > > To: ceph-users@xxxxxxxxxxxxxx > > Subject: Re: Best Network Switches for Redundancy > > > > > > > 4. As Ceph has lots of connections on lots of IP's and port's, LACP > > > or the Linux ALB mode should work really well to balance connections. > > Linux ALB Mode looks promising. Does that work with two switches? Each > > server has 4 ports which are 'splitted' and connected to each switch. > > _ > > / _[switch] > > / / || > > [server] || > > \ \_ || > > \__[switch] > > _______________________________________________ > > ceph-users mailing list > > ceph-users@xxxxxxxxxxxxxx > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > Confidentiality: This email and any attachments are confidential and may > be subject to copyright, legal or some other professional privilege. > They are intended solely for the attention and use of the named > addressee(s). They may only be copied, distributed or disclosed with the > consent of the copyright owner. If you have received this email by > mistake or by breach of the confidentiality clause, please notify the > sender immediately by return email and delete or destroy all copies of > the email. Any confidentiality, privilege or copyright is not waived or > lost because this email has been sent to you by mistake. > _______________________________________________ ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > -- Christian Balzer Network/Systems Engineer chibi@xxxxxxx Global OnLine Japan/Rakuten Communications http://www.gol.com/ _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |