> -----Original Message----- > From: Gandalf Corvotempesta [mailto:gandalf.corvotempesta@xxxxxxxxx] > Sent: 15 June 2016 22:13 > To: nick@xxxxxxxxxx > Cc: ceph-users@xxxxxxxx > Subject: Re: Switches and latency > > 2016-06-15 22:59 GMT+02:00 Nick Fisk <nick@xxxxxxxxxx>: > > Possibly, but by how much? 20GB of bandwidth is a lot to feed 12x7.2k > disks, particularly if they start doing any sort of non-sequential IO. > > Assuming 100MB/s for each SATA disk, 12 disks are 1200MB/s = 9600mbit/s > Why are you talking about 20Gb/s ? By using VLANs on the same port for > both public and cluster traffic, i'll have 10Gb/s to share, but all disks can > saturate the whole nic (9600mbit/s on a 10000mbit/s network) So this is probably a very optimistic figure, any sort of non 4MB sequential workload will rapidly decrease this number, are you planning on using SSD journals, this will impact the possible bandwidth you will achieve. I was assuming each node has 2 Nic's in a Bond going to separate switch. You get 20Gb/s of bandwidth and redundancy. > > I can't aggregate 2 ports, or I have to buy stackable switches with support for > LAG across both switches, much more expansive. > And obviously I can't use only one switch. Network must be fault tollerance. As above, check out the linux bonding options. ALB mode gives both RX and TX load balancing, although I think it may have some weird fringe cases you need to test before going live with it. > > > I think you want to try and keep it simple as possible and make the right > decision 1st time round. Buy a TOR switch that will accommodate the number > of servers you wish to put in your rack and you should never have a need to > change it. > > > > I think there are issues when one of networks is down and not the other, > so stick to keeping each server terminating into the same switch for all its > connections, otherwise you are just inviting trouble to happen. > > This is not good. a network could fail. In a HA cluster, network failure must be > taken in consideration. > What I would like to do is to unplug cable from switch 1 and plug to switch 2. a > couple of seconds max. (obviously switch2 will be temporary connected to > switch1) _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |