> -----Original Message----- > From: Luis Periquito [mailto:periquito@xxxxxxxxx] > Sent: 06 June 2016 14:30 > To: Nick Fisk <nick@xxxxxxxxxx> > Cc: Ceph Users <ceph-users@xxxxxxxxxxxxxx> > Subject: Re: OSPF to the host > > Nick, > > TL;DR: works brilliantly :) Excellent, just what I wanted to hear!!! > > Where I work we have all of the ceph nodes (and a lot of other stuff) using > OSPF and BGP server attachment. With that we're able to implement > solutions like Anycast addresses, removing the need to add load balancers, > for the radosgw solution. > > The biggest issues we've had were around the per-flow vs per-packets traffic > load balancing, but as long as you keep it simple you shouldn't have any > issues. > > Currently we have a P2P network between the servers and the ToR switches > on a /31 subnet, and then create a virtual loopback address, which is the > interface we use for all communications. Running tests like iperf we're able > to reach 19Gbps (on a 2x10Gbps network). OTOH we no longer have the > ability to separate traffic between public and osd network, but never really > felt the need for it. Yeah, I've come to pretty much the same conclusion. > > Also spend a bit of time planning how the network will look like and it's > topology. If done properly (think details like route summarization) then it's > really worth the extra effort. How are you doing the route summarization with OSPF? Is each rack (for example) a separate OSPF area, which you then summarize and send up to area 0? > > > > On Mon, Jun 6, 2016 at 11:57 AM, Nick Fisk <mailto:nick@xxxxxxxxxx> wrote: > Hi All, > > Has anybody had any experience with running the network routed down all > the way to the host? > > I know the standard way most people configured their OSD nodes is to bond > the two nics which will then talk via a VRRP gateway and then probably from > then on the networking is all Layer3. The main disadvantage I see here is that > you need a beefy inter switch link to cope with the amount of traffic flowing > between switches to the VRRP address. I’ve been trying to design around > this by splitting hosts into groups with different VRRP gateways on either > switch, but this relies on using active/passive bonding on the OSD hosts to > make sure traffic goes from the correct Nic to the directly connected switch. > > What I was thinking, instead of terminating the Layer3 part of the network at > the access switches, terminate it at the hosts. If each Nic of the OSD host had > a different subnet and the actual “OSD Server” address bound to a loopback > adapter, OSPF should advertise this loopback adapter address as reachable > via the two L3 links on the physically attached Nic’s. This should give you a > redundant topology which also will respect your physically layout and > potentially give you higher performance due to ECMP. > > Any thoughts, any pitfalls? > > > > _______________________________________________ > ceph-users mailing list > mailto:ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |