I'd suggest running your Ceph network on the "public" interface /24 you have assigned to that rack. Assign loopback IP's from another (not otherwise routed) network to your MONs.
I have no experience with the RBD client, but with RGW you can assign the same IP to all of them and let your ToR switches do ECMP to all of them. Built in fault-tolerance, no hokey DNS failover necessary, no HA proxy, etc. Just straight ToR ECMP. Like I said though, no experience with RBD but it ace with RGW.
Example:
TOR /24 -> 192.168.0.0/24
Servers assigned eth0 public interface -> 192.168.0.2-254 (or whatnot)
MON nodes assigned lo0 IP -> 196.168.50.1/32
You mentioned BGP, but you can use something lighter weight like OSPF inside the rack to advertise the 192.168.50.1 IP from each server into the ToR switch routing table (which is then announced to the fabric over BGP).
-richard
On Tue, Apr 18, 2017 at 9:30 AM, Jan Marquardt <jm@xxxxxxxxxxx> wrote:
Am 17.04.17 um 22:12 schrieb Richard Hesse:
> A couple of questions:
>
> 1) What is your rack topology? Are all ceph nodes in the same rack
> communicating with the same top of rack switch?
The cluster is planned for one rack with two ToR-/cluster-internal
switches. The cluster will be accessed from 2-3 machines mounted into
the same rack, which have uplinks to the outside world.
> 2) Why did you choose to run the ceph nodes on loopback interfaces as
> opposed to the /24 for the "public" interface?
The fabric needs loopback ip addresses and the plan was/is to use them
directly for Ceph. What would you suggest instead?
> 3) Are you planning on using RGW at all?
No, there won't be any RGW. It is a plain rbd cluster, which will be
used for backup purposes.
Best Regards
Jan
> On Thu, Apr 13, 2017 at 10:57 AM, Jan Marquardt <jm@xxxxxxxxxxx
> <mailto:jm@xxxxxxxxxxx>> wrote:
>
> Hi,
>
> I am currently working on Ceph with an underlying Clos IP fabric and I
> am hitting some issues.
>
> The setup looks as follows: There are 3 Ceph nodes which are running
> OSDs and MONs. Each server has one /32 loopback ip, which it announces
> via BGP to its uplink switches. Besides the loopback ip each server has
> an management interface with a public (not to be confused with ceph's
> public network) ip address. For BGP switches and servers are running
> quagga/frr.
>
> Loopback ips:
>
> 10.10.100.1 # switch1
> 10.10.100.2 # switch2
> 10.10.100.21 # server1
> 10.10.100.22 # server2
> 10.10.100.23 # server3
>
> Ceph's public network is 10.10.100.0/24 <http://10.10.100.0/24>.
> E-Mail: support@xxxxxxxxxxx <mailto:support@xxxxxxxxxxx> | Web:>
> Here comes the current main problem: There are two options for
> configuring the loopback address.
>
> 1.) Configure it on lo. In this case the routing works as inteded, but,
> as far as I found out, Ceph can not be run on lo interface.
>
> root@server1:~# ip route get 10.10.100.22
> 10.10.100.22 via 169.254.0.1 dev enp4s0f1 src 10.10.100.21
> cache
>
> 2.) Configure it on dummy0. In this case Ceph is able to start, but
> quagga installs learned routes with wrong source addresses - the public
> management address from each host. This results in network problems,
> because Ceph uses the management ips to communicate to the other Ceph
> servers.
>
> root@server1:~# ip route get 10.10.100.22
> 10.10.100.22 via 169.254.0.1 dev enp4s0f1 src a.b.c.d
> cache
>
> (where a.b.c.d is the machine's public ip address on its management
> interface)
>
> Has already someone done something similar?
>
> Please let me know, if you need any further information. Any help would
> really be appreciated.
>
> Best Regards
>
> Jan
>
> --
> Artfiles New Media GmbH | Zirkusweg 1 | 20359 Hamburg
> Tel: 040 - 32 02 72 90 | Fax: 040 - 32 02 72 95
> http://www.artfiles.de
> Geschäftsführer: Harald Oltmanns | Tim Evers
> Eingetragen im Handelsregister Hamburg - HRB 81478
>
>
> _______________________________________________ > ceph-users@xxxxxxxxxxxxxx <mailto:ceph-users@xxxxxxxxxx.
> ceph-users mailing list
com >
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph. com
> <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph. >com
>
>
--
Artfiles New Media GmbH | Zirkusweg 1 | 20359 Hamburg
Tel: 040 - 32 02 72 90 | Fax: 040 - 32 02 72 95
E-Mail: support@xxxxxxxxxxx | Web: http://www.artfiles.de
Geschäftsführer: Harald Oltmanns | Tim Evers
Eingetragen im Handelsregister Hamburg - HRB 81478
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
