Great, now regarding FRR no, the truth is that I never had experience before, is there any specific guide to frr-ceph configuration or any advice? Actually, we have for the public network a bond created with two 10GbE link, so we can break the bond and get two 10GbE fibers per node for the public network On Tue, Jul 6, 2021 at 10:27 AM Serkan Çoban <cobanserkan@xxxxxxxxx> wrote: > Without using FRR you are still L2 below ToR. Do you have any > experience with FRR? > You should think about multiple uplinks and how you handle them with FRR. > Other than that, it should work, we have been in production with BGP > above ToR for years without any problem. > > On Tue, Jul 6, 2021 at 4:10 PM German Anders <yodasbunker@xxxxxxxxx> > wrote: > > > > Hello Stefan, thanks a lot for the reply. > > > > We have everything in the same datacenter, and the fault domains are > per > > rack. Regarding BGP, the idea of the networking team is to stop using > layer > > 2+3 and move everything to full layer 3, for this they want to implement > > BGP and the idea is that each node is connected to a different ToR edge > > switch and that the clients have a single IP and can then reach the > entire > > cluster. > > Currently, we have an environment configured with a specific VLAN > > without GW, and they want to get the VLAN out of the way and that each > node > > has its own IP with its own GW (and that is the ToR Switch). We already > > have a separate cluster network that is running on Infiniband and it's > > completely separated. So the idea is to use BGP on the public network > only. > > > > Thanks in advance, > > > > Cheers, > > > > On Tue, Jul 6, 2021 at 2:10 AM Stefan Kooman <stefan@xxxxxx> wrote: > > > > > On 7/5/21 6:26 PM, German Anders wrote: > > > > Hi All, > > > > > > > > I have an already created and functional ceph cluster (latest > > > luminous > > > > release) with two networks one for the public (layer 2+3) and the > other > > > for > > > > the cluster, the public one uses VLAN and its 10GbE and the other one > > > uses > > > > Infiniband with 56Gb/s, the cluster works ok. The public network uses > > > > Juniper QFX5100 switches with VLAN in layer2+3 configuration but the > > > > network team needs to move to a full layer3 and they want to use > BGP, so > > > > the question is, how can we move to that schema? What are the > > > > considerations? Is it possible? Is there any step-by-step way to > move to > > > > that schema? Also is anything better than BGP or other alternatives? > > > > > > Ceph doesn't care at all. Just as long as the nodes can communicate to > > > each other, it's fine. It depends on your failure domains how easy you > > > can move to this L3 model. Do you have separate datacenters that you > can > > > do one by one, or separate racks? > > > > > > And you can do BGP on different levels: router, top of rack switches, > or > > > even on the Ceph host itselfs (FRR). > > > > > > We use BGP / VXLAN / EVPN for our Ceph cluster. But it all depends on > > > why your networking teams wants to change to L3, and why. > > > > > > There are no step by step guides, as most deployments are unique. > > > > > > This might be a good time to reconsider a separate cluster network. > > > Normally there is no need for that, and might make things simpler. > > > > > > Do you have separate storage switches? Whre are your clients connected > > > to (separate switches or connected to storage switches as well). > > > > > > This is not easy to answer without all the details. But for sure there > > > are cluster running with BGP in the field just fine. > > > > > > Gr. Stefan > > > _______________________________________________ > > > ceph-users mailing list -- ceph-users@xxxxxxx > > > To unsubscribe send an email to ceph-users-leave@xxxxxxx > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users@xxxxxxx > > To unsubscribe send an email to ceph-users-leave@xxxxxxx > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |