Hi Robert,
Another option is if both RDMA ports are on the same card, then you can do RDMA with a bond. This does not work if you have two separate cards.
Yes, we recently talked to Mellanox and their engineers also recommend this way. As far as your questions go, my guess would be that you would want to have the different NICs in different broadcast domains
yes, the idea was to use two public networks on two different NICs with addresses from different subnets. It is possible to set 2+ networks in Ceph configuration, but it’s unclear how Ceph is going to use this configuration.
or set up Source Based Routing and bind the source port on the connection (not the easiest, but allows you to have multiple NICs in the same broadcast domain). I don't have experience with Ceph in this type of configuration.
it’s too complicated and, frankly, when you’re trying to reach max performance, Source Based Routing is a bit from another area :-)
At the end of the all, we’re going to test bonding of two ports on same NIC.
Thank you.
--
Volodymyr Litovka
"Vision without Execution is Hallucination." -- Thomas Edison
Dear colleagues,
at the moment, we use Ceph in routed environment (OSPF, ECMP) and
everything is ok, reliability is high and there is nothing to
complain about. But for hardware reasons (to be more precise - RDMA
offload), we are faced with the need to operate Ceph directly on
physical interfaces.
According to documentation, "W e generally recommend that dual-NIC
systems either be configured with two IPs on the same network, or
bonded."
Q1: Did anybody test and can explain, how Ceph will behave in
first scenario (two IPs on the same network)? I think this
configuration require just one statement in 'public network'
(where both interfaces reside)? How it will distribute traffic
between links, how it will detect link failures and how it will
switchover?
Q2: Did anybody test a bit another scenario - both NICs have
addresses in different networks and Ceph configuration contain two
'public networks'? Questions are same - how Ceph distributes
traffic between links and how it recovers from link failures?
Thank you.
--
Volodymyr Litovka
"Vision without Execution is Hallucination." -- Thomas Edison
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
|
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
