Infiniband uses multicast internally. It's not something you have a choice with. You won't see it on the local interface any more than you'd see individual drives of a raid 5.
I believe it's one of the reasons the connection setup speeds are kept under the requisite 1.2usec limits etc.
On Jun 10, 2016 4:16 AM, "Daniel Swarbrick" <daniel.swarbrick@xxxxxxxxxxxxxxxx> wrote:
On 10/06/16 02:33, Christian Balzer wrote:I'd be interested to see some concrete proof of that. We run several IB
>
>
> This thread brings back memories of this one:
> http://lists.ceph.com/pipermail/ceph-users-ceph.com/2016-April/008792.html
>
> According to Robert IPoIB still uses IB multicast under the hood even when
> from an IP perspective traffic would be unicast.
fabrics here using Mellanox QDR HCAs, and run a mixture of SRP and IPoIB
over them. We don't explicitly override the mcast rate, so it's safe to
assume that this is the default SDR, 10 Gbps rate.
Testing with iperf3, I've seen single flow IPoIB (CM) reach about 20
Gbps, and multiple flows top out at around a combined 25 Gbps.
On the other hand, testing with ib_write_bw (RDMA, single "flow"), we
usually get just under 30 Gbps. So there is a fair bit of overhead in
IPoIB, but I'm skeptical that it uses mcast IB all the time. Nothing in
the Linux IPoIB kernel modules stands out as looking like "use multicast
for everything."
>
> The biggest issue pointed out in that mail and the immensely long
> and complex thread he mentioned in it is that you can't change the speed
> settings on the fly.
> which means that if you're already in production it's unlikely that there
> will ever be a time to entirely tear down your IB network...
>
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
