On 1/7/22 10:24 PM, Jason Gunthorpe wrote:
On Thu, Dec 23, 2021 at 08:59:14PM +0800, Cheng Xu wrote:
On 12/23/21 6:23 PM, Leon Romanovsky wrote:
On Wed, Dec 22, 2021 at 11:35:44AM +0800, Cheng Xu wrote:
<...>
For the ECS instance with RDMA enabled, there are two kinds of devices
allocated, one for ERDMA, and one for the original netdev (virtio-net).
They are different PCI deivces. ERDMA driver can get the information about
which netdev attached to in its PCIe barspace (by MAC address matching).
This is very questionable. The netdev part should be kept in the
drivers/ethernet/... part of the kernel.
Thanks
The net device used in Alibaba ECS instance is virtio-net device, driven
by virtio-pci/virtio-net drivers. ERDMA device does not need its own net
device, and will be attached to an existed virtio-net device. The
relationship between ibdev and netdev in erdma is similar to siw/rxe.
siw/rxe binds through RDMA_NLDEV_CMD_NEWLINK netlink command and not
through MAC's matching.
Thanks
Both siw/rxe/erdma don't need to implement netdev part, this is what I
wanted to express when I said 'similar'.
What you mentioned (the bind mechanism) is one major difference between
erdma and siw/rxe. For siw/rxe, user can attach ibdev to every netdev if
he/she wants, but it is not true for erdma. When user buys the erdma
service, he/she must specify which ENI (elastic network interface) to be
binded, it means that the attached erdma device can only be binded to
the specific netdev. Due to the uniqueness of MAC address in our ECS
instance, we use the MAC address as the identification, then the driver
knows which netdev should be binded to.
It really doesn't match our driver binding model to rely on MAC
addreses.
Our standard model would expect that the virtio-net driver would
detect it has RDMA capability and spawn an aux device to link the two
things together.
Using net notifiers to try to link the lifecycles together has been a
mess so far.
Thanks for your explanation.
I guess this model requires the netdev and its associated ibdev share
the same physical hardware (pci device or platform device)? ERDMA is a
separated pci device. Only because that ENIs in our cloud are
virtio-net devices, and we let ERDMA binded to virtio-net. Actually it
also can work with other type of netdev.
As you and Leon said, using net notifiers is not a good way. And I'm
modifying our bind mechanism, using RDMA_NLDEV_CMD_NEWLINK to fix it.
Thanks,
Cheng Xu
Jason
