On Mon, Oct 24, 2022 at 06:15:01AM +0000, yanjun.zhu@xxxxxxxxx wrote: >October 24, 2022 9:10 AM, "Dust Li" <dust.li@xxxxxxxxxxxxxxxxx> wrote: > >> On Sun, Oct 23, 2022 at 06:04:47PM -0400, Zhu Yanjun wrote: >> >>> From: Zhu Yanjun <yanjun.zhu@xxxxxxxxx> >>> >>> There are shared and exclusive modes in RDMA net namespace. After >>> discussion with Leon, the above modes are compatible with legacy IB >>> device. >>> >>> To the RoCE and iWARP devices, the ib devices should be in the same net >>> namespace with the related net devices regardless of in shared or >>> exclusive mode. >> >> Does this mean that shared mode is no longer supported for RoCE and iWarp >> devices ? > >From the discussion, a RoCE and iWarp device should make ib devices and net devices in the same net. So a RoCE and iWarp device has no shared/exclusive modes. > >Shared/exclusive modes are for legacy ib devices, such as ipoib. > >In this patch series, shared/exclusive modes are left for legacy ib devices. >To a RoCE and iWarp device, we just keep net devices and ib devices in the same net. I think this may limit the use case of RoCE and iWarp. See the following use case: In the container enviroment, we may have lots of containers on a host, for example, more than 100. And we don't have that much VFs, so we use ipvlan or other virtual network devices for each container, and put those virtual network devices into each container(net namespace). Since we only use 1 physical network device for all those containers, there is only one RoCE device. If we don't support shared mode, we cannot even enable RDMA for those containers with RoCE. I don't know any other way to solve this, maybe I missed something ? Thanks > > > >> >>> In the first commit, when the net devices are moved to a new net >>> namespace, the related ib devices are also moved to the same net >>> namespace. >>> >>> In the second commit, the shared/exclusive modes still work with legacy >>> ib devices. To the RoCE and iWARP devices, these modes will not be >>> considered. >>> >>> Because MLX4/5 do not call the function ib_device_set_netdev to map ib >>> devices and the related net devices, the function ib_device_get_by_netdev >>> can not get ib devices from net devices. In the third commit, all the >>> registered ib devices are parsed to get the net devices, then compared >>> with the given net devices. >>> >>> The steps to make tests: >>> 1) Create a new net namespace net0 >>> >>> ip netns add net0 >>> >>> 2) Show the rdma links in init_net >>> >>> rdma link >>> >>> " >>> link mlx5_0/1 state DOWN physical_state DISABLED netdev enp7s0np1 >>> " >>> >>> 3) Move the net device to net namespace net0 >>> >>> ip link set enp7s0np1 netns net0 >>> >>> 4) Show the rdma links in init_net again >>> >>> rdma link >>> >>> There is no rdma links >>> >>> 5) Show the rdma links in net0 >>> >>> ip netns exec net0 rdma link >>> >>> " >>> link mlx5_0/1 state DOWN physical_state DISABLED netdev enp7s0np1 >>> " >>> >>> We can confirm that rdma links are moved to the same net namespace with >>> the net devices. >>> >>> Zhu Yanjun (3): >>> RDMA/core: Move ib device to the same net namespace with net device >>> RDMA/core: The legacy IB devices still work with shared/exclusive mode >>> RDMA/core: Get all the ib devices from net devices >>> >>> drivers/infiniband/core/device.c | 107 ++++++++++++++++++++++++++++++- >>> 1 file changed, 105 insertions(+), 2 deletions(-) >>> >>> -- >>> 2.27.0
