在 2022/10/24 22:35, Dust Li 写道:
On Mon, Oct 24, 2022 at 09:12:56PM +0800, Yanjun Zhu wrote:
在 2022/10/24 19:52, Dust Li 写道:
On Mon, Oct 24, 2022 at 06:15:01AM +0000, yanjun.zhu@xxxxxxxxx wrote:
October 24, 2022 9:10 AM, "Dust Li" <dust.li@xxxxxxxxxxxxxxxxx> wrote:
On Sun, Oct 23, 2022 at 06:04:47PM -0400, Zhu Yanjun wrote:
From: Zhu Yanjun <yanjun.zhu@xxxxxxxxx>
There are shared and exclusive modes in RDMA net namespace. After
discussion with Leon, the above modes are compatible with legacy IB
device.
To the RoCE and iWARP devices, the ib devices should be in the same net
namespace with the related net devices regardless of in shared or
exclusive mode.
Does this mean that shared mode is no longer supported for RoCE and iWarp
devices ?
>From the discussion, a RoCE and iWarp device should make ib devices and net devices in the same net. So a RoCE and iWarp device has no shared/exclusive modes.
Shared/exclusive modes are for legacy ib devices, such as ipoib.
In this patch series, shared/exclusive modes are left for legacy ib devices.
To a RoCE and iWarp device, we just keep net devices and ib devices in the same net.
I think this may limit the use case of RoCE and iWarp.
See the following use case:
In the container enviroment, we may have lots of containers on a host,
for example, more than 100. And we don't have that much VFs, so we use
ipvlan or other virtual network devices for each container, and put
those virtual network devices into each container(net namespace).
Since we only use 1 physical network device for all those containers,
there is only one RoCE device. If we don't support shared mode, we
cannot even enable RDMA for those containers with RoCE.
You use the ipvlan or other virtual network devices for each container.
In these containers, you also use RDMA, correct?
Since all the packets for these virtual network devices finally come to
the physical network devices, without shared/exclusive modes, it should work.
So we do not consider the shared/exclusive mode.
For the netdevice, that's true. But for RDMA, we should not even see
the ib device in the containers any more, so I think we cannot create
qp/cq, and RDMA is not available for these containers in this case.
I can not get you.
Do you mean that RDMA can not be accessed in the container after these
patches are applied?
Can you share a test case with me?
Zhu Yanjun
Thanks
Zhu Yanjun
I don't know any other way to solve this, maybe I missed something ?
Thanks
In the first commit, when the net devices are moved to a new net
namespace, the related ib devices are also moved to the same net
namespace.
In the second commit, the shared/exclusive modes still work with legacy
ib devices. To the RoCE and iWARP devices, these modes will not be
considered.
Because MLX4/5 do not call the function ib_device_set_netdev to map ib
devices and the related net devices, the function ib_device_get_by_netdev
can not get ib devices from net devices. In the third commit, all the
registered ib devices are parsed to get the net devices, then compared
with the given net devices.
The steps to make tests:
1) Create a new net namespace net0
ip netns add net0
2) Show the rdma links in init_net
rdma link
"
link mlx5_0/1 state DOWN physical_state DISABLED netdev enp7s0np1
"
3) Move the net device to net namespace net0
ip link set enp7s0np1 netns net0
4) Show the rdma links in init_net again
rdma link
There is no rdma links
5) Show the rdma links in net0
ip netns exec net0 rdma link
"
link mlx5_0/1 state DOWN physical_state DISABLED netdev enp7s0np1
"
We can confirm that rdma links are moved to the same net namespace with
the net devices.
Zhu Yanjun (3):
RDMA/core: Move ib device to the same net namespace with net device
RDMA/core: The legacy IB devices still work with shared/exclusive mode
RDMA/core: Get all the ib devices from net devices
drivers/infiniband/core/device.c | 107 ++++++++++++++++++++++++++++++-
1 file changed, 105 insertions(+), 2 deletions(-)
--
2.27.0
