> Didn’t get a chance to review the thread discussion. > The way to use VF is: Virtual functions were just a debugging aid. We really just want to use a single physical function and put it into the netns. However, we will do additional VF tests as it still may be a viable workaround. When using the physical function, we are still having no joy using exclusive mode with mlx5: # nvme discover -t rdma -a 192.168.42.11 -s 4420 Discovery Log Number of Records 2, Generation counter 2 =====Discovery Log Entry 0====== ... (works as expected) # rdma system set netns exclusive # ip netns add netnstest # ip link set eth1 netns netnstest # rdma dev set mlx5_0 netns netnstest # nsenter --net=/var/run/netns/netnstest /bin/bash # ip link set eth1 up # ip addr add 192.168.42.12/24 dev eth1 (tested ib_send_bw here, works perfectly) # nvme discover -t rdma -a 192.168.42.11 -s 4420 Failed to write to /dev/nvme-fabrics: Connection reset by peer failed to add controller, error Unknown error -1 # dmesg | tail -3 [ 240.361647] mlx5_core 0000:05:00.0 eth1: Link up [ 240.371772] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready [ 259.964542] nvme nvme0: rdma connection establishment failed (-104) Am I missing something here? Thanks, Mark On Thu, Apr 13, 2023 at 7:05 AM Parav Pandit <parav@xxxxxxxxxx> wrote: > > > > > From: Mark Lehrer <lehrer@xxxxxxxxx> > > Sent: Thursday, April 13, 2023 9:01 AM > > > > > Do you make tests nvme + mlx5 + net ns in your host? Can it work? > > > > Sort of, but not really. In our last test, we configured a virtual function and put > > it in the netns context, but also configured a physical function outside the netns > > context. TCP NVMe connections always used the correct interface. > > > Didn’t get a chance to review the thread discussion. > The way to use VF is: > > 1. rdma system in exclusive mode > $ rdma system set netns exclusive > > 2. Move netdevice of the VF to the net ns > $ ip link set [ DEV ] netns NSNAME > > 3. Move RDMA device of the VF to the net ns > $ rdma dev set [ DEV ] netns NSNAME > > You are probably missing #1 and #3 configuration. > #1 should be done before creating any namespaces. > > Man pages for #1 and #3: > [a] https://man7.org/linux/man-pages/man8/rdma-system.8.html > [b] https://man7.org/linux/man-pages/man8/rdma-dev.8.html > > > However, the RoCEv2 NVMe connection always used the physical function, > > regardless of the user space netns context of the nvme-cli process. > > When we ran "ip link set <physical function> down" the RoCEv2 NVMe > > connections stopped working, but TCP NVMe connections were fine. > > We'll be doing more tests today to make sure we're not doing something > > wrong. > > > > Thanks, > > Mark > > > > > > > > > > On Thu, Apr 13, 2023 at 7:22 AM Zhu Yanjun <yanjun.zhu@xxxxxxxxx> wrote: > > > > > > > > > 在 2023/4/13 5:01, Mark Lehrer 写道: > > > >> the fabrics device and writing the host NQN etc. Is there an easy > > > >> way to prove that rdma_resolve_addr is working from userland? > > > > Actually I meant "is there a way to prove that the kernel > > > > rdma_resolve_addr() works with netns?" > > > > > > I think rdma_resolve_addr can work with netns because rdma on mlx5 can > > > work well with netns. > > > > > > I do not delve into the source code. But IMO, this function should be > > > used in rdma on mlx5. > > > > > > > > > > > It seems like this is the real problem. If we run commands like > > > > nvme discover & nvme connect within the netns context, the system > > > > will use the non-netns IP & RDMA stacks to connect. As an aside - > > > > this seems like it would be a major security issue for container > > > > systems, doesn't it? > > > > > > Do you make tests nvme + mlx5 + net ns in your host? Can it work? > > > > > > Thanks > > > > > > Zhu Yanjun > > > > > > > > > > > I'll investigate to see if the fabrics module & nvme-cli have a way > > > > to set and use the proper netns context. > > > > > > > > Thanks, > > > > Mark
