> From: Mark Lehrer <lehrer@xxxxxxxxx> > Sent: Thursday, April 13, 2023 9:01 AM > > > Do you make tests nvme + mlx5 + net ns in your host? Can it work? > > Sort of, but not really. In our last test, we configured a virtual function and put > it in the netns context, but also configured a physical function outside the netns > context. TCP NVMe connections always used the correct interface. > Didn’t get a chance to review the thread discussion. The way to use VF is: 1. rdma system in exclusive mode $ rdma system set netns exclusive 2. Move netdevice of the VF to the net ns $ ip link set [ DEV ] netns NSNAME 3. Move RDMA device of the VF to the net ns $ rdma dev set [ DEV ] netns NSNAME You are probably missing #1 and #3 configuration. #1 should be done before creating any namespaces. Man pages for #1 and #3: [a] https://man7.org/linux/man-pages/man8/rdma-system.8.html [b] https://man7.org/linux/man-pages/man8/rdma-dev.8.html > However, the RoCEv2 NVMe connection always used the physical function, > regardless of the user space netns context of the nvme-cli process. > When we ran "ip link set <physical function> down" the RoCEv2 NVMe > connections stopped working, but TCP NVMe connections were fine. > We'll be doing more tests today to make sure we're not doing something > wrong. > > Thanks, > Mark > > > > > On Thu, Apr 13, 2023 at 7:22 AM Zhu Yanjun <yanjun.zhu@xxxxxxxxx> wrote: > > > > > > 在 2023/4/13 5:01, Mark Lehrer 写道: > > >> the fabrics device and writing the host NQN etc. Is there an easy > > >> way to prove that rdma_resolve_addr is working from userland? > > > Actually I meant "is there a way to prove that the kernel > > > rdma_resolve_addr() works with netns?" > > > > I think rdma_resolve_addr can work with netns because rdma on mlx5 can > > work well with netns. > > > > I do not delve into the source code. But IMO, this function should be > > used in rdma on mlx5. > > > > > > > > It seems like this is the real problem. If we run commands like > > > nvme discover & nvme connect within the netns context, the system > > > will use the non-netns IP & RDMA stacks to connect. As an aside - > > > this seems like it would be a major security issue for container > > > systems, doesn't it? > > > > Do you make tests nvme + mlx5 + net ns in your host? Can it work? > > > > Thanks > > > > Zhu Yanjun > > > > > > > > I'll investigate to see if the fabrics module & nvme-cli have a way > > > to set and use the proper netns context. > > > > > > Thanks, > > > Mark
