> From: Mark Lehrer <lehrer@xxxxxxxxx>
> Sent: Thursday, April 13, 2023 11:39 AM
>
> > Didn’t get a chance to review the thread discussion.
> > The way to use VF is:
>
> Virtual functions were just a debugging aid. We really just want to use a single
> physical function and put it into the netns. However, we will do additional VF
> tests as it still may be a viable workaround.
>
> When using the physical function, we are still having no joy using exclusive
> mode with mlx5:
>
static int nvmet_rdma_enable_port(struct nvmet_rdma_port *port)
{
struct sockaddr *addr = (struct sockaddr *)&port->addr;
struct rdma_cm_id *cm_id;
int ret;
cm_id = rdma_create_id(&init_net, nvmet_rdma_cm_handler, port,
^^^^^^^
Nvme target is not net ns aware.
RDMA_PS_TCP, IB_QPT_RC);
if (IS_ERR(cm_id)) {
pr_err("CM ID creation failed\n");
return PTR_ERR(cm_id);
}
>
> # nvme discover -t rdma -a 192.168.42.11 -s 4420 Discovery Log Number of
> Records 2, Generation counter 2 =====Discovery Log Entry 0====== ... (works
> as expected)
>
> # rdma system set netns exclusive
> # ip netns add netnstest
> # ip link set eth1 netns netnstest
> # rdma dev set mlx5_0 netns netnstest
> # nsenter --net=/var/run/netns/netnstest /bin/bash # ip link set eth1 up # ip
> addr add 192.168.42.12/24 dev eth1 (tested ib_send_bw here, works perfectly)
>
> # nvme discover -t rdma -a 192.168.42.11 -s 4420 Failed to write to /dev/nvme-
> fabrics: Connection reset by peer failed to add controller, error Unknown error
> -1
>
> # dmesg | tail -3
> [ 240.361647] mlx5_core 0000:05:00.0 eth1: Link up [ 240.371772] IPv6:
> ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready [ 259.964542] nvme
> nvme0: rdma connection establishment failed (-104)
>
> Am I missing something here?
>
> Thanks,
> Mark
>
>
> On Thu, Apr 13, 2023 at 7:05 AM Parav Pandit <parav@xxxxxxxxxx> wrote:
> >
> >
> >
> > > From: Mark Lehrer <lehrer@xxxxxxxxx>
> > > Sent: Thursday, April 13, 2023 9:01 AM
> > >
> > > > Do you make tests nvme + mlx5 + net ns in your host? Can it work?
> > >
> > > Sort of, but not really. In our last test, we configured a virtual
> > > function and put it in the netns context, but also configured a
> > > physical function outside the netns context. TCP NVMe connections always
> used the correct interface.
> > >
> > Didn’t get a chance to review the thread discussion.
> > The way to use VF is:
> >
> > 1. rdma system in exclusive mode
> > $ rdma system set netns exclusive
> >
> > 2. Move netdevice of the VF to the net ns $ ip link set [ DEV ] netns
> > NSNAME
> >
> > 3. Move RDMA device of the VF to the net ns $ rdma dev set [ DEV ]
> > netns NSNAME
> >
> > You are probably missing #1 and #3 configuration.
> > #1 should be done before creating any namespaces.
> >
> > Man pages for #1 and #3:
> > [a] https://man7.org/linux/man-pages/man8/rdma-system.8.html
> > [b] https://man7.org/linux/man-pages/man8/rdma-dev.8.html
> >
> > > However, the RoCEv2 NVMe connection always used the physical
> > > function, regardless of the user space netns context of the nvme-cli process.
> > > When we ran "ip link set <physical function> down" the RoCEv2 NVMe
> > > connections stopped working, but TCP NVMe connections were fine.
> > > We'll be doing more tests today to make sure we're not doing
> > > something wrong.
> > >
> > > Thanks,
> > > Mark
> > >
> > >
> > >
> > >
> > > On Thu, Apr 13, 2023 at 7:22 AM Zhu Yanjun <yanjun.zhu@xxxxxxxxx>
> wrote:
> > > >
> > > >
> > > > 在 2023/4/13 5:01, Mark Lehrer 写道:
> > > > >> the fabrics device and writing the host NQN etc. Is there an
> > > > >> easy way to prove that rdma_resolve_addr is working from userland?
> > > > > Actually I meant "is there a way to prove that the kernel
> > > > > rdma_resolve_addr() works with netns?"
> > > >
> > > > I think rdma_resolve_addr can work with netns because rdma on mlx5
> > > > can work well with netns.
> > > >
> > > > I do not delve into the source code. But IMO, this function should
> > > > be used in rdma on mlx5.
> > > >
> > > > >
> > > > > It seems like this is the real problem. If we run commands like
> > > > > nvme discover & nvme connect within the netns context, the
> > > > > system will use the non-netns IP & RDMA stacks to connect. As
> > > > > an aside - this seems like it would be a major security issue
> > > > > for container systems, doesn't it?
> > > >
> > > > Do you make tests nvme + mlx5 + net ns in your host? Can it work?
> > > >
> > > > Thanks
> > > >
> > > > Zhu Yanjun
> > > >
> > > > >
> > > > > I'll investigate to see if the fabrics module & nvme-cli have a
> > > > > way to set and use the proper netns context.
> > > > >
> > > > > Thanks,
> > > > > Mark
