Re: [PATCH 0/4] RDMA device net namespace support for SMC

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Mon, 2022-02-21 at 14:54 +0800, Tony Lu wrote:
> On Thu, Feb 17, 2022 at 12:33:06PM +0100, Niklas Schnelle wrote:
> > On Tue, 2021-12-28 at 21:06 +0800, Tony Lu wrote:
> > 
---8<---
> > Hi Tony,
> > 
> > I'm having a bit of trouble getting this to work for me and was
> > wondering if you could test my scenario or help me figure out what's
> > wrong.
> > 
> > I'm using network namespacing to be able to test traffic between two
> > VFs of the same card/port with a single Linux system. By having one VF
> > in each of a client and server namespace, traffic doesn't shortcut via
> > loopback. This works great for TCP and with "rdma system set netns
> > exclusive" I can also verify that RDMA with "qperf -cm1 ... rc_bw" only
> > works once the respective RDMA device is also added to each namespace.
> > 
> > When I try the same with SMC-R I tried:
> > 
> >   ip netns exec server smc_run qperf &
> >   ip netns exec client smc_run qperf <ip_server> tcp_bw
> > 
> > With that however I only see fallback TCP connections in "ip netns exec
> > client watch smc_dbg". It doesn't seem to be an "smc_dbg" problem
> > either since the performance with and without smc_run is the same. I
> > also do have the same PNET_ID set on the interfaces.
> 
> Hi Niklas,
> 
> I understood your problem. This connection falls back to TCP for unknown
> reasons. You can find out the fallback reason of this connection. It can
> help us find out the root cause of fallbacks. For example,
> if SMC_CLC_DECL_MEM (0x01010000) is occurred in this connection, it
> means that there is no enough memory (smc_init_info, sndbuf, RMB,
> proposal buf, clc msg).

Regarding fallback reason. It seems to be that the RDMA device is not
found (0x03030000) in smd_dbg on I see the following lines:

Server:
State          UID   Inode   Local Address           Peer Address            Intf Mode Shutd Token    Sndbuf ..
LISTEN         00000 0103804 0.0.0.0:37373
ACTIVE         00000 0112895 ::ffff:10.10.93..:46093 ::ffff:10.10.93..:54474 0000 TCP 0x03030000
ACTIVE         00000 0112701 ::ffff:10.10.93..:19765 ::ffff:10.10.93..:51934 0000 TCP 0x03030000
LISTEN         00000 0112699 0.0.0.0:19765

Client:
State          UID   Inode   Local Address           Peer Address            Intf Mode Shutd Token    Sndbuf ...
ACTIVE         00000 0116203 10.10.93.11:54474       10.10.93.12:46093       0000 TCP 0x05000000/0x03030000
ACTIVE         00000 0116201 10.10.93.11:51934       10.10.93.12:19765       0000 TCP 0x05000000/0x03030000


However this doesn't match what I'm seeing in the other commands below

> 
> Before you giving out the fallback reason, based on your environment,
> this are some potential possibilities. You can check this list:
> 
> - RDMA device availability in netns. Run "ip netns exec server rdma dev"
>   to check RDMA device in both server/client. If exclusive mode is setted,
>   it should have different devices in different netns.

I get the following output that looks as expected to me:

Server:
2: roceP9p0s0: node_type ca fw 14.25.1020 node_guid 1d82:ff9b:1bfe:2c28 sys_image_guid 282c:001b:9b03:9803
Client:
4: roceP11p0s0: node_type ca fw 14.25.1020 node_guid 0982:ff9b:63fe:64e7 sys_image_guid e764:0063:9b03:9803


> - SMC-R device availability in netns. Run "ip netns exec server smcr d"
>   to check SMC device available list. Only if we have eth name in the
>   list, it can access by this netns. smc-tools matches ethernet NIC and
>   RDMA device, it can only find the name of eth nic in this netns, so
>   there is no name if this eth nic doesn't belong to this netns.
> 
>   Net-Dev         IB-Dev   IB-P  IB-State  Type          Crit  #Links  PNET-ID
>                   mlx5_0      1    ACTIVE  RoCE_Express2   No       0
>   eth2            mlx5_1      1    ACTIVE  RoCE_Express2   No       0
> 
>   This output shows we have ONE available RDMA device in this netns.

Here too things look good to me:

Server:

Net-Dev         IB-Dev   IB-P  IB-State  Type          Crit  #Links  PNET-ID
...
          roceP12p    1    ACTIVE  RoCE_Express2   No       0  NET26
          roceP11p    1    ACTIVE  RoCE_Express2   No       0  NET25
ens2076         roceP9p0    1    ACTIVE  RoCE_Express2   No       0  NET25

Client:

Net-Dev         IB-Dev   IB-P  IB-State  Type          Crit  #Links  PNET-ID
...
          roceP12p    1    ACTIVE  RoCE_Express2   No       0  NET26
ens1296         roceP11p    1    ACTIVE  RoCE_Express2   No       0  NET25
          roceP9p0    1    ACTIVE  RoCE_Express2   No       0  NET25

And I again confirmed that a pure RDMA workload ("qperf -cm1 ... rc_bw")
works with the RDMA namespacing set to exclusive but only if I add the
RDMA devices to the namespaces. I do wonder why the other RDMA devices are still
visible in the above output though?

> - Misc checks, such as memory usage, loop back connection and so on.
>   Also, you can check dmesg for device operations if you moved netns of
>   RDMA device. Every device's operation will log in dmesg.
> 
>   # SMC module init, adds two RDMA device.
>   [  +0.000512] smc: adding ib device mlx5_0 with port count 1
>   [  +0.000534] smc:    ib device mlx5_0 port 1 has pnetid
>   [  +0.000516] smc: adding ib device mlx5_1 with port count 1
>   [  +0.000525] smc:    ib device mlx5_1 port 1 has pnetid
> 
>   # Move one RDMA device to another netns.
>   [Feb21 14:16] smc: removing ib device mlx5_1
>   [  +0.015723] smc: adding ib device mlx5_1 with port count 1
>   [  +0.000600] smc:    ib device mlx5_1 port 1 has pnetid

There is no memory pressure and SMC-R between two systems works.

I also see the smc add/remove messages in dmesg as you describe:

smc: removing ib device roceP11p0s0
smc: adding ib device roceP11p0s0 with port count 1
smc:    ib device roceP11p0s0 port 1 has pnetid NET25
smc: removing ib device roceP9p0s0
smc: adding ib device roceP9p0s0 with port count 1
smc:    ib device roceP9p0s0 port 1 has pnetid NET25
mlx5_core 000b:00:00.0 ens1296: Link up
mlx5_core 0009:00:00.0 ens2076: Link up
IPv6: ADDRCONF(NETDEV_CHANGE): ens2076: link becomes ready
smc: removing ib device roceP11p0s0
smc: adding ib device roceP11p0s0 with port count 1
smc:    ib device roceP11p0s0 port 1 has pnetid NET25
mlx5_core 000b:00:00.0 ens1296: Link up
mlx5_core 0009:00:00.0 ens2076: Link up
smc: removing ib device roceP9p0s0
smc: adding ib device roceP9p0s0 with port count 1
smc:    ib device roceP9p0s0 port 1 has pnetid NET25
IPv6: ADDRCONF(NETDEV_CHANGE): ens1296: link becomes ready

(The PCI addresses and resulting names are normal for s390)

One thing I notice is that you don't seem to have a pnetid set
in your output, did you redact those or are you dealing differently
with PNETIDs? Maybe there is an issue with matching PNETIDs betwen
RDMA devices and network devices when namespaced?

I also tested with smc_chk instead of qperf to make sure it's not a
problem with LD_PRELOAD or anything like that. With that it simply
doesn't connect. 





[Index of Archives]     [Linux USB Devel]     [Video for Linux]     [Linux Audio Users]     [Photo]     [Yosemite News]     [Yosemite Photos]     [Linux Kernel]     [Linux SCSI]     [XFree86]

  Powered by Linux