Re: [question] when bonding with CX5 network card that support ROCE

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 




On 06.05.24 12:45, shaozhengchao wrote:
Hi yanjun:
  The following is the command output after the cat /proc/net/bonding
/bond0 command is run:

If I remember it correctly, it seems that it is a rdma LAG and bonding problem.

Not sure if it is a known problem or not. Please contact your local support.

Zhu Yanjun

[root@localhost ~]# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v5.10.0+

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2 (0)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0
Peer Notification Delay (ms): 0

802.3ad info
LACP rate: slow
Min links: 0
Aggregator selection policy (ad_select): stable
System priority: 65535
System MAC address: f4:1d:6b:6f:3b:97
Active Aggregator Info:
        Aggregator ID: 2
        Number of ports: 1
        Actor Key: 23
        Partner Key: 1
        Partner Mac Address: 00:00:00:00:00:00

Slave Interface: enp145s0f0
MII Status: up
Speed: 40000 Mbps
Duplex: full
Link Failure Count: 1
Permanent HW addr: f4:1d:6b:6f:3b:97
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: churned
Partner Churn State: churned
Actor Churned Count: 1
Partner Churned Count: 2
details actor lacp pdu:
    system priority: 65535
    system mac address: f4:1d:6b:6f:3b:97
    port key: 23
    port priority: 255
    port number: 1
    port state: 69
details partner lacp pdu:
    system priority: 65535
    system mac address: 00:00:00:00:00:00
    oper key: 1
    port priority: 255
    port number: 1
    port state: 1

Slave Interface: enp145s0f1
MII Status: up
Speed: 40000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: f4:1d:6b:6f:3b:98
Slave queue ID: 0
Aggregator ID: 2
Actor Churn State: none
Partner Churn State: churned
Actor Churned Count: 0
Partner Churned Count: 1
details actor lacp pdu:
    system priority: 65535
    system mac address: f4:1d:6b:6f:3b:97
    port key: 23
    port priority: 255
    port number: 2
    port state: 77
details partner lacp pdu:
    system priority: 65535
    system mac address: 00:00:00:00:00:00
    oper key: 1
    port priority: 255
    port number: 1
    port state: 1

Thank you
Zhengchao Shao


On 2024/5/6 16:26, Zhu Yanjun wrote:
On 06.05.24 06:46, shaozhengchao wrote:

When using the 5.10 kernel, I can find two IB devices using the ibv_devinfo command.
----------------------------------
[root@localhost ~]# lspci
91:00.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]
91:00.1 Ethernet controller: Mellanox Technologies MT27800 Family
----------------------------------
[root@localhost ~]# ibv_devinfo
hca_id: mlx5_0
         transport:                      InfiniBand (0)
         fw_ver:                         16.31.1014
         node_guid:                      f41d:6b03:006f:4743
         sys_image_guid:                 f41d:6b03:006f:4743
         vendor_id:                      0x02c9
         vendor_part_id:                 4119
         hw_ver:                         0x0
         board_id:                       HUA0000000004
         phys_port_cnt:                  1
                 port:   1
                         state:                  PORT_ACTIVE (4)
                         max_mtu:                4096 (5)
                         active_mtu:             1024 (3)
                         sm_lid:                 0
                         port_lid:               0
                         port_lmc:               0x00
                         link_layer:             Ethernet

hca_id: mlx5_1
         transport:                      InfiniBand (0)
         fw_ver:                         16.31.1014
         node_guid:                      f41d:6b03:006f:4744
         sys_image_guid:                 f41d:6b03:006f:4743
         vendor_id:                      0x02c9
         vendor_part_id:                 4119
         hw_ver:                         0x0
         board_id:                       HUA0000000004
         phys_port_cnt:                  1
                 port:   1
                         state:                  PORT_ACTIVE (4)
                         max_mtu:                4096 (5)
                         active_mtu:             1024 (3)
                         sm_lid:                 0
                         port_lid:               0
                         port_lmc:               0x00
                         link_layer:             Ethernet
----------------------------------
But after the two network ports are bonded, only one IB device is
available, and only PF0 can be used.
[root@localhost shaozhengchao]# ibv_devinfo
hca_id: mlx5_bond_0
         transport:                      InfiniBand (0)
         fw_ver:                         16.31.1014
         node_guid:                      f41d:6b03:006f:4743
         sys_image_guid:                 f41d:6b03:006f:4743
         vendor_id:                      0x02c9
         vendor_part_id:                 4119
         hw_ver:                         0x0
         board_id:                       HUA0000000004
         phys_port_cnt:                  1
                 port:   1
                         state:                  PORT_ACTIVE (4)
                         max_mtu:                4096 (5)
                         active_mtu:             1024 (3)
                         sm_lid:                 0
                         port_lid:               0
                         port_lmc:               0x00
                         link_layer:             Ethernet

The current Linux mainline driver is the same.

I found the comment ("If bonded, we do not add an IB device for PF1.")
in the mlx5_lag_intf_add function of the 5.10 branch driver code.

Not sure if rdma lag is enabled for this or not. /proc/net/bonding will provide more more details normally.

Zhu Yanjun

This indicates that wthe the same NIC is used, only PF0 support bonding?
Are there any other constraints, when enable bonding with CX5?

Thank you
Zhengchao Shao





[Index of Archives]     [Linux USB Devel]     [Video for Linux]     [Linux Audio Users]     [Photo]     [Yosemite News]     [Yosemite Photos]     [Linux Kernel]     [Linux SCSI]     [XFree86]

  Powered by Linux