-----"Potnuri Bharat Teja" <bharat@xxxxxxxxxxx> wrote: -----
>To: "Bernard Metzler" <BMT@xxxxxxxxxxxxxx>
>From: "Potnuri Bharat Teja" <bharat@xxxxxxxxxxx>
>Date: 02/17/2020 03:51PM
>Cc: "Kamal Heib" <kamalheib1@xxxxxxxxx>, "linux-rdma@xxxxxxxxxxxxxxx"
><linux-rdma@xxxxxxxxxxxxxxx>, "Jason Gunthorpe" <jgg@xxxxxxxx>, "Doug
>Ledford" <dledford@xxxxxxxxxx>
>Subject: [EXTERNAL] Re: [PATH for-next] RDMA/siw: Fix setting
>active_{speed, width} attributes
>
>On Monday, February 02/17/20, 2020 at 19:57:54 +0530, Bernard Metzler
>wrote:
>> -----"Kamal Heib" <kamalheib1@xxxxxxxxx> wrote: -----
>>
>> >To: "Bernard Metzler" <BMT@xxxxxxxxxxxxxx>
>> >From: "Kamal Heib" <kamalheib1@xxxxxxxxx>
>> >Date: 02/17/2020 03:12PM
>> >Cc: linux-rdma@xxxxxxxxxxxxxxx, "Jason Gunthorpe" <jgg@xxxxxxxx>,
>> >"Doug Ledford" <dledford@xxxxxxxxxx>
>> >Subject: [EXTERNAL] Re: [PATH for-next] RDMA/siw: Fix setting
>> >active_{speed, width} attributes
>> >
>> >On Mon, Feb 17, 2020 at 10:13:21AM +0000, Bernard Metzler wrote:
>> >> -----"Kamal Heib" <kamalheib1@xxxxxxxxx> wrote: -----
>> >>
>> >> >To: "Bernard Metzler" <BMT@xxxxxxxxxxxxxx>
>> >> >From: "Kamal Heib" <kamalheib1@xxxxxxxxx>
>> >> >Date: 02/16/2020 02:43PM
>> >> >Cc: linux-rdma@xxxxxxxxxxxxxxx, "Jason Gunthorpe"
><jgg@xxxxxxxx>,
>> >> >"Doug Ledford" <dledford@xxxxxxxxxx>
>> >> >Subject: [EXTERNAL] Re: [PATH for-next] RDMA/siw: Fix setting
>> >> >active_{speed, width} attributes
>> >> >
>> >> >On Thu, Feb 13, 2020 at 01:59:30PM +0000, Bernard Metzler
>wrote:
>> >> >> -----"Kamal Heib" <kamalheib1@xxxxxxxxx> wrote: -----
>> >> >>
>> >> >> >To: linux-rdma@xxxxxxxxxxxxxxx
>> >> >> >From: "Kamal Heib" <kamalheib1@xxxxxxxxx>
>> >> >> >Date: 02/13/2020 02:07PM
>> >> >> >Cc: "Jason Gunthorpe" <jgg@xxxxxxxx>, "Doug Ledford"
>> >> >> ><dledford@xxxxxxxxxx>, "Bernard Metzler"
><bmt@xxxxxxxxxxxxxx>,
>> >> >"Kamal
>> >> >> >Heib" <kamalheib1@xxxxxxxxx>
>> >> >> >Subject: [EXTERNAL] [PATH for-next] RDMA/siw: Fix setting
>> >> >> >active_{speed, width} attributes
>> >> >> >
>> >> >> >Make sure to set the active_{speed, width} attributes to
>avoid
>> >> >> >reporting
>> >> >> >the same values regardless of the underlying device.
>> >> >> >
>> >> >> >Fixes: 303ae1cdfdf7 ("rdma/siw: application interface")
>> >> >> >Signed-off-by: Kamal Heib <kamalheib1@xxxxxxxxx>
>> >> >> >---
>> >> >> > drivers/infiniband/sw/siw/siw_verbs.c | 7 ++++---
>> >> >> > 1 file changed, 4 insertions(+), 3 deletions(-)
>> >> >> >
>> >> >> >diff --git a/drivers/infiniband/sw/siw/siw_verbs.c
>> >> >> >b/drivers/infiniband/sw/siw/siw_verbs.c
>> >> >> >index 73485d0da907..b1aaec912edb 100644
>> >> >> >--- a/drivers/infiniband/sw/siw/siw_verbs.c
>> >> >> >+++ b/drivers/infiniband/sw/siw/siw_verbs.c
>> >> >> >@@ -165,11 +165,12 @@ int siw_query_port(struct ib_device
>> >> >*base_dev,
>> >> >> >u8 port,
>> >> >> > struct ib_port_attr *attr)
>> >> >> > {
>> >> >> > struct siw_device *sdev = to_siw_dev(base_dev);
>> >> >> >+ int rc;
>> >> >> >
>> >> >> > memset(attr, 0, sizeof(*attr));
>> >> >> >
>> >> >> >- attr->active_speed = 2;
>> >> >> >- attr->active_width = 2;
>> >> >> >+ rc = ib_get_eth_speed(base_dev, port, &attr->active_speed,
>> >> >> >+ &attr->active_width);
>> >> >> > attr->gid_tbl_len = 1;
>> >> >> > attr->max_msg_sz = -1;
>> >> >> > attr->max_mtu = ib_mtu_int_to_enum(sdev->netdev->mtu);
>> >> >> >@@ -192,7 +193,7 @@ int siw_query_port(struct ib_device
>> >*base_dev,
>> >> >u8
>> >> >> >port,
>> >> >> > * attr->subnet_timeout = 0;
>> >> >> > * attr->init_type_repy = 0;
>> >> >> > */
>> >> >> >- return 0;
>> >> >> >+ return rc;
>> >> >> > }
>> >> >> >
>> >> >> > int siw_get_port_immutable(struct ib_device *base_dev, u8
>> >port,
>> >> >> >--
>> >> >> >2.21.1
>> >> >> >
>> >> >> >
>> >> >> Hi Kamal,
>> >> >
>> >> >Hi Bernard,
>> >> >
>> >> >> Many thanks for looking after this! So there definitely seem
>to
>> >> >> be applications which are taking care of those values. So,
>good
>> >> >> to get my obvious laziness fixed.
>> >> >>
>> >> >
>> >> >Sure :)
>> >> >
>> >> >> I tried your patch on a 40Gbs Ethernet link (Chelsio cxgb4
>> >driver).
>> >> >> Works in principle, but reported numbers are off. I am not
>> >saying
>> >> >> I would get right numbers when using Chelsio HW iWarp
>> >(iw_cxgb4),
>> >> >> but it's closer to reality (using ibv_devinfo <ibname> -vv)
>> >> >>
>> >> >> iw_cxgb4 driver:
>> >> >> ...
>> >> >> active_width: 4X (2)
>> >> >> active_speed: 25.0 Gbps (32)
>> >> >>
>> >> >> siw driver with your patch:
>> >> >> ...
>> >> >> active_width: 4X (2)
>> >> >> active_speed: 10.0 Gbps (8)
>> >> >>
>> >> >> Any idea how we can improve that, maybe coming even
>> >> >> close to reality (40Gbs)?
>> >> >
>> >> >Could you please share the output of ethtool <if_name> for the
>> >> >underlying
>> >> >net device that used for both iw_cxgb4 and siw?
>> >> >
>> >> H Kamal,
>> >>
>> >> Sure! Speed looks correct, and its also what I get
>> >> at maximum:
>> >>
>> >> [bmt@spoke ~]$ ethtool enp1s0f4
>> >> Settings for enp1s0f4:
>> >> Supported ports: [ FIBRE ]
>> >> Supported link modes: 40000baseSR4/Full
>> >> Supported pause frame use: Symmetric
>> >> Supports auto-negotiation: Yes
>> >> Supported FEC modes: None
>> >> Advertised link modes: 40000baseSR4/Full
>> >> Advertised pause frame use: Symmetric
>> >> Advertised auto-negotiation: Yes
>> >> Advertised FEC modes: None
>> >> Link partner advertised link modes: 40000baseSR4/Full
>> >> Link partner advertised pause frame use: Symmetric
>> >> Link partner advertised auto-negotiation: Yes
>> >> Link partner advertised FEC modes: None
>> >> Speed: 40000Mb/s
>> >> Duplex: Full
>> >> Port: Direct Attach Copper
>> >> PHYAD: 255
>> >> Transceiver: internal
>> >> Auto-negotiation: on
>> >> Cannot get wake-on-lan settings: Operation not permitted
>> >> Current message level: 0x000000ff (255)
>> >> drv probe link timer ifdown ifup
>> >rx_err tx_err
>> >> Link detected: yes
>> >>
>> >
>> >Hi Bernard,
>> >
>> >Well, this is the expected value for 40GbE, Because the reported
>> >value
>> >is 4X aggregation of FDR10. For more info please the table of
>speeds
>> >under [1].
>> >
>> >[1] -
>>
>>https://urldefense.proofpoint.com/v2/url?u=https-3A__en.wikipedia.or
>g
>>
>>_wiki_InfiniBand&d=DwIBAg&c=jf_iaSHvJObTbx-siA1ZOg&r=2TaYXQ0T-r8ZO1P
>P
>>
>>1alNwU_QJcRRLfmYTAgd3QCvqSc&m=Ay2tYZ7Z-SHKRw8UZDCk76kwlZzvkXhRMrO_0j
>k
>> >YjcY&s=D4Z0CAH5UVO95WHNnUozLzrqjxRQgVe-2lc8_jwVlhw&e=
>> >
>> >Thanks,
>> >Kamal
>> >
>> Hi Kamal, so I have to do the math! 4 x 10Gbs = 40Gbs.
>> So it is all correct as reported by ibv_devinfio
>> (and somehow strange what the iw_cxgb4 makes out of it).
>Hi Bernard,
>Are you running siw and iw_cxgb4 on the same port or different ports?
>if not same, Is the port running iw_cxgb4 connected with 100G cable?
>If 40G cable is conencted to iw_cxgb4 as well then its reporting
>wrong.
>
This is a single port 40Gbs setup. All my fault:
For iw_cxgb4, I looked at the reported values for the second
port, which is not physically connected but reported PORT_DOWN
and 4 x 25Gbs.
The connected port 1 has 4 x 10Gbs for both siw and iw_cxgb4.
Sorry about the confusion!
So its all good.
Thanks
Bernard.
