Dear Jakub,
Am 15.03.22 um 03:57 schrieb Jakub Kicinski:
On Mon, 14 Mar 2022 16:07:08 +0100 Paul Menzel wrote:
There might be something more wrong with the patch in the subject: The
usability of the ports from a single card (with older firmware?) now
depends on the order the ports are enabled (first port enabled is
working, second port enabled is not working, driver complaining about a
firmware mismatch).
In the following examples, the driver was not built-in to the kernel but
loaded from the root filesystem instead, so there is no initramfs
related problem here.
For the records:
root@ira:~# dmesg|grep bnx2x
[ 18.749871] bnx2x 0000:45:00.0: msix capability found
[ 18.766534] bnx2x 0000:45:00.0: part number 394D4342-31373735-31314131-473331
[ 18.799198] bnx2x 0000:45:00.0: 32.000 Gb/s available PCIe bandwidth (5.0 GT/s PCIe x8 link)
[ 18.807638] bnx2x 0000:45:00.1: msix capability found
[ 18.824509] bnx2x 0000:45:00.1: part number 394D4342-31373735-31314131-473331
[ 18.857171] bnx2x 0000:45:00.1: 32.000 Gb/s available PCIe bandwidth (5.0 GT/s PCIe x8 link)
[ 18.865619] bnx2x 0000:46:00.0: msix capability found
[ 18.882636] bnx2x 0000:46:00.0: part number 394D4342-31373735-31314131-473331
[ 18.915196] bnx2x 0000:46:00.0: 32.000 Gb/s available PCIe bandwidth (5.0 GT/s PCIe x8 link)
[ 18.923636] bnx2x 0000:46:00.1: msix capability found
[ 18.940505] bnx2x 0000:46:00.1: part number 394D4342-31373735-31314131-473331
[ 18.973167] bnx2x 0000:46:00.1: 32.000 Gb/s available PCIe bandwidth (5.0 GT/s PCIe x8 link)
[ 46.480660] bnx2x 0000:45:00.0 net04: renamed from eth4
[ 46.494677] bnx2x 0000:45:00.1 net05: renamed from eth5
[ 46.508544] bnx2x 0000:46:00.0 net06: renamed from eth6
[ 46.524641] bnx2x 0000:46:00.1 net07: renamed from eth7
root@ira:~# ls /lib/firmware/bnx2x/
bnx2x-e1-6.0.34.0.fw bnx2x-e1-7.13.1.0.fw bnx2x-e1-7.8.2.0.fw
bnx2x-e1h-7.12.30.0.fw bnx2x-e1h-7.8.19.0.fw bnx2x-e2-7.10.51.0.fw bnx2x-e2-7.8.17.0.fw
bnx2x-e1-6.2.5.0.fw bnx2x-e1-7.13.11.0.fw bnx2x-e1h-6.0.34.0.fw
bnx2x-e1h-7.13.1.0.fw bnx2x-e1h-7.8.2.0.fw bnx2x-e2-7.12.30.0.fw bnx2x-e2-7.8.19.0.fw
bnx2x-e1-6.2.9.0.fw bnx2x-e1-7.13.15.0.fw bnx2x-e1h-6.2.5.0.fw
bnx2x-e1h-7.13.11.0.fw bnx2x-e2-6.0.34.0.fw bnx2x-e2-7.13.1.0.fw bnx2x-e2-7.8.2.0.fw
bnx2x-e1-7.0.20.0.fw bnx2x-e1-7.13.21.0.fw bnx2x-e1h-6.2.9.0.fw
bnx2x-e1h-7.13.15.0.fw bnx2x-e2-6.2.5.0.fw bnx2x-e2-7.13.11.0.fw
bnx2x-e1-7.0.23.0.fw bnx2x-e1-7.2.16.0.fw bnx2x-e1h-7.0.20.0.fw
bnx2x-e1h-7.13.21.0.fw bnx2x-e2-6.2.9.0.fw bnx2x-e2-7.13.15.0.fw
bnx2x-e1-7.0.29.0.fw bnx2x-e1-7.2.51.0.fw bnx2x-e1h-7.0.23.0.fw
bnx2x-e1h-7.2.16.0.fw bnx2x-e2-7.0.20.0.fw bnx2x-e2-7.13.21.0.fw
bnx2x-e1-7.10.51.0.fw bnx2x-e1-7.8.17.0.fw bnx2x-e1h-7.0.29.0.fw
bnx2x-e1h-7.2.51.0.fw bnx2x-e2-7.0.23.0.fw bnx2x-e2-7.2.16.0.fw
bnx2x-e1-7.12.30.0.fw bnx2x-e1-7.8.19.0.fw bnx2x-e1h-7.10.51.0.fw
bnx2x-e1h-7.8.17.0.fw bnx2x-e2-7.0.29.0.fw bnx2x-e2-7.2.51.0.fw
Now with v5.10.95, the first kernel of the series which includes
fdcfabd0952d ("bnx2x: Utilize firmware 7.13.21.0") and later:
root@ira:~# dmesg -w &
[...]
root@ira:~# ip link set net04 up
[ 88.504536] bnx2x 0000:45:00.0 net04: using MSI-X IRQs: sp 47 fp[0] 49 ... fp[7] 56
root@ira:~# ip link set net05 up
[ 90.825820] bnx2x: [bnx2x_compare_fw_ver:2380(net05)]bnx2x with FW 120d07 was already loaded which mismatches my 150d07 FW. Aborting
RTNETLINK answers: Device or resource busy
root@ira:~# ip link set net04 down
root@ira:~# ip link set net05 down
root@ira:~# ip link set net05 up
[ 114.462448] bnx2x 0000:45:00.1 net05: using MSI-X IRQs: sp 58 fp[0] 60 ... fp[7] 67
root@ira:~# ip link set net04 up
[ 117.247763] bnx2x: [bnx2x_compare_fw_ver:2380(net04)]bnx2x with FW 120d07 was already loaded which mismatches my 150d07 FW. Aborting
RTNETLINK answers: Device or resource busy
With v5.10.94, both ports work fine:
root@ira:~# dmesg -w &
[...]
root@ira:~# ip link set net04 up
[ 133.126647] bnx2x 0000:45:00.0 net04: using MSI-X IRQs: sp 47 fp[0] 49 ... fp[7] 56
root@ira:~# ip link set net05 up
[ 136.215169] bnx2x 0000:45:00.1 net05: using MSI-X IRQs: sp 58 fp[0] 60 ... fp[7] 67
One additional note, that it’s totally unclear to us, where FW version
120d07 in the error message comes from. It maps to 7.13.18.0, which is
nowhere to be found and too new to be on the cards EEPROM, which should
be from 2013 or so.
Hrm, any chance an out-of-tree driver or another OS loaded it?
No, no such chance. The system firmware was not touched in the last five
years, and there is no other OS.
Does the dmesg indicate that the host loaded the FW at all?
Unfortunately, no such Linux log messages exist (in the code).
Looks like upstream went from .15 to .21, .18 was never in the picture.
Yes, that is the strange thing. Is there any chance, that the `.18` is
in the firmware file somehow, and was forgotten to be updated to `.21`?
```
$ md5sum /lib/firmware/bnx2x/*{15,21}*
4137c813f3ff937f01fddf13c309d972 /lib/firmware/bnx2x/bnx2x-e1-7.13.15.0.fw
075539e1072908a0c86ba352c1130f60 /lib/firmware/bnx2x/bnx2x-e1h-7.13.15.0.fw
e59925dedf7ed95679e629dfeeaf5803 /lib/firmware/bnx2x/bnx2x-e2-7.13.15.0.fw
a470a0d77930ef72ac7b725a5ff447fe /lib/firmware/bnx2x/bnx2x-e1-7.13.21.0.fw
f02012b118664c0579ab58757f276982 /lib/firmware/bnx2x/bnx2x-e1h-7.13.21.0.fw
47667a7bf156d1e531fde42d800e4a66 /lib/firmware/bnx2x/bnx2x-e2-7.13.21.0.fw
```
Also, will the revert work for you?
I tested reverting the three patches you also listed on top of 5.10.103,
and that restored the working behavior. Manish’s patch *[RFC net] bnx2x:
fix built-in kernel driver load failure* also fixes the issue.
Kind regards,
Paul
