On Wed, Apr 10, 2024 at 11:39 PM Jiri Pirko <jiri@xxxxxxxxxxx> wrote: > > Wed, Apr 10, 2024 at 11:07:02PM CEST, alexander.duyck@xxxxxxxxx wrote: > >On Wed, Apr 10, 2024 at 1:01 PM Andrew Lunn <andrew@xxxxxxx> wrote: > >> > >> On Wed, Apr 10, 2024 at 08:56:31AM -0700, Alexander Duyck wrote: > >> > On Tue, Apr 9, 2024 at 4:42 PM Andrew Lunn <andrew@xxxxxxx> wrote: > >> > > > >> > > > What is less clear to me is what do we do about uAPI / core changes. > >> > > > >> > > I would differentiate between core change and core additions. If there > >> > > is very limited firmware on this device, i assume Linux is managing > >> > > the SFP cage, and to some extend the PCS. Extending the core to handle > >> > > these at higher speeds than currently supported would be one such core > >> > > addition. I've no problem with this. And i doubt it will be a single > >> > > NIC using such additions for too long. It looks like ClearFog CX LX2 > >> > > could make use of such extensions as well, and there are probably > >> > > other boards and devices, maybe the Zynq 7000? > >> > > >> > The driver on this device doesn't have full access over the PHY. > >> > Basically we control everything from the PCS north, and the firmware > >> > controls everything from the PMA south as the physical connection is > >> > MUXed between 4 slices. So this means the firmware also controls all > >> > the I2C and the QSFP and EEPROM. The main reason for this is that > >> > those blocks are shared resources between the slices, as such the > >> > firmware acts as the arbitrator for 4 slices and the BMC. > >> > >> Ah, shame. You took what is probably the least valuable intellectual > >> property, and most shareable with the community and locked it up in > >> firmware where nobody can use it. > >> > >> You should probably stop saying there is not much firmware with this > >> device, and that Linux controls it. It clearly does not... > >> > >> Andrew > > > >Well I was referring more to the data path level more than the phy > >configuration. I suspect different people have different levels of > >expectations on what minimal firmware is. With this hardware we at > >least don't need to use firmware commands to enable or disable queues, > >get the device stats, or update a MAC address. > > > >When it comes to multi-host NICs I am not sure there are going to be > >any solutions that don't have some level of firmware due to the fact > > A small linux host on the nic that controls the eswitch perhaps? I mean, > the multi pf nic without a host in charge of the physical port and > swithing between it and pf is simply broken design. And yeah you would > probably now want to argue others are doing it already in the same way :) > True that. Well in our case there isn't an eswitch. The issue is more the logic for the Ethernet PHY isn't setup to only run one port. Instead the PHY is MUXed over 2 ports per interface, and then the QSFP interface itself is spread over 4 ports. What you end up with is something like the second to last image in this article[1] where you have a MAC/PCS pair per host sitting on top of one PMA with some blocks that are shared between the hosts and some that are not. The issue becomes management of access to the QSFP and PHY and how to prevent one host from being able to monopolize the PHY/QSFP or crash the others if something goes sideways. Then you have to also add in the BMC management on top of that. [1]: https://semiengineering.com/integrated-ethernet-pcs-and-phy-ip-for-400g-800g-hyperscale-data-centers/
