On Tue, Jun 04, 2024 at 04:20:16PM +0200, Andrew Lunn wrote: > On Sun, Jun 02, 2024 at 09:36:05AM +0530, Siddharth Vadapalli wrote: [...] > > If you do want to add a third model, where Linux has some insight into > the switch, you need to coordinate with other vendors in the > automotive world, and come up with a model which everybody can > use. What i don't want is a TI model, followed by a Realtek model, > followed by a vendor XYZ model. So if you need more than what the > first model above provides, you will need to get a consortium of > vendors together to design a new model a few vendors agree on. I believe that a third model is required given the System Architecture and the use-case that it must cater to. I agree completely that having a vendor specific implementation should always be the last step when it is just not possible to generalize any portion of the implementation. I will describe the existing Architecture on the TI SoC and will also attempt to generalize the implementation below. I hope that you could review it and guide me towards the generic, vendor-agnostic implementation which will also address the use-case that this series corresponds to. I am willing to work on the generic implementation since I assume that this series does keep it generic enough that it could be extended to be vendor independent. So there might be minor changes required when switching to the generic model. On the other hand, based on the description that I provide below, if you think that the existing models can also be slightly modified to accomodate the use-case, I will surely take that into consideration and work on the corresponding implementation. System Architecture and Implementation Details ============================================== The CPSW Ethernet Switch has a single Host Port (CPU facing port) through which it can receive data from the Host(s) and transmit data to the Host(s). The exchange of data occurs via TX/RX DMA Channels (Hardware Queues). These Hardware Queues are a limited resource (8 TX Channels and up to 64 RX Flows). If the Operating System on any of the cores is the sole user of CPSW then all of these Hardware Queues can be claimed by that OS. However, when CPSW has to be shared across the Operating Systems on various cores with the aim of enabling Ethernet Functionality for the Applications running on different cores, it is necessary to share these Hardware Queues in a manner that prevents conflicts. On the control path which corresponds to the configuration of CPSW to get it up and running, since there is no Integrated Processor within CPSW that can be programmed with a startup configuration, either the Operating System or Firmware running on one of the cores has to take the responsibility of setting it. One option in this case happens to be the Ethernet Switch Firmware (EthFw) which is loaded by the Bootloader on a remote core at the same time that Linux and other Operating Systems begin booting. EthFw quickly powers on and configures CPSW getting the Forwarding Path functional. Once Linux and other Operating Systems on various cores are ready, they can communicate with EthFw to obtain details of the Hardware Queues allocated to them to exchange data with CPSW. With the knowledge of the Hardware Queues that have been allocated, Linux can use the DMA APIs to setup these queues to exchange data with CPSW. Setting up the Hardware Queues alone isn't sufficient to exchange data with the external network. Consider the following example: The ethX interface in userspace which has been created to transmit/receive data to/from CPSW has the user-assigned MAC Address of "M". The ping command is run with the destination IP of "D". This results in an ARP request sent from ethX which is transmitted out of all MAC Ports of CPSW since it is a Broadcast request. Assuming that "D" is a valid destination IP, the ARP reply is received on one of the MAC Ports which is now a Unicast reply with the destination MAC Address of "M". The ALE (Address Lookup Engine) in CPSW has learnt that the MAC Address "M" corresponds to the Host Port when the ARP request was sent out. So the Unicast reply isn't dropped. The challenge however is determining which RX DMA Channel (Flow) to send the Unicast reply on. In the case of a single Operating System owning all Hardware Queues, sending it on any of the RX DMA Channels would have worked. In the current case where the RX DMA Channels map to different Hosts (Operating Systems and Applications), the mapping between the MAC Address "M" and the RX DMA Channel has to be setup to ensure that the correct Host receives the ARP reply. This necessitates a method to inform the MAC Address "M" associated with the interface ethX to EthFw so that EthFw can setup the MAC Address "M" to RX DMA Channel map accordingly. At this point, Linux can exchange data with the external network via CPSW, but no device on the external network can initiate the communication by itself unless it already has the ARP entry for the IP Address of ethX. That's because CPSW doesn't support packet replication implying that any Broadcast/Multicast packets received on the MAC Ports can only be sent on one of the RX DMA Channels. So the Broadcast/Multicast packets can only be received by one Host. Consider the following example: A PC on the network tries to ping the IP Address of ethX. In both of the following cases: 1. Linux hasn't yet exchanged data with the PC via ethX. 2. The MAC Address of ethX has changed. the PC sends an ARP request to one of the MAC Ports on CPSW to figure out the MAC Address of ethX. Since the ARP request is a Broadcast request, it is not possible for CPSW to determine the correct Host, since the Broadcast MAC isn't unique to any Host. So CPSW is forced to send the Broadcast request to a preconfigured RX DMA Channel which in this case happens to be the one mapped to EthFw. Thus, if EthFw is aware of the IP Address of ethX, it can generate and send the ARP reply containing the MAC Address "M" of ethX that it was informed of. With this, the PC can initiate communication with Linux as well. Similarly, in the case of Multicast packets, if Linux wishes to receive certain Multicast packets, it needs to inform the same to EthFw which shall then replicate the Multicast packets it received from CPSW and transmit them via alternate means (Shared Memory for example) to Linux. The following is a summary of the steps so far required to enable Ethernet Functionality for applications running on Linux: 1. Determine and setup the Hardware Queues allocated to Linux 2. Inform the MAC Address of ethX to EthFw 3. Inform the IP Address of ethX to EthFw 4. Inform any of the Multicast Addresses associated with ethX to EthFw All data between Linux (Or any Operating System) and EthFw is exchanged via the Hardware Mailboxes with the help of the RPMsg framework. Since all the resource allocation information comes from EthFw, the vendor-specific implementation in the Linux Client is limited to the DMA APIs used to setup the Hardware Queues and to transmit/receive data with the Ethernet Switch. Therefore, it might be possible to move most of the vendor specific implementation to the Switch Configuration Firmware (similar to EthFw), to make the Linux Client implementation as generic and vendor agnostic as possible. I believe that this series more or less does the same, just using custom terminology which can be made generic. I can update this series to a generic implementation along with proper documentation and naming convention to enable any vendor to reuse the same without having to modify the implementation. The RPMsg ABIs can be given generic names with extensive documentation and also designed to be extensible enough to cater to functional enhancements over time. Kindly let me know your thoughts on this. Regards, Siddharth.
