On Fri, Oct 14, 2022 at 11:18:36AM +0000, Radovanovic, Aleksandar wrote: > And that still does not imply lack of ordering or sharing of MSI > target addresses between devices. Either the end point generates the MSI, and maybe the bridge mangles it, or it raises a lot of suspicion that this is not right. If the end point generates the MSI then it raises the question why do we need to tolerate these limits? > This is a highly programmable IP block, at the core of which is an > interconnect interfacing to programmable logic (PL), a number of > PCIe controllers (either endpoint or root-port), DMA engines, > offload engines, the embedded processor subsystem (PSX), etc. DMA > and interrupts can be routed across it in almost any (meaningful) > direction. The datapath 'endpoints' request DMA and interrupts, but > don't concern themselves with the mechanics of delivering that in > the target domain. It is the responsibility of the egress bridges to > the target domains to convert the interconnect interrupt > transactions to whatever the interrupt delivery mechanism for that > domain is. E.g. for PCIe controllers in endpoint mode, that would be > through PCIe MSI-X tables internal to the controller (and managed by > the PCIe host), for PSX that would be the PSX bridge (partially > managed by the PSX OS, mediated through firmware, i.e. through CDX > bus driver) and so on. It is the responsibility of the interconnect > to maintain transaction ordering (including DMA vs. interrupts). It > is the responsibility of the firmware to manage the bridges > according to the implemented use-case, so everything works as > expected. Again, this all just seems wrongly designed. MSI should not be part of an interconnect bridge. We did that already 20 years ago, it was called IOAPICs on x86 and I think everyone is happy to see it gone. If you want to build IOAPICs again, I guess you can, but that is a slightly different SW setup than the MSI you are trying to use here, and even that didn't have the same limitations you are proposing. > So, yes, the hardware that translates interrupt transactions to GIC > AXI writes is shared between endpoints, but what I said above still > applies. And that doesn't necessarily make it weird/wrong, it's just > more complex than you might think. If it doesn't fit the architecture, then I think it must be considered wrong. Mis-using platform architected components like MSI in HW is problematic. You should design the HW properly so you don't have these problems. Involving FW in the MSI setup is also a bad idea, POWER did this and it made a big mess of their arch code :( Jason
