https://lore.kernel.org/all/87bljg7u4f.fsf@xxxxxxxxxxxxxxxxxxxxxxx/ My view on that issue was wrong because the Intel people described the problem wrong. But the above pretty much begs for a proper separation and hierarchical design because you provide an actual bus and distinct devices. Reusing the ASCII art from that old thread for the second case, but it's probably the same for the first one: ]-------------------------------------------| | PCI device | ]-------------------| | | Physical function | | ]-------------------| | ]-------------------|----------| | | Control block for subdevices | | ]------------------------------| | | | <- "Subdevice BUS" | | | | | |-- Subddevice 0 | | |-- Subddevice 1 | | |-- ... | | |-- Subddevice N | ]-------------------------------------------| 1) cxl_pci driver binds to the PCI device. 2) cxl_pci driver creates AUXbus 3) Bus enumerates devices on AUXbus 4) Drivers bind to the AUXbus devices So you have a clear provider consumer relationship. Whether the subdevice utilizes resources of the PCI device or not is a hardware implementation detail. The important aspect is that you want to associate the subdevice resources to the subdevice instances and not to the PCI device which provides them. Let me focus on interrupts, but it's probably the same for everything else which is shared. Look at how the PCI device manages interrupts with the per device MSI mechanism. Forget doing this with either one of the legacy mechanisms. 1) It creates a hierarchical interrupt domain and gets the required resources from the provided parent domain. The PCI side does not care whether this is x86 or arm64 and it neither cares whether the parent domain does remapping or not. The only way it cares is about the features supported by the different parent domains (think multi-MSI). 2) Driver side allocations go through the per device domain That AUXbus is not any different. When the CPMU driver binds it wants to allocate interrupts. So instead of having a convoluted construct reaching into the parent PCI device, you simply can do: 1) Let the cxl_pci driver create a MSI parent domain and set that in the subdevice::msi::irqdomain pointer. 2) Provide cxl_aux_bus_create_irqdomain() which allows the CPMU device driver to create a per device interrupt domain. 3) Let the CPMU driver create its per device interrupt domain with the provided parent domain 4) Let the CPMU driver allocate its MSI-X interrupts through the per device domain Now on the cxl_pci side the AUXbus parent interrupt domain allocation function does: if (!pci_msix_enabled(pdev)) return pci_msix_enable_and_alloc_irq_at(pdev, ....); else return pci_msix_alloc_irq_at(pdev, ....); The condition can be avoided if the clx_pci driver enables MSI-X anyway for it's own purposes. Obviously you need the corresponding counterpart for free(), but that's left as an exercise for the reader. That still associates the subdevice specific MSI-X interrups to the underlying PCI device, but then you need to look at teardown. The cxl_pci driver cannot go away before the CPMU drivers are shutdown. The CPMU driver shutdown releases the interrupts through the interrupt domain hierarchy, which removes them from the parent PCI device. Once all CPMU drivers are gone, the shutdown of the cxl_pci driver gets rid of the cxl_pci driver specific interrupts and everything is cleaned up. This works out of the box on x86. The rest needs to gain support for MSI_FLAG_PCI_MSIX_ALLOC_DYN. ARM64 and parts of ARM32 are already converted over to the MSI parent domain concept, they just lack support for dynamic allocation. The rest of the arch/ world needs some more work. That's the problem of the architecture folks and that CXL auxbus thing can simply tell them if (!pci_msix_can_alloc_dyn(pdev)) return -E_MAKE_YOUR_HOME_WORK; and be done with it. Proliferating support for "two" legacy PCI/MSI mechanisms by violating layering is not going to happen. Neither I'm interrested to have creative workarounds for MSI as they are going to create more problems than they solve, unless you come up with a clean, simple and maintainable solution which works under all circumstances. I'm not holding my breath... I might not have all the crucial information as it happened in the previous IMS discussion, but looking at your ASCII art makes me halfways confident that I'm on the right track. Thanks, tglx
