On Wed, Oct 11, 2017 at 06:52:57PM +0800, Dongdong Liu wrote: > After removing and adding back the PCI root port device, > we see the PCIe port service drivers request irq failed. > pcie_pme: probe of 0000:00:00.0:pcie001 failed with error -22 > aer: probe of 0000:00:00.0:pcie002 failed with error -22 > pciehp 0000:00:00.0:pcie004: Slot #0 AttnBtn- PwrCtrl- MRL- AttnInd- > PwrInd- HotPlug+ Surprise+ Interlock- NoCompl- LLActRep+ > pciehp 0000:00:00.0:pcie004: Cannot get irq 20 for the hotplug controller > pciehp 0000:00:00.0:pcie004: Notification initialization failed (-1) > dpc 0000:00:00.0:pcie010: request IRQ22 failed: -22 > dpc: probe of 0000:00:00.0:pcie010 failed with error -22 > > The current code basically does this: > > - allocate 32 vectors > - figure out vector used by PME and hotplug > - figure out vector used by AER > - figure out vector used by DPC > - free the 32 vectors we allocated > - allocate only as many vectors as we need > > but it is broken as calling pci_free_irq_vectors() > invalidates the IRQ numbers returned before by pci_irq_vectors(); > > The hardware works: > - PME and hotplug use the Interrupt Message Number from the PCIe > Capability register. > > - AER uses the AER Interrupt Message Number from the AER Root Error > Status register. > > - DPC uses the DPC Interrupt Message Number from the DPC Capability > register. > > - FRS (not supported by Linux yet) uses the FRS Interrupt Message > Number from the FRS Queuing Capability register. > > - That's a total of 4 possible MSI/MSI-X vectors used for PME, > hotplug, AER, DPC, and FRS, so there's no point in trying to > allocate more than 4 vectors (we currently start with 32). You pointed out that there actually may be some benefit to allocating more than 4 vectors because the hardware may not distribute the Interrupt Message Numbers across the allocated vectors, e.g., you have a system with HP/PME - message number 0 AER - message number 2 DPC - message number 6 In that case, if we allocate 8 vectors, there's probably no sharing (other than HP/PME), but if we allocate 4 vectors, the hardware might put DPC on message number 0 or 2. Out of curiosity, what does happen on this system when we only allocate 4 vectors? Does DPC end up sharing a vector with PME or AER? Or is it smart enough to put DPC on message number 1 or 3? Bjorn