On Wed, Jul 21, 2021 at 09:11:27PM +0200, Thomas Gleixner wrote:
> The ordering of MSI-X enable in hardware is disfunctional:
>
> 1) MSI-X is disabled in the control register
> 2) Various setup functions
> 3) pci_msi_setup_msi_irqs() is invoked which ends up accessing
> the MSI-X table entries
> 4) MSI-X is enabled and masked in the control register with the
> comment that enabling is required for some hardware to access
> the MSI-X table
>
> #4 obviously contradicts #3. The history of this is an issue with the NIU
> hardware. When #4 was introduced the table access actually happened in
> msix_program_entries() which was invoked after enabling and masking MSI-X.
>
> This was changed in commit d71d6432e105 ("PCI/MSI: Kill redundant call of
> irq_set_msi_desc() for MSI-X interrupts") which removed the table write
> from msix_program_entries().
>
> Interestingly enough nobody noticed and either NIU still works or it did
> not get any testing with a kernel 3.19 or later.
>
> Nevertheless this is inconsistent and there is no reason why MSI-X can't be
> enabled and masked in the control register early on, i.e. move #4 above to
Does the above comment also apply to legacy MSI when it support per-vector
masking capability? Probably not interesting since without IR, we only give
1 vector to MSI.
Reviewed-by: Ashok Raj <ashok.raj@xxxxxxxxx>
> #1. This preserves the NIU workaround and has no side effects on other
> hardware.
>
> Fixes: d71d6432e105 ("PCI/MSI: Kill redundant call of irq_set_msi_desc() for MSI-X interrupts")
> Signed-off-by: Thomas Gleixner <tglx@xxxxxxxxxxxxx>
> Cc: David S. Miller <davem@xxxxxxxxxxxxx>
> Cc: Bjorn Helgaas <bhelgaas@xxxxxxxxxx>
> Cc: linux-pci@xxxxxxxxxxxxxxx
> ---
> drivers/pci/msi.c | 28 +++++++++++++++-------------
> 1 file changed, 15 insertions(+), 13 deletions(-)
>
> --- a/drivers/pci/msi.c
> +++ b/drivers/pci/msi.c
> @@ -772,18 +772,25 @@ static int msix_capability_init(struct p
> u16 control;
> void __iomem *base;
>
> - /* Ensure MSI-X is disabled while it is set up */
> - pci_msix_clear_and_set_ctrl(dev, PCI_MSIX_FLAGS_ENABLE, 0);
> + /*
> + * Some devices require MSI-X to be enabled before the MSI-X
> + * registers can be accessed. Mask all the vectors to prevent
> + * interrupts coming in before they're fully set up.
> + */
> + pci_msix_clear_and_set_ctrl(dev, 0, PCI_MSIX_FLAGS_MASKALL |
> + PCI_MSIX_FLAGS_ENABLE);
>
> pci_read_config_word(dev, dev->msix_cap + PCI_MSIX_FLAGS, &control);
> /* Request & Map MSI-X table region */
> base = msix_map_region(dev, msix_table_size(control));
> - if (!base)
> - return -ENOMEM;
> + if (!base) {
> + ret = -ENOMEM;
> + goto out_disable;
> + }
>
> ret = msix_setup_entries(dev, base, entries, nvec, affd);
> if (ret)
> - return ret;
> + goto out_disable;
>
> ret = pci_msi_setup_msi_irqs(dev, nvec, PCI_CAP_ID_MSIX);
> if (ret)
> @@ -794,14 +801,6 @@ static int msix_capability_init(struct p
> if (ret)
> goto out_free;
>
> - /*
> - * Some devices require MSI-X to be enabled before we can touch the
> - * MSI-X registers. We need to mask all the vectors to prevent
> - * interrupts coming in before they're fully set up.
> - */
> - pci_msix_clear_and_set_ctrl(dev, 0,
> - PCI_MSIX_FLAGS_MASKALL | PCI_MSIX_FLAGS_ENABLE);
> -
> msix_program_entries(dev, entries);
>
> ret = populate_msi_sysfs(dev);
> @@ -836,6 +835,9 @@ static int msix_capability_init(struct p
> out_free:
> free_msi_irqs(dev);
>
> +out_disable:
> + pci_msix_clear_and_set_ctrl(dev, PCI_MSIX_FLAGS_ENABLE, 0);
> +
> return ret;
> }
>
>
