> The MSI Driver Guide HOWTO
> Tom L Nguyen tom.l.nguyen@xxxxxxxxx
> 10/03/2003
> Revised Feb 12, 2004 by Martine Silbermann
> email: Martine.Silbermann@xxxxxx
> Revised Jun 25, 2004 by Tom L Nguyen
> Revised Jul 9, 2008 by Matthew Wilcox <willy@xxxxxxxxxxxxxxx>
> Copyright 2003, 2008 Intel Corporation
>
> 1. About this guide
>
> This guide describes the basics of Message Signaled Interrupts (MSIs),
good (Signaled). Please be consistent with that spelling...
> the advantages of using MSI over traditional interrupt mechanisms, how
> to change your driver to use MSI or MSI-X and some basic diagnostics to
> try if a device doesn't support MSIs.
>
>
> 2. What are MSIs?
>
> A Message Signalled Interrupt is a write from the device to a special
vs. that one. Just spell it however the PCI SIG spells it, please.
> address which causes an interrupt to be received by the CPU.
>
> The MSI capability was first specified in PCI 2.2 and was later enhanced
> in PCI 3.0 to allow each interrupt to be masked individually. The MSI-X
> capability was also introduced with PCI 3.0. It supports more interrupts
> per device than MSI and allows interrupts to be independently configured.
>
> Devices may support both MSI and MSI-X, but only one can be enabled at
> a time.
>
>
> 3. Why use MSIs?
>
...
>
> 4. How to use MSIs
>
> PCI devices are initialised to use pin-based interrupts. The device
> driver has to set up the device to use MSI or MSI-X. Not all machines
> support MSIs correctly, and for those machines, the APIs described below
> will simply fail and the device will continue to use pin-based interrupts.
>
> 4.1 Include kernel support for MSIs
>
> To support MSI or MSI-X, the kernel must be built with the CONFIG_PCI_MSI
> option enabled. This option is only available on some architectures,
> and it may depend on some other options also being set. For example,
> on x86, you must also enable X86_UP_APIC or SMP in order to see the
> CONFIG_PCI_MSI option.
in 2.6.27-rc8, it needs both X86_LOCAL_APIC and X86_IO_APIC, but SMP will do that.
> 4.2 Using MSI
>
> Most of the hard work is done for the driver in the PCI layer. It simply
> has to request that the PCI layer set up the MSI capability for this
> device.
>
> 4.2.1 pci_enable_msi
>
...
>
> 4.2.2 pci_disable_msi
>
> void pci_disable_msi(struct pci_dev *dev)
>
> This function should be used to undo the effect of pci_enable_msi().
> Calling it restores dev->irq to the pin-based interrupt number and frees
> the previously allocated message signaled interrupt(s). The interrupt
> may subsequently be assigned to another device, so drivers should not
> cache the value of pdev->irq.
s/pdev/dev/
>
> A device driver must always call free_irq() on the interrupt(s)
> for which it has called request_irq() before calling this function.
> Failure to do so will result in a BUG_ON(), the device will be left with
s/,/;/ to kill run-on sentences.
> MSI enabled and will leak its vector.
>
> 4.3 Using MSI-X
>
> The MSI-X capability is much more flexible than the MSI capability.
> It supports up to 2048 interrupts, each of which can be separately
> assigned. To support this flexibility, drivers must use an array of
> `struct msix_entry':
>
> struct msix_entry {
> u16 vector; /* kernel uses to write alloc vector */
> u16 entry; /* driver uses to specify entry */
> };
>
> This allows for the device to use these interrupts in a sparse fashion;
> for example it could use interrupts 3 and 1027 and allocate only a
> two-element array. The driver is expected to fill in the 'entry' value
> in each element of the array to indicate which entries it wants the kernel
> to assign interrupts for. It is invalid to fill in two entries with the
> same number.
so in this example, the driver would set msix[0].entry = 3 and
msix[1].entry = 1-27 (??)
and then continue with (below...):
>
> 4.3.1 pci_enable_msix
>
> int pci_enable_msix(struct pci_dev *dev, struct msix_entry *entries, int nvec)
>
> Calling this function asks the PCI subsystem to allocate 'nvec' MSIs.
> The 'entries' argument is a pointer to an array of msix_entry structs
> which should be at least 'nvec' entries in size. On success, the
> function will return 0 and the device will have been switched into
> MSI-X interrupt mode. The 'vector' elements in each entry will have
> been filled in with the interrupt number. The driver should then call
> request_irq() for each 'vector' that it decides to use.
>
> If this function returns a negative number, it indicates an error and
> the driver should not attempt to allocate any more MSI-X interrupts for
> this device. If it returns a positive number, it indicates the maximum
> number of interrupt vectors that could have been allocated.
>
> This function, in contrast with pci_enable_msi(), does not adjust
> pdev->irq. The device will not generate interrupts for this interrupt
s/pdev/dev/ /* or change the parameter name */
> number once MSI-X is enabled. The device driver is responsible for
> keeping track of the interrupts assigned to the MSI-X vectors so it can
> free them again later.
>
> Device drivers should normally call this function once per device
> during the initialization phase.
or initialisation ? (was "initialised" above)
>
> 4.3.2 pci_disable_msix
>
> void pci_disable_msix(struct pci_dev *dev)
>
> This API should be used to undo the effect of pci_enable_msix(). It frees
> the previously allocated message signaled interrupts. The interrupts may
Good here ;)
> subsequently be assigned to another device, so drivers should not cache
> the value of the 'vector' elements over a call to pci_disable_msix().
>
> A device driver must always call free_irq() on the interrupt(s)
> for which it has called request_irq() before calling this function.
> Failure to do so will result in a BUG_ON(), the device will be left with
s/,/;/ to avoid run-on sentences.
> MSI enabled and will leak its vector.
>
> 4.3.3 The MSI-X Table
>
> The MSI-X capability specifies a BAR and offset within that BAR for the
> MSI-X Table. This address is mapped by the PCI subsystem, and should not
> be accessed directly by the device driver. If the driver wishes to
> mask or unmask an interrupt, it should call disable_irq() / enable_irq().
>
> 4.4 Handling devices implementing both MSI and MSI-X capabilities
>
...
>
> 4.5 Considerations when using MSIs
>
> 4.5.1 Choosing between MSI-X and MSI
>
> If your device supports both MSI-X and MSI capabilities, you should use
> the MSI-X facilities in preference to the MSI facilities. As mentioned
> above, MSI-X supports any number of interrupts between 1 and 2048.
> In constrast, MSI is restricted to a maximum of 32 interrupts (and
> must be a power of two). In addition, the MSI interrupt vectors must
> be allocated consecutively, so the system may not be able to allocate
> as many vectors for MSI as it could for MSI-X. On some platforms, MSI
> interrupts must all be targetted at the same set of CPUs whereas MSI-X
> interrupts can all be targetted at different CPUs.
Yes, I would spell it "targeted", but you are writing it, so it's OK.
>
> 4.5.2 Spinlocks
>
...
>
> 4.6 How to tell whether MSI/MSI-X is enabled on a device
>
> Using lspci -v (as root) will show some devices with "Message Signalled
Is that how 'lspci' spells it? Ugh, it is. oh well.
> Interrupts" and others with "MSI-X". Each of these capabilities have an
> 'Enable' flag which will be followed with either "+" (enabled) or "-"
> (disabled).
>
>
> 5. MSI quirks
>
> Several PCI chipsets or devices are known not to support MSIs.
> The PCI stack provides three ways to disable MSIs:
>
> 1. globally
> 2. on all devices behind a specific bridge
> 3. on a single device
>
> 5.1. Disabling MSIs globally
>
...
>
> 5.2. Disabling MSIs below a bridge
>
> Some PCI bridges are not able to route MSIs between busses properly.
> In this case, MSIs must be disabled on all devices behind the bridge.
>
> Some bridges allow you to enable MSIs by changing some bits in their
> PCI configuration space (especially the Hypertransport chipsets such
> as the nVidia nForce and Serverworks HT2000). As with host chipsets,
> Linux mostly knows about them and automatically enables MSIs if it can.
> If you have a bridge which Linux doesn't yet know about, you can enable
> MSIs in configuration space using whatever method you know works, then
> enable MSIs on that bridge by doing:
>
> echo 1 > /sys/bus/pci/devices/$bridge/msi_bus
>
> where $bridge is the PCI address of the bridge you've enabled (eg
preferred: e.g.
> 0000:00:0e.0).
>
> To disable MSIs, echo 0 instead of 1. Changing this value should be
> done with caution as it can break interrupt handling for all devices
> below this bridge.
>
> Again, please notify linux-pci@xxxxxxxxxxxxxxx of any bridges that need
> special handling.
>
> 5.3. Disabling MSIs on a single device
>
> Some devices are known to have faulty MSI implementations. Usually this
> is handled in the individual device driver but occasionally it's necessary
> to handle this with a quirk. Some drivers have an option to disable MSIs;
> this is deprecated.
>
> 5.4. Finding why MSIs are disabled on a device
>
...
---
~Randy
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
