During suspend on an SMP system, {read,write}_msi_msg_desc() may be
called to mask and unmask interrupts on a device that is already in a
reduced power state. At this point memory-mapped registers including
MSI-X tables are not accessible, and config space may not be fully
functional either.
While a device is in a reduced power state its interrupts are
effectively masked and its MSI(-X) state will be restored when it is
brought back to D0. Therefore these functions can simply read and
write msi_desc::msg for devices not in D0.
Further, read_msi_msg_desc() should only ever be used to update a
previously written message, so it can always read msi_desc::msg
and never needs to touch the hardware.
Signed-off-by: Ben Hutchings <bhutchings@xxxxxxxxxxxxxx>
---
On Mon, 2010-06-14 at 18:13 -0700, Michael Chan wrote:
> I'm debugging the bnx2 driver which doesn't work after suspend/resume if
> it is running in MSI-X mode. The problem is that during suspend, the
> MSI-X vectors are disabled by the following sequence on x86:
>
> take_cpu_down() -> cpu_disable_common() -> fixup_irqs()
>
> The MSI-X address/data used to disable the vectors are remembered in the
> above sequence. During resume, these address/data are then programmed
> back to the device during pci_restore_state(), causing all the vectors
> to remain disabled.
That's not quite what I see. What I see is that the message is read
back from the table *after* the driver's suspend method has been called.
At this point the device is already in D3 and memory-mapped registers
are not accessible, so we get random bits as the message. At least,
that's what I see happening with the sfc driver.
> Some drivers call free_irq() during suspend and request_irq() during
> resume, and that should avoid the problem. bnx2 and some other drivers
> do not do that. These drivers rely on pci_restore_state() to restore
> the MSI-X vectors to the same working state before suspend.
>
> What's the right way to fix this? Thanks.
This is my attempt, which works for sfc. See if it works for bnx2.
Ben.
drivers/pci/msi.c | 34 +++++++++++-----------------------
1 files changed, 11 insertions(+), 23 deletions(-)
diff --git a/drivers/pci/msi.c b/drivers/pci/msi.c
index 77b68ea..03f04dc 100644
--- a/drivers/pci/msi.c
+++ b/drivers/pci/msi.c
@@ -196,30 +196,15 @@ void unmask_msi_irq(unsigned int irq)
void read_msi_msg_desc(struct irq_desc *desc, struct msi_msg *msg)
{
struct msi_desc *entry = get_irq_desc_msi(desc);
- if (entry->msi_attrib.is_msix) {
- void __iomem *base = entry->mask_base +
- entry->msi_attrib.entry_nr * PCI_MSIX_ENTRY_SIZE;
- msg->address_lo = readl(base + PCI_MSIX_ENTRY_LOWER_ADDR);
- msg->address_hi = readl(base + PCI_MSIX_ENTRY_UPPER_ADDR);
- msg->data = readl(base + PCI_MSIX_ENTRY_DATA);
- } else {
- struct pci_dev *dev = entry->dev;
- int pos = entry->msi_attrib.pos;
- u16 data;
+ /* We do not touch the hardware (which may not even be
+ * accessible at the moment) but return the last message
+ * written. Assert that this is valid, assuming that
+ * valid messages are not all-zeroes. */
+ BUG_ON(!(entry->msg.address_hi | entry->msg.address_lo |
+ entry->msg.data));
- pci_read_config_dword(dev, msi_lower_address_reg(pos),
- &msg->address_lo);
- if (entry->msi_attrib.is_64) {
- pci_read_config_dword(dev, msi_upper_address_reg(pos),
- &msg->address_hi);
- pci_read_config_word(dev, msi_data_reg(pos, 1), &data);
- } else {
- msg->address_hi = 0;
- pci_read_config_word(dev, msi_data_reg(pos, 0), &data);
- }
- msg->data = data;
- }
+ *msg = entry->msg;
}
void read_msi_msg(unsigned int irq, struct msi_msg *msg)
@@ -232,7 +217,10 @@ void read_msi_msg(unsigned int irq, struct msi_msg *msg)
void write_msi_msg_desc(struct irq_desc *desc, struct msi_msg *msg)
{
struct msi_desc *entry = get_irq_desc_msi(desc);
- if (entry->msi_attrib.is_msix) {
+
+ if (entry->dev->current_state != PCI_D0) {
+ /* Don't touch the hardware now */
+ } else if (entry->msi_attrib.is_msix) {
void __iomem *base;
base = entry->mask_base +
entry->msi_attrib.entry_nr * PCI_MSIX_ENTRY_SIZE;
--
1.6.2.5
--
Ben Hutchings, Senior Software Engineer, Solarflare Communications
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
