On 2019-04-24 8:11 a.m., Bjorn Helgaas wrote:
[+cc Mika for runtime PM of bridges, Logan for switchtec question]
On Wed, Apr 24, 2019 at 01:01:02PM +0300, Alexander Fomichev wrote:
On Tue, Apr 23, 2019 at 04:53:40PM -0500, Bjorn Helgaas wrote:
On Mon, Apr 15, 2019 at 09:15:54AM -0500, Bjorn Helgaas wrote:
On Mon, Apr 15, 2019 at 04:59:03PM +0300, Alexander Fomichev wrote:
PLX switches have an issue that their internal registers
become inaccessible when runtime PM is enabled. Therefore PLX
service tools can't communicate with the hardware. A kernel
option "pcie_port_pm=off" can be used as a workaround. But it
affects all the devices.
So this solution is to add PLX switch devices to the quirk
list for disabling runtime PM only for them.
I assume the problem is actually that the config space registers
are inaccessible when the device is in D3hot?
Reading this again, I realize you said "internal registers". I
don't know whether that actually means config space registers
(which *should* work even when the device is in D3hot (see the
PCIe reference below and PCI Power Management Spec r1.2, sec
5.4.1)), or MMIO registers (which are not expected to work while
in D3hot).
If the service tools read MMIO registers, presumably that goes
through some driver that should be able to manage runtime PM. Or,
if there's no driver, I think your service tool could prevent
runtime power management by changing
/sys/devices/.../power/control to "on" (see
Documentation/power/runtime_pm.txt).
You're right. Config space registers are accessible. The driver
can't read/write MMIO registers (Device-Specific Registers as
they're called by Broadcom).
Ah, perfect. That's exactly what's supposed to happen from a PCI
hardware point of view. Unfortunately I don't know much about how
Linux power management works, but Rafael and Mika do.
How do your service tools access these MMIO registers?
- via a PLX driver that provides read/write/ioctl on a char dev?
- read/write on /sys/devices/pci*/.../resource<x> ?
- mmap on /sys/devices/pci*/.../resource<x> ?
- read/write on /dev/mem?
- mmap on /dev/mem?
- something else?
I think there are several ways we might be able to fix this:
- Write a driver along the lines of drivers/pci/switch/switchtec.c.
I don't see any runtime PM stuff in that driver, so maybe it's
magically taken care of by the PM/PCI core? There might also be
an issue if both portdrv and your driver want to claim the same
device. I don't know how switchtec deals with that.
Right, the switchtec device puts all its management MMIO registers
behind a separate PCIe function which lets us bind a different driver
and not conflict with the portdrv. I've noticed the PLX parts have an
extra unused BAR in their upstream port function which means it will be
an annoying hack to write a driver to use it seeing the portdrv needs to
be registered to it.
Logan
