On Wed, May 25, 2022 at 01:27:21PM +0200, Rafael J. Wysocki wrote:
> On Tue, May 24, 2022 at 10:58 PM Bjorn Helgaas <helgaas@xxxxxxxxxx> wrote:
> >
> > On Tue, May 17, 2022 at 08:43:19PM +0800, Yicong Yang wrote:
> > > When the bus bridge is runtime suspended, we'll fail to rescan
> > > the devices through sysfs as we cannot access the configuration
> > > space correctly when the bridge is in D3hot.
> >
> > Is the "D3hot" above a typo? I think devices are supposed to respond
> > to config accesses when in D3hot. PCIe r6.0, sec 5.3.1.4.1:
> >
> > Configuration and Message requests are the only TLPs accepted by a
> > Function in the D3Hot state. ...
> >
> > Functions that are in D3Hot are permitted to be transitioned by
> > software (writing to their PMCSR PowerState field) to the D0active
> > state or the D0uninitialized state. Functions that are in D3Hot must
> > respond to Configuration Space accesses as long as power and clock
> > are supplied so that they can be returned to D0 by software.
>
> That applies to the device itself, though, and not to the bus under it.
>
> In general, a bridge in D3hot causes the bus segment on the other side
> of it to be inaccessible even for config space accesses AFAICS.
Oh, thank you! That was the piece I was missing. I'll tweak the
commit log to say something like:
A bridge in a non-D0 power state does not forward config accesses to
its secondary side (PCIe r6.0, sec 5.3.1). Make sure the bridge is
in D0 while we enumerate devices below it.
> > > It can be reproduced like:
> > >
> > > $ echo 1 > /sys/bus/pci/devices/0000:80:00.0/0000:81:00.1/remove
> > > $ echo 1 > /sys/bus/pci/devices/0000:80:00.0/pci_bus/0000:81/rescan
> > >
> > > 0000:80:00.0 is a Root Port and it is runtime-suspended, so
> > > 0000:81:00.1 is unreachable after a rescan.
> > >
> > > Power up the bridge when scanning the child bus and allow it to
> > > suspend again by adding pm_runtime_get_sync()/pm_runtime_put()
> > > in pci_scan_child_bus_extend().
> > >
> > > Cc: Rafael J. Wysocki <rafael@xxxxxxxxxx>
> > > Cc: Mika Westerberg <mika.westerberg@xxxxxxxxxxxxxxx>
> > > Cc: Bjorn Helgaas <bhelgaas@xxxxxxxxxx>
> > > Signed-off-by: Yicong Yang <yangyicong@xxxxxxxxxxxxx>
> > > Reviewed-by: Rafael J. Wysocki <rafael.j.wysocki@xxxxxxxxx>
> > > ---
> > > drivers/pci/probe.c | 12 ++++++++++++
> > > 1 file changed, 12 insertions(+)
> > >
> > > diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
> > > index 17a969942d37..b108e72b6586 100644
> > > --- a/drivers/pci/probe.c
> > > +++ b/drivers/pci/probe.c
> > > @@ -2859,11 +2859,20 @@ static unsigned int pci_scan_child_bus_extend(struct pci_bus *bus,
> > > unsigned int used_buses, normal_bridges = 0, hotplug_bridges = 0;
> > > unsigned int start = bus->busn_res.start;
> > > unsigned int devfn, fn, cmax, max = start;
> > > + struct pci_dev *bridge = bus->self;
> > > struct pci_dev *dev;
> > > int nr_devs;
> > >
> > > dev_dbg(&bus->dev, "scanning bus\n");
> > >
> > > + /*
> > > + * Make sure the bus bridge is powered on, otherwise we may not be
> > > + * able to scan the devices as we may fail to access the configuration
> > > + * space of subordinates.
> > > + */
> > > + if (bridge)
> > > + pm_runtime_get_sync(&bridge->dev);
> > > +
> > > /* Go find them, Rover! */
> > > for (devfn = 0; devfn < 256; devfn += 8) {
> > > nr_devs = pci_scan_slot(bus, devfn);
> > > @@ -2976,6 +2985,9 @@ static unsigned int pci_scan_child_bus_extend(struct pci_bus *bus,
> > > }
> > > }
> > >
> > > + if (bridge)
> > > + pm_runtime_put(&bridge->dev);
> > > +
> > > /*
> > > * We've scanned the bus and so we know all about what's on
> > > * the other side of any bridges that may be on this bus plus
> > > --
> > > 2.24.0
> > >
