On Thu, Feb 08, 2024 at 03:23:21PM +0200, Ilpo Järvinen wrote:
> On runtime resume, pci_dev_wait() is called:
> pci_pm_runtime_resume()
> pci_pm_bridge_power_up_actions()
> pci_bridge_wait_for_secondary_bus()
> pci_dev_wait()
>
> While a device is runtime suspended along with its PCI hierarchy, the
> device could get disconnected. In such case, the link will not come up
> no matter how long pci_dev_wait() waits for it.
>
> Besides the above mentioned case, there could be other ways to get the
> device disconnected while pci_dev_wait() is waiting for the link to
> come up.
>
> Make pci_dev_wait() to exit if the device is already disconnected to
> avoid unnecessary delay.
>
> The use cases of pci_dev_wait() boil down to two:
> 1. Waiting for the device after reset
> 2. pci_bridge_wait_for_secondary_bus()
>
> The callers in both cases seem to benefit from propagating the
> disconnection as error even if device disconnection would be more
> analoguous to the case where there is no device in the first place
> which return 0 from pci_dev_wait(). In the case 2, it results in
> unnecessary marking of the devices disconnected again but that is
> just harmless extra work.
>
> Also make sure compiler does not become too clever with
> dev->error_state and use READ_ONCE() to force a fetch for the
> up-to-date value.
>
> Reported-by: Mika Westerberg <mika.westerberg@xxxxxxxxxxxxxxx>
> Tested-by: Mika Westerberg <mika.westerberg@xxxxxxxxxxxxxxx>
> Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@xxxxxxxxxxxxxxx>
Applied to pci/enumeration for v6.10, thanks!
> ---
>
> v2:
>
> Sent independent of the other patch.
>
> Return -ENOTTY instead of 0 because it aligns better with the
> expecations of the reset use case and only causes unnecessary
> disconnect marking in the pci_bridge_wait_for_secondary_bus()
> case for devices that are already marked disconnected.
>
> drivers/pci/pci.c | 5 +++++
> drivers/pci/pci.h | 9 ++++++++-
> 2 files changed, 13 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
> index ca4159472a72..14c57296a0aa 100644
> --- a/drivers/pci/pci.c
> +++ b/drivers/pci/pci.c
> @@ -1250,6 +1250,11 @@ static int pci_dev_wait(struct pci_dev *dev, char *reset_type, int timeout)
> for (;;) {
> u32 id;
>
> + if (pci_dev_is_disconnected(dev)) {
> + pci_dbg(dev, "disconnected; not waiting\n");
> + return -ENOTTY;
> + }
> +
> pci_read_config_dword(dev, PCI_COMMAND, &id);
> if (!PCI_POSSIBLE_ERROR(id))
> break;
> diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h
> index 2336a8d1edab..58a32d2d2e96 100644
> --- a/drivers/pci/pci.h
> +++ b/drivers/pci/pci.h
> @@ -4,6 +4,8 @@
>
> #include <linux/pci.h>
>
> +#include <asm/rwonce.h>
> +
> /* Number of possible devfns: 0.0 to 1f.7 inclusive */
> #define MAX_NR_DEVFNS 256
>
> @@ -370,7 +372,12 @@ static inline int pci_dev_set_disconnected(struct pci_dev *dev, void *unused)
>
> static inline bool pci_dev_is_disconnected(const struct pci_dev *dev)
> {
> - return dev->error_state == pci_channel_io_perm_failure;
> + /*
> + * error_state is set in pci_dev_set_io_state() using xchg/cmpxchg()
> + * and read w/o common lock. READ_ONCE() ensures compiler cannot cache
> + * the value (e.g. inside the loop in pci_dev_wait()).
> + */
> + return READ_ONCE(dev->error_state) == pci_channel_io_perm_failure;
> }
>
> /* pci_dev priv_flags */
> --
> 2.39.2
>
