Re: [PATCH] PCI: Release unused bridge resources during resize

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Tue, 7 May 2024, Alex Williamson wrote:

> Resizing BARs can be blocked when a device in the bridge hierarchy
> itself consumes resources from the resized range.  This scenario is
> common with Intel Arc DG2 GPUs where the following is a typical
> topology:
> 
>  +-[0000:5d]-+-00.0-[5e-61]----00.0-[5f-61]--+-01.0-[60]----00.0  Intel Corporation DG2 [Arc A380]
>                                              \-04.0-[61]----00.0  Intel Corporation DG2 Audio Controller
> 
> Here the system BIOS has provided a large 64bit, prefetchable window:
> 
> pci_bus 0000:5d: root bus resource [mem 0xb000000000-0xbfffffffff window]
> 
> But only a small portion is programmed into the root port aperture:
> 
> pci 0000:5d:00.0:   bridge window [mem 0xbfe0000000-0xbff07fffff 64bit pref]
> 
> The upstream port then provides the following aperture:
> 
> pci 0000:5e:00.0:   bridge window [mem 0xbfe0000000-0xbfefffffff 64bit pref]
> 
> With the missing range found to be consumed by the switch port itself:
> 
> pci 0000:5e:00.0: BAR 0 [mem 0xbff0000000-0xbff07fffff 64bit pref]
> 
> The downstream port above the GPU provides the same aperture as upstream:
> 
> pci 0000:5f:01.0:   bridge window [mem 0xbfe0000000-0xbfefffffff 64bit pref]
> 
> Which is entirely consumed by the GPU:
> 
> pci 0000:60:00.0: BAR 2 [mem 0xbfe0000000-0xbfefffffff 64bit pref]
> 
> In summary, iomem reports the following:
> 
> b000000000-bfffffffff : PCI Bus 0000:5d
>   bfe0000000-bff07fffff : PCI Bus 0000:5e
>     bfe0000000-bfefffffff : PCI Bus 0000:5f
>       bfe0000000-bfefffffff : PCI Bus 0000:60
>         bfe0000000-bfefffffff : 0000:60:00.0
>     bff0000000-bff07fffff : 0000:5e:00.0
> 
> The GPU at 0000:60:00.0 supports a Resizable BAR:
> 
> 	Capabilities: [420 v1] Physical Resizable BAR
> 		BAR 2: current size: 256MB, supported: 256MB 512MB 1GB 2GB 4GB 8GB
> 
> However when attempting a resize we get -ENOSPC:
> 
> pci 0000:60:00.0: BAR 2 [mem 0xbfe0000000-0xbfefffffff 64bit pref]: releasing
> pcieport 0000:5f:01.0: bridge window [mem 0xbfe0000000-0xbfefffffff 64bit pref]: releasing
> pcieport 0000:5e:00.0: bridge window [mem 0xbfe0000000-0xbfefffffff 64bit pref]: releasing
> pcieport 0000:5e:00.0: bridge window [mem size 0x200000000 64bit pref]: can't assign; no space
> pcieport 0000:5e:00.0: bridge window [mem size 0x200000000 64bit pref]: failed to assign
> pcieport 0000:5f:01.0: bridge window [mem size 0x200000000 64bit pref]: can't assign; no space
> pcieport 0000:5f:01.0: bridge window [mem size 0x200000000 64bit pref]: failed to assign
> pci 0000:60:00.0: BAR 2 [mem size 0x200000000 64bit pref]: can't assign; no space
> pci 0000:60:00.0: BAR 2 [mem size 0x200000000 64bit pref]: failed to assign
> pcieport 0000:5d:00.0: PCI bridge to [bus 5e-61]
> pcieport 0000:5d:00.0:   bridge window [mem 0xb9000000-0xba0fffff]
> pcieport 0000:5d:00.0:   bridge window [mem 0xbfe0000000-0xbff07fffff 64bit pref]
> pcieport 0000:5e:00.0: PCI bridge to [bus 5f-61]
> pcieport 0000:5e:00.0:   bridge window [mem 0xb9000000-0xba0fffff]
> pcieport 0000:5e:00.0:   bridge window [mem 0xbfe0000000-0xbfefffffff 64bit pref]
> pcieport 0000:5f:01.0: PCI bridge to [bus 60]
> pcieport 0000:5f:01.0:   bridge window [mem 0xb9000000-0xb9ffffff]
> pcieport 0000:5f:01.0:   bridge window [mem 0xbfe0000000-0xbfefffffff 64bit pref]
> pci 0000:60:00.0: BAR 2 [mem 0xbfe0000000-0xbfefffffff 64bit pref]: assigned
> 
> In this example we need to resize all the way up to the root port
> aperture, but we refuse to change the root port aperture while resources
> are allocated for the upstream port BAR.
> 
> The solution proposed here builds on the idea in commit 91fa127794ac
> ("PCI: Expose PCIe Resizable BAR support via sysfs") where the BAR can
> be resized while there is no driver attached.  In this case, when there
> is no driver bound to the upstream switch port we'll release resources
> of the bridge which match the reallocation.  Therefore we can achieve
> the below successful resize operation by unbinding 0000:5e:00.0 from the
> pcieport driver before invoking the resource2_resize interface on the
> GPU at 0000:60:00.0.
> 
> pci 0000:60:00.0: BAR 2 [mem 0xbfe0000000-0xbfefffffff 64bit pref]: releasing
> pcieport 0000:5f:01.0: bridge window [mem 0xbfe0000000-0xbfefffffff 64bit pref]: releasing
> pci 0000:5e:00.0: bridge window [mem 0xbfe0000000-0xbfefffffff 64bit pref]: releasing
> pci 0000:5e:00.0: BAR 0 [mem 0xbff0000000-0xbff07fffff 64bit pref]: releasing
> pcieport 0000:5d:00.0: bridge window [mem 0xbfe0000000-0xbff07fffff 64bit pref]: releasing
> pcieport 0000:5d:00.0: bridge window [mem 0xb000000000-0xb2ffffffff 64bit pref]: assigned
> pci 0000:5e:00.0: bridge window [mem 0xb000000000-0xb1ffffffff 64bit pref]: assigned
> pci 0000:5e:00.0: BAR 0 [mem 0xb200000000-0xb2007fffff 64bit pref]: assigned
> pcieport 0000:5f:01.0: bridge window [mem 0xb000000000-0xb1ffffffff 64bit pref]: assigned
> pci 0000:60:00.0: BAR 2 [mem 0xb000000000-0xb1ffffffff 64bit pref]: assigned
> pci 0000:5e:00.0: PCI bridge to [bus 5f-61]
> pci 0000:5e:00.0:   bridge window [mem 0xb9000000-0xba0fffff]
> pci 0000:5e:00.0:   bridge window [mem 0xb000000000-0xb1ffffffff 64bit pref]
> pcieport 0000:5d:00.0: PCI bridge to [bus 5e-61]
> pcieport 0000:5d:00.0:   bridge window [mem 0xb9000000-0xba0fffff]
> pcieport 0000:5d:00.0:   bridge window [mem 0xb000000000-0xb2ffffffff 64bit pref]
> pci 0000:5e:00.0: PCI bridge to [bus 5f-61]
> pci 0000:5e:00.0:   bridge window [mem 0xb9000000-0xba0fffff]
> pci 0000:5e:00.0:   bridge window [mem 0xb000000000-0xb1ffffffff 64bit pref]
> pcieport 0000:5f:01.0: PCI bridge to [bus 60]
> pcieport 0000:5f:01.0:   bridge window [mem 0xb9000000-0xb9ffffff]
> pcieport 0000:5f:01.0:   bridge window [mem 0xb000000000-0xb1ffffffff 64bit pref]
> 
> Signed-off-by: Alex Williamson <alex.williamson@xxxxxxxxxx>

Yes. Looks another case where an already assigned resource prevents some 
operation from succeeding.

> diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c
> index 909e6a7c3cc3..15fc8e4e84c9 100644
> --- a/drivers/pci/setup-bus.c
> +++ b/drivers/pci/setup-bus.c
> @@ -2226,6 +2226,26 @@ void pci_assign_unassigned_bridge_resources(struct pci_dev *bridge)
>  }
>  EXPORT_SYMBOL_GPL(pci_assign_unassigned_bridge_resources);
>  
> +static void pci_release_resource_type(struct pci_dev *pdev, unsigned long type)
> +{
> +	int i;
> +
> +	if (!device_trylock(&pdev->dev))
> +		return;
> +
> +	if (pdev->dev.driver)

Isn't portdrv bound to bridges so how does this ends up working?

-- 
 i.

> +		goto unlock;
> +
> +	for (i = 0; i < PCI_STD_NUM_BARS; i++) {
> +		if (pci_resource_len(pdev, i) &&
> +		    !((pci_resource_flags(pdev, i) ^ type) & PCI_RES_TYPE_MASK))
> +			pci_release_resource(pdev, i);
> +	}
> +
> +unlock:
> +	device_unlock(&pdev->dev);
> +}
> +
>  int pci_reassign_bridge_resources(struct pci_dev *bridge, unsigned long type)
>  {
>  	struct pci_dev_resource *dev_res;
> @@ -2260,8 +2280,10 @@ int pci_reassign_bridge_resources(struct pci_dev *bridge, unsigned long type)
>  
>  			pci_info(bridge, "%s %pR: releasing\n", res_name, res);
>  
> -			if (res->parent)
> +			if (res->parent) {
>  				release_resource(res);
> +				pci_release_resource_type(bridge, type);
> +			}
>  			res->start = 0;
>  			res->end = 0;
>  			break;
> 




[Index of Archives]     [DMA Engine]     [Linux Coverity]     [Linux USB]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [Greybus]

  Powered by Linux