From: Dexuan Cui <decui@xxxxxxxxxxxxx> Sent: Tuesday, April 19, 2022 3:00 PM
>
> A VM on Azure can have 14 GPUs, and each GPU may have a huge MMIO BAR,
> e.g. 128 GB. Currently the boot time of such a VM can be 4+ minutes, and
> most of the time is used by the host to unmap/map the vBAR from/to pBAR
> when the VM clears and sets the PCI_COMMAND_MEMORY bit: each unmap/map
> operation for a 128GB BAR needs about 1.8 seconds, and the pci-hyperv
> driver and the Linux PCI subsystem flip the PCI_COMMAND_MEMORY bit
> eight times (see pci_setup_device() -> pci_read_bases() and
> pci_std_update_resource()), increasing the boot time by 1.8 * 8 = 14.4
> seconds per GPU, i.e. 14.4 * 14 = 201.6 seconds in total.
>
> Fix the slowness by not turning on the PCI_COMMAND_MEMORY in pci-hyperv.c,
> so the bit stays in the off state before the PCI device driver calls
> pci_enable_device(): when the bit is off, pci_read_bases() and
> pci_std_update_resource() don't cause Hyper-V to unmap/map the vBARs.
> With this change, the boot time of such a VM is reduced by
> 1.8 * (8-1) * 14 = 176.4 seconds.
>
> Tested-by: Boqun Feng (Microsoft) <boqun.feng@xxxxxxxxx>
> Signed-off-by: Dexuan Cui <decui@xxxxxxxxxxxxx>
> Cc: Jake Oshins <jakeo@xxxxxxxxxxxxx>
> ---
> drivers/pci/controller/pci-hyperv.c | 17 +++++++++++------
> 1 file changed, 11 insertions(+), 6 deletions(-)
>
> diff --git a/drivers/pci/controller/pci-hyperv.c b/drivers/pci/controller/pci-hyperv.c
> index d270a204324e..f9fbbd8d94db 100644
> --- a/drivers/pci/controller/pci-hyperv.c
> +++ b/drivers/pci/controller/pci-hyperv.c
> @@ -2082,12 +2082,17 @@ static void prepopulate_bars(struct hv_pcibus_device
> *hbus)
> }
> }
> if (high_size <= 1 && low_size <= 1) {
> - /* Set the memory enable bit. */
> - _hv_pcifront_read_config(hpdev, PCI_COMMAND, 2,
> - &command);
> - command |= PCI_COMMAND_MEMORY;
> - _hv_pcifront_write_config(hpdev, PCI_COMMAND, 2,
> - command);
> + /*
> + * No need to set the PCI_COMMAND_MEMORY bit as
> + * the core PCI driver doesn't require the bit
> + * to be pre-set. Actually here we intentionally
> + * keep the bit off so that the PCI BAR probing
> + * in the core PCI driver doesn't cause Hyper-V
> + * to unnecessarily unmap/map the virtual BARs
> + * from/to the physical BARs multiple times.
> + * This reduces the VM boot time significantly
> + * if the BAR sizes are huge.
> + */
> break;
> }
> }
> --
> 2.17.1
Reviewed-by: Michael Kelley <mikelley@xxxxxxxxxxxxx>
