On Mon, 25 Nov 2024 16:46:29 -0600 Mitchell Augustin <mitchell.augustin@xxxxxxxxxxxxx> wrote: > Hello, > > I've been working on a bug regarding slow PCI initialization and BAR > assignment times for Nvidia GPUs passed-through to VMs on our DGX H100 > that I originally believed to be an issue in OVMF, but upon further > investigation, I'm now suspecting that it may be an issue somewhere in > the kernel. (Here is the original edk2 mailing list thread, with extra > context: https://edk2.groups.io/g/devel/topic/109651206) [0] > > > When running the 6.12 kernel on a DGX H100 host with 4 GPUs passed > through using CPU passthrough and this virt-install command[1], VMs > using the latest OVMF version will take around 2 minutes for the guest > kernel to boot and initialize PCI devices/BARs for the GPUs. > Originally, I was investigating this as an issue in OVMF, because GPU > initialization takes much less time when our host is running an OVMF > version with this patch[2] removed (which only calculates the MMIO > window size differently). Without that patch, the guest kernel does > boot quickly, but we can only use the Nvidia GPUs within the guest if > `pci=nocrs pci=realloc` are set in the guest (evidently since the MMIO > windows advertised by OVMF to the kernel without this patch are > incorrect). So, the OVMF patch being present does evidently result in > correct MMIO windows and prevent us from needing those kernel options, > but the VM boot time is much slower. > > > In discussing this, another contributor reported slow PCIe/BAR > initialization times for large BAR Nvidia GPUs in Linux when using VMs > with SeaBIOS as well. This, combined with me not seeing any slowness > when these GPUs are initialized on the host, and the fact that this > slowness only happens when CPU passthrough is used, are leading me to > suspect that this may actually be a problem somewhere in the KVM or > vfio-pci stack. I did also attempt manually setting different MMIO > window sizes using the X-PciMmio64Mb OVMF/QEMU knob, and it seems that > any MMIO window size large enough to accommodate all GPU memory > regions does result in this slower initialization time (but also a > valid mapping). The VM needs to be given enough 64-bit MMIO space for the devices, at which point the BIOS should be able to fully assign the BARs and then pci=nocrs,realloc should not be necessary. > I did some profiling of the guest kernel during boot, and I've > identified that it seems like the most time is spent in this > pci_write_config_word() call in __pci_read_base() of > drivers/pci/probe.c.[3] Each of those pci_write_config_word() calls > takes about 2 seconds, but it adds up to a significant chunk of the > initialization time since __pci_read_base() is executed somewhere > between 20-40 times in my VM boot. A lot happens in the VMM when the memory enable bit is set in the command register. This is when the MMIO BARs of the device enter the AddressSpace in QEMU and are caught by the MemoryListener to create DMA mappings though the IOMMU. The BAR space is walked, faulted, and mapped. I'm sure you're at least experiencing scaling issues of that with 128GB BARs. Are you seeing __pci_read_base() called 20-40 times repeatedly for a given device or in total for the VM boot. Minimally it needs to be called once for every PCI device, so it's not clear to me if you're reporting something excessive based on using pci=realloc or there are just enough devices to justify this many calls. > As a point of comparison, I measured the time it took to hot-unplug > and re-plug these GPUs after the VM booted, and observed much more > reasonable times (under 5s for each GPU to re-initialize its memory > regions). I've also been trying to get this hotplugging working in VMs > where the GPUs aren't initially attached at boot, but in any such > configuration, the memory regions for the PCI slots that the GPUs get > attached to during hotplug are too small for the full 128GB these GPUs > require (and I have yet to figure out a way to fix that. More details > on that in [0]). If 5s is the measure of enabling one GPU, the VM BIOS will need to do that at least once and the guest OS will need to do that at least once, we're up to 4 GPUs * 5s * 2 times = 40s. If the guest OS toggles memory enable more than once, your 2 minute boot time isn't sounding too far off. Then there's also the fact that the VM appears to be given 900+GB of RAM, which also needs to be pinned and mapped for DMA and there's no indication I can find in virt-install command of using hugepages. There are essentially two things in play here, how long does it take to map that BAR into the VM address space, and how many times is it done. I know there's room to improve the former. We just recently added kernel support for pgd and pmd large page mapping for pfnmap regions and QEMU 9.2-rc includes alignment of vfio-pci BARs to try to optimize that towards pgd where possible. We're still trying to pin individual pages though, so percolating the mapping sizes up through the vfio IOMMU backend could help. Thanks, Alex
