On Mon, 30 Dec 2024 21:03:30 +0000 Precific <precification@xxxxxxxxx> wrote: > On 23.12.24 17:59, Peter Xu wrote: > > On Mon, Dec 23, 2024 at 07:37:46AM +0000, Athul Krishna wrote: > >> Can confirm. Reverting f9e54c3a2f5b from v6.13-rc1 fixed the problem. > >>> > >>> Device: Asus Zephyrus GA402RJ > >>> CPU: Ryzen 7 6800HS > >>> GPU: RX 6700S > >>> Kernel: 6.13.0-rc3-g8faabc041a00 > >>> > >>> Problem: > >>> Launching games or gpu bench-marking tools in qemu windows 11 vm will cause > >>> screen artifacts, ultimately qemu will pause with unrecoverable error. > > > > Is there more information on what setup can reproduce it? > > > > For example, does it only happen with Windows guests? Does the GPU > > vendor/model matter? > > In my case, both Windows and Linux guests fail to initialize the GPU in > the first place since 6.12; QEMU does not crash. I also found commit > f9e54c3a2f5b79ecc57c7bc7d0d3521e461a2101 by bisection. > > CPU: AMD 7950X3D > GPU (guest): AMD RX 6700XT (12GB) > Mainboard: ASRock X670E Steel Legend > Kernel: 6.12.0-rc0 .. 6.13.0-rc2 > > Based on a handful of reports on the Arch forum and on r/vfio, my guess > is that affected users have Resizable BAR or similar settings enabled in > the firmware, which usually applies the maximum possible BAR size > advertised by the GPU on startup. Non-2^n-sized VRAM buffers may be > another commonality. > > Some other reports I found that could fit to this regression: > [1] https://bbs.archlinux.org/viewtopic.php?id=301352 > - Several reports (besides mine), not clear which of those cases are > triggered by the vfio-pci commit. One case is clearly caused by a > different commit in KVM. Potential candidates for the vfio-pci commit > (speculation): (a) 6700XT GPU; (b) 5950X CPU, RTX 3090 GPU > [2] https://old.reddit.com/r/VFIO/comments/1hkvedq/ > - Two users, 7900XT and 7900XTX, reported that reverting 6.12 or > disabling ReBAR resolves Windows guest GPU initialization. > > On my 6700XT (PCI address 03:00.0, 12GB of VRAM), I get the following > Resizable BAR default configuration with the host firmware/UEFI setting > enabled: > > [root]# lspci -s 03:00.0 -vv > ... > Capabilities: [200 v1] Physical Resizable BAR > BAR 0: current size: 16GB, supported: 256MB 512MB 1GB 2GB 4GB 8GB 16GB > BAR 2: current size: 256MB, supported: 2MB 4MB 8MB 16MB 32MB 64MB 128MB > 256MB > ... > > The 16GB configuration above fails with 6.12 (unless I revert commit > f9e54c3a2f5b79ecc57c7bc7d0d3521e461a2101). > Now, if I change BAR 0 to 8GB (as below), which is below the GPU's VRAM > size of 12GB, the Linux guest manages to initialize the GPU. Interesting test. > [root]# echo "0000:03:00.0" > /sys/bus/pci/drivers/vfio-pci/unbind > [root]# #13: 8GB, 14: 16GB > [root]# echo 13 > /sys/bus/pci/devices/0000:03:00.0/resource0_resize > [root]# echo "0000:03:00.0" > /sys/bus/pci/drivers/vfio-pci/bind > > On my mainboard, 'Resizable BAR off' sets BAR 0 to 256MB, which also > works with 6.12. > > Only the size of BAR 0 (VRAM) appears to be relevant here. BAR 2 sizes > of 2MB vs. 256MB have no effect on the outcome. > > > > >>> > >>> Commit: > >>> f9e54c3a2f5b79ecc57c7bc7d0d3521e461a2101 is the first bad commit > >>> commit f9e54c3a2f5b79ecc57c7bc7d0d3521e461a2101 > >>> Author: Alex Williamson <alex.williamson@xxxxxxxxxx> > >>> Date: Mon Aug 26 16:43:53 2024 -0400 > >>> > >>> vfio/pci: implement huge_fault support > > > > Personally I have no clue yet on how this could affect it. I was initially > > worrying on any implicit cache mode changes on the mappings, but I don't > > think any of such was involved in this specific change. > > > > This commit majorly does two things: (1) allow 2M/1G mappings for BARs > > instead of small 4Ks always, and (2) always lazy faults rather than > > "install everything in the 1st fault". Maybe one of the two could have > > some impact in some way. > > In my case, commenting out (1) the huge_fault callback assignment from > f9e54c3a2f5b suffices for GPU initialization in the guest, even if (2) > the 'install everything' loop is still removed. > > I have uploaded host kernel logs with vfio-pci-core debugging enabled > (one log with stock sources, one large log with vfio-pci-core's > huge_fault handler patched out): > https://bugzilla.kernel.org/show_bug.cgi?id=219619#c1 > I'm not sure if the logs of handled faults say much about what > specifically goes wrong here, though. > > The dmesg portion attached to my mail is of a Linux guest failing to > initialize the GPU (BAR 0 size 16GB with 12GB of VRAM). Thanks for the logs with debugging enabled. Would you be able to repeat the test with QEMU 9.2? There's a patch in there that aligns the mmaps, which should avoid mixing 1G and 2MB pages for huge faults. With this you should only see order 18 mappings for BAR0. Also, in a different direction, it would be interesting to run tests disabling 1G huge pages and 2MB huge pages independently. The following would disable 1G pages: diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index 1ab58da9f38a..dd3b748f9d33 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -1684,7 +1684,7 @@ static vm_fault_t vfio_pci_mmap_huge_fault(struct vm_fault *vmf, PFN_DEV), false); break; #endif -#ifdef CONFIG_ARCH_SUPPORTS_PUD_PFNMAP +#if 0 case PUD_ORDER: ret = vmf_insert_pfn_pud(vmf, __pfn_to_pfn_t(pfn + pgoff, PFN_DEV), false); This should disable 2M pages: diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index 1ab58da9f38a..d7dd359e19bb 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -1678,7 +1678,7 @@ static vm_fault_t vfio_pci_mmap_huge_fault(struct vm_fault *vmf, case 0: ret = vmf_insert_pfn(vma, vmf->address, pfn + pgoff); break; -#ifdef CONFIG_ARCH_SUPPORTS_PMD_PFNMAP +#if 0 case PMD_ORDER: ret = vmf_insert_pfn_pmd(vmf, __pfn_to_pfn_t(pfn + pgoff, PFN_DEV), false); And applying both together should be functionally equivalent to pre-v6.12. Thanks, Alex
