My lord, can't believe that was the answer, all GPUs are successfully passed through now, thanks Alex! In case it is of interest, here's the resulting dmesg: https://drive.google.com/file/d/1b0f4bWPhJXifC8j9_NtB34R7GBA8Dky3/view For the gal/chap finding this thread in 12 years trying to make an old machine work: In the qemu command I added "-cpu host", current kernel parameters were "rcutree.rcu_idle_gp_delay=1 mem_encrypt=off iommu=off amd_iommu=off pci=realloc=off,check_enable_amd_mmconf". On to the next chapter of this machine, which is to enable MMIO filtering by the hypervisor which is required to make VMs with less than all resources/GPUs. Luckily NVidia provides 0 detailed documentation on the matter. Op vr 16 jul. 2021 om 02:08 schreef Alex Williamson <alex.williamson@xxxxxxxxxx>: > > On Thu, 15 Jul 2021 18:05:06 -0500 > Bjorn Helgaas <helgaas@xxxxxxxxxx> wrote: > > > On Thu, Jul 15, 2021 at 11:39:54PM +0300, Ruben wrote: > > > Thanks for the response, here's a link to the entire dmesg log: > > > https://drive.google.com/file/d/1Uau0cgd2ymYGDXNr1mA9X_UdLoMH_Azn/view > > > > > > Some entries that might be of interest: > > > > ACPI tells us the host bridge windows are: > > > > acpi PNP0A08:00: host bridge window [mem 0x000a0000-0x000bffff window] (ignored) > > acpi PNP0A08:00: host bridge window [mem 0x80000000-0xafffffff window] (ignored) > > acpi PNP0A08:00: host bridge window [mem 0xc0000000-0xfebfffff window] (ignored) > > acpi PNP0A08:00: host bridge window [mem 0x800000000-0xfffffffff window] (ignored) > > > > The 0xc0000000 window is about 1GB and is below 4GB. > > The 0x800000000 window looks like 32GB. > > > > But "pci=nocrs" means we ignore these windows ... > > > > > pci_bus 0000:00: root bus resource [mem 0x00000000-0xffffffffff] > > > > and instead use this 1TB of address space, from which DRAM is > > excluded. I think this is basically everything the CPU can address, > > and I *think* it comes from this in setup_arch(): > > > > iomem_resource.end = (1ULL << boot_cpu_data.x86_phys_bits) - 1; > > > > But you have 8 GPUs, each of which needs 128GB + 32MB + 16MB, so you > > need 1TB + 384MB to map them all, and the CPU can't address that much. > > > > Since you're running this on qemu, I assume x86_phys_bits is telling > > us about the capabilities of the CPU qemu is emulating. Maybe there's > > a way to tell qemu to emulate a CPU with more address bits? > > "-cpu host" perhaps >
