On Tue, 05 Jun 2012 08:27:05 -0600 Alex Williamson <alex.williamson@xxxxxxxxxx> wrote: > On Tue, 2012-06-05 at 12:39 +0200, Andreas Hartmann wrote: > > Hello Alex, > > > > thanks for your efforts! > > > > Maybe, you already know that I'm suffering the same problem :-( with a > > GA-990XA-UD3 (990X chipset). > > My system is a GA-990FXA-UD3, so just a slightly different version of > the chipset. > > > On Mon, 04 Jun 2012 21:44:05 -0600 > > Alex Williamson <alex.williamson@xxxxxxxxxx> wrote: > > [...] > > > I have a setup with an AMD 990FX system and a spare PVR-350 card that I > > > installed to reproduce. The sad answer is that it's nearly impossible > > > to assign PCI devices on these systems due to the aliasing of devices > > > below the PCIe-to-PCI bridge (PCIe devices are much, much easier to > > > assign). If you boot with amd_iommu_dump, you'll see some output like > > > this: > > > > > > AMD-Vi: DEV_ALIAS_RANGE devid: 05:00.0 flags: 00 devid_to: 00:14.4 > > > AMD-Vi: DEV_RANGE_END devid: 05:1f.7 > > > > here: > > AMD-Vi: DEV_ALIAS_RANGE devid: 06:00.0 flags: 00 devid_to: 00:14.4 > > AMD-Vi: DEV_RANGE_END devid: 06:1f.7 > > > > > > This means according to your description, the following devices have to > > be unbound here, too (0000:06:07.0 is the device I want to passthrough): > > > > 0000:00:14.0 -> ../../../../devices/pci0000:00/0000:00:14.0 > > 0000:00:14.1 -> ../../../../devices/pci0000:00/0000:00:14.1 > > 0000:00:14.2 -> ../../../../devices/pci0000:00/0000:00:14.2 > > 0000:00:14.3 -> ../../../../devices/pci0000:00/0000:00:14.3 > > 0000:00:14.4 -> ../../../../devices/pci0000:00/0000:00:14.4 > > 0000:00:14.5 -> ../../../../devices/pci0000:00/0000:00:14.5 > > 0000:06:07.0 -> ../../../../devices/pci0000:00/0000:00:14.4/0000:06:07.0 > > Yep. > > > These are the following additional devices: > > > > 00:14.0 SMBus: Advanced Micro Devices [AMD] nee ATI SBx00 SMBus Controller (rev 42) > > 00:14.1 IDE interface: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 IDE Controller (rev 40) (prog-if 8a [Master SecP PriP]) > > 00:14.2 Audio device: Advanced Micro Devices [AMD] nee ATI SBx00 Azalia (Intel HDA) (rev 40) > > 00:14.3 ISA bridge: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 LPC host controller (rev 40) > > 00:14.4 PCI bridge: Advanced Micro Devices [AMD] nee ATI SBx00 PCI to PCI Bridge (rev 40) (prog-if 01 [Subtractive decode]) > > 00:14.5 USB controller: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 USB OHCI2 Controller (prog-if 10 [OHCI]) > > My system shows this for lspci -n -s 14. > > 00:14.0 0c05: 1002:4385 (rev 42) > 00:14.2 0403: 1002:4383 (rev 40) > 00:14.3 0601: 1002:439d (rev 40) > 00:14.4 0604: 1002:4384 (rev 40) > 00:14.5 0c03: 1002:4399 My system: 00:14.0 0c05: 1002:4385 (rev 42) 00:14.1 0101: 1002:439c (rev 40) 00:14.2 0403: 1002:4383 (rev 40) 00:14.3 0601: 1002:439d (rev 40) 00:14.4 0604: 1002:4384 (rev 40) 00:14.5 0c03: 1002:4399 You miss 00.14.1!? But the others are equal. > Can you confirm yours matches and fill in 14.1 so we have the > vendor:device IDs for these. Yes. > > Among them is the sound device and the USB device - no good idea to > > disable them on a desktop. > > You can always blacklist drivers if the device is unique or unbind > devices in an rc.local type script. I'm hoping that Joerg or someone > else from AMD can tell us this devices does not allow internal > peer-to-peer between functions. Then we can add it to the device > specific ACS checks and you wouldn't need to worry about displacing all > the other 14.x functions. This would be cool. > > Disabling 00:14.1 seems to be possible (rmmod pata_atiixp works), but > > I'm getting errors after rebooting the system (the filesystems haven't > > been cleanly unmounted during shutdown). > > Did you have a disk off that controller? My system doesn't expose > function 1, but is the same otherwise. Filesystems on my ssd (Corsair Force GT) show the problem. Maybe the reason is another? > > Anyway, I would have tested it, but I'm getting a compile error while > > compiling qemu. It complains about missing pci/header.h while > > processing hw/vfio_pci.c:49:24. I can't find any pci/header.h. Where > > should it come from? > > It is from pciutils-devel for me. I'll see what I'm getting out of > there and try to remove it. Meanwhile I found it :-). Thanks for your hint! > > > The downside is that VFIO is strict about > > > multifunction devices supporting ACS to prevent peer-to-peer between > > > domains, so will require all of the 14.x devices to be bound to pci-stub > > > as well. On my system, this includes an smbus controller, audio device, > > > lpc controller, and usb device. If AMD could confirm this device > > > doesn't allow peer-to-peer between functions, we could relax this > > > requirement a bit. > > > > Please Joerg, could you take a look at this problem? > > Joerg, the question is whether the multifunction device above allows > peer-to-peer between functions that could bypass the iommu. If not, we > can make it the first entry in device specific acs enabled function I > proposed: > > https://lkml.org/lkml/2012/5/30/438 > > and it would greatly simplify assigning PCI devices on these systems > with VFIO. Thanks, I tried to run qemu-system-x86_64 but got this error on startup: qemu-system-x86_64: -device vfio-pci,host=06:07.0,id=hostdev0,bus=pci.0,addr=0x5: vfio: failed to set iommu for container: Operation not permitted qemu-system-x86_64: -device vfio-pci,host=06:07.0,id=hostdev0,bus=pci.0,addr=0x5: vfio: failed to setup container for group 9 qemu-system-x86_64: -device vfio-pci,host=06:07.0,id=hostdev0,bus=pci.0,addr=0x5: vfio: failed to get group 9 ** ERROR:qom/object.c:389:object_delete: assertion failed: (obj->ref == 0) I started qemu-system-x86_64 with this option among others -device vfio-pci,host=06:07.0,id=hostdev0,bus=pci.0,addr=0x5 after all of the devices have been added to pci-stub but 06:07.0, which was added to vfio-pci. Could you please tell me, why the operation isn't permitted? I started qemu-system-x86_64 as root. Thanks, kind regards, Andreas Hartmann -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
