On Tue, 2012-06-05 at 17:17 +0200, Andreas Hartmann wrote: > On Tue, 05 Jun 2012 08:27:05 -0600 > Alex Williamson <alex.williamson@xxxxxxxxxx> wrote: > > > On Tue, 2012-06-05 at 12:39 +0200, Andreas Hartmann wrote: > > > Hello Alex, > > > > > > thanks for your efforts! > > > > > > Maybe, you already know that I'm suffering the same problem :-( with a > > > GA-990XA-UD3 (990X chipset). > > > > My system is a GA-990FXA-UD3, so just a slightly different version of > > the chipset. > > > > > On Mon, 04 Jun 2012 21:44:05 -0600 > > > Alex Williamson <alex.williamson@xxxxxxxxxx> wrote: > > > [...] > > > > I have a setup with an AMD 990FX system and a spare PVR-350 card that I > > > > installed to reproduce. The sad answer is that it's nearly impossible > > > > to assign PCI devices on these systems due to the aliasing of devices > > > > below the PCIe-to-PCI bridge (PCIe devices are much, much easier to > > > > assign). If you boot with amd_iommu_dump, you'll see some output like > > > > this: > > > > > > > > AMD-Vi: DEV_ALIAS_RANGE devid: 05:00.0 flags: 00 devid_to: 00:14.4 > > > > AMD-Vi: DEV_RANGE_END devid: 05:1f.7 > > > > > > here: > > > AMD-Vi: DEV_ALIAS_RANGE devid: 06:00.0 flags: 00 devid_to: 00:14.4 > > > AMD-Vi: DEV_RANGE_END devid: 06:1f.7 > > > > > > > > > This means according to your description, the following devices have to > > > be unbound here, too (0000:06:07.0 is the device I want to passthrough): > > > > > > 0000:00:14.0 -> ../../../../devices/pci0000:00/0000:00:14.0 > > > 0000:00:14.1 -> ../../../../devices/pci0000:00/0000:00:14.1 > > > 0000:00:14.2 -> ../../../../devices/pci0000:00/0000:00:14.2 > > > 0000:00:14.3 -> ../../../../devices/pci0000:00/0000:00:14.3 > > > 0000:00:14.4 -> ../../../../devices/pci0000:00/0000:00:14.4 > > > 0000:00:14.5 -> ../../../../devices/pci0000:00/0000:00:14.5 > > > 0000:06:07.0 -> ../../../../devices/pci0000:00/0000:00:14.4/0000:06:07.0 > > > > Yep. > > > > > These are the following additional devices: > > > > > > 00:14.0 SMBus: Advanced Micro Devices [AMD] nee ATI SBx00 SMBus Controller (rev 42) > > > 00:14.1 IDE interface: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 IDE Controller (rev 40) (prog-if 8a [Master SecP PriP]) > > > 00:14.2 Audio device: Advanced Micro Devices [AMD] nee ATI SBx00 Azalia (Intel HDA) (rev 40) > > > 00:14.3 ISA bridge: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 LPC host controller (rev 40) > > > 00:14.4 PCI bridge: Advanced Micro Devices [AMD] nee ATI SBx00 PCI to PCI Bridge (rev 40) (prog-if 01 [Subtractive decode]) > > > 00:14.5 USB controller: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 USB OHCI2 Controller (prog-if 10 [OHCI]) > > > > My system shows this for lspci -n -s 14. > > > > 00:14.0 0c05: 1002:4385 (rev 42) > > 00:14.2 0403: 1002:4383 (rev 40) > > 00:14.3 0601: 1002:439d (rev 40) > > 00:14.4 0604: 1002:4384 (rev 40) > > 00:14.5 0c03: 1002:4399 > > My system: > 00:14.0 0c05: 1002:4385 (rev 42) > 00:14.1 0101: 1002:439c (rev 40) > 00:14.2 0403: 1002:4383 (rev 40) > 00:14.3 0601: 1002:439d (rev 40) > 00:14.4 0604: 1002:4384 (rev 40) > 00:14.5 0c03: 1002:4399 > > You miss 00.14.1!? But the others are equal. Yeah, my system doesn't expose the IDE, *shrug* > > Can you confirm yours matches and fill in 14.1 so we have the > > vendor:device IDs for these. > > Yes. Thanks > > > Among them is the sound device and the USB device - no good idea to > > > disable them on a desktop. > > > > You can always blacklist drivers if the device is unique or unbind > > devices in an rc.local type script. I'm hoping that Joerg or someone > > else from AMD can tell us this devices does not allow internal > > peer-to-peer between functions. Then we can add it to the device > > specific ACS checks and you wouldn't need to worry about displacing all > > the other 14.x functions. > > This would be cool. > > > > Disabling 00:14.1 seems to be possible (rmmod pata_atiixp works), but > > > I'm getting errors after rebooting the system (the filesystems haven't > > > been cleanly unmounted during shutdown). > > > > Did you have a disk off that controller? My system doesn't expose > > function 1, but is the same otherwise. > > Filesystems on my ssd (Corsair Force GT) show the problem. Maybe the > reason is another? > > > > Anyway, I would have tested it, but I'm getting a compile error while > > > compiling qemu. It complains about missing pci/header.h while > > > processing hw/vfio_pci.c:49:24. I can't find any pci/header.h. Where > > > should it come from? > > > > It is from pciutils-devel for me. I'll see what I'm getting out of > > there and try to remove it. > > Meanwhile I found it :-). Thanks for your hint! Great. > > > > The downside is that VFIO is strict about > > > > multifunction devices supporting ACS to prevent peer-to-peer between > > > > domains, so will require all of the 14.x devices to be bound to pci-stub > > > > as well. On my system, this includes an smbus controller, audio device, > > > > lpc controller, and usb device. If AMD could confirm this device > > > > doesn't allow peer-to-peer between functions, we could relax this > > > > requirement a bit. > > > > > > Please Joerg, could you take a look at this problem? > > > > Joerg, the question is whether the multifunction device above allows > > peer-to-peer between functions that could bypass the iommu. If not, we > > can make it the first entry in device specific acs enabled function I > > proposed: > > > > https://lkml.org/lkml/2012/5/30/438 > > > > and it would greatly simplify assigning PCI devices on these systems > > with VFIO. Thanks, > > I tried to run qemu-system-x86_64 but got this error on startup: > > qemu-system-x86_64: -device vfio-pci,host=06:07.0,id=hostdev0,bus=pci.0,addr=0x5: vfio: failed to set iommu for container: Operation not permitted > > qemu-system-x86_64: -device vfio-pci,host=06:07.0,id=hostdev0,bus=pci.0,addr=0x5: vfio: failed to setup container for group 9 > > qemu-system-x86_64: -device vfio-pci,host=06:07.0,id=hostdev0,bus=pci.0,addr=0x5: vfio: failed to get group 9 > ** > ERROR:qom/object.c:389:object_delete: assertion failed: (obj->ref == 0) > > > I started qemu-system-x86_64 with this option among others > > -device vfio-pci,host=06:07.0,id=hostdev0,bus=pci.0,addr=0x5 > > after all of the devices have been added to pci-stub but 06:07.0, which was added to vfio-pci. > > > Could you please tell me, why the operation isn't permitted? I started > qemu-system-x86_64 as root. Check dmesg, it's probably complaining about no interrupt remapping support. If so: # modprobe -r vfio_iommu_type1 # modprobe vfio_iommu_type1 allow_unsafe_interrupts=1 Then try again. Thanks, Alex -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
