On 08/12/2015 02:34 PM, Alex Holst wrote: > I would really appreciate some pointers on what I am doing wrong here. > > I have a need to run multiple virtual guests which have each their own GPU and > some USB controllers passed-through. I am able to run one of the guests like > this (assuming vfio stuff has happened elsewhere), but I would prefer to use > virsh: > > kvm -M q35 -m 8192 -cpu host,kvm=off \ > -smp 4,sockets=1,cores=4,threads=1 \ > -bios /usr/share/seabios/bios.bin -vga none \ > -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ > -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ > -device vfio-pci,host=02:00.1,bus=root.1,addr=00.1 \ > -device vfio-pci,host=00:1d.0,bus=pcie.0 \ > -device vfio-pci,host=00:1a.0,bus=pcie.0 \ > -nographic -boot menu=on /vm2/foo.img > > I found the hardware addresses using lspci. When I invoke the same machine with virsh with what > I believe are the same addresses, 1) The XML that you provide at the end shows you only assigning 02:00.0, but not 02:00.1, and you are using managed='yes', so you are depending on libvirt to unbind any host driver and bind vfio-pci to the devices. If 02:00.0 and 02:00.1 are both in iommu group 18, but one of them isn't bound to vfio-pci, qemu will give "some kind of error" when it tries to assign any other device in the group. 2) your example qemu commandline uses the q35 machinetype, but your example libvirt domain uses pc-i440fx-utopic machinetype, so the bus structure is completely different. Assuming a recent enough libvirt, you should be able to create a virtual machine based on q35 and attach the devices to pcie-root (bus 0). You wouldn't be able to add an ioh3420 controller in unless you are running libvirt built from upstream master - support for that was just pushed last weekend. (it is a new controller defined like this: <controller type='pci' model='pcie-root-port'/> 3) Doing the above *may* eliminate another problem that your current definition has - it has an emulated video device at 00:02.0, then the passthrough video device at 00:04.0. It's likely that even if you do get the domain started up, you'll end up with the passthrough vga as a secondary (which I'm guessing isn't what you want). 4) I don't know what "x-vga=on" is, but libvirt doesn't directly support that. If it is necessary, you would need to add it using libvirt's "qemu commandline passthrough": http://blog.vmsplice.net/2011/04/how-to-pass-qemu-command-line-options.html (you can add a "-set", "device.hostdev0.x-vga=on" I think) 5) The libvirt definition you've provided add several other devices that you don't have in your qemu commandline which may get in the way of what you want to do; in particular the tablet, mouse, and keyboard. You'll probably want to trim those out (and if you're passing through USB, you'll likely want to replace all of the <controller type='usb'> elements with a single <controller type='usb' model='none'/> 6) your qemu commandline specifies an exact bios location, which you haven't done in the libvirt definition. If that isn't the default bios, then you'll want to look into how to specify a non-default bios file here: http://www.libvirt.org/formatdomain.html 7) If you're going to stick with the pc-ii440fx machinetype and your guest doesn't absolutely require a SATA disk, you'd probably be better off letting it attach to the 440fx's IDE controller instead (or even better, if your guest OS supports the virtio disk driver, use that since the performance will be *greatly* improved). (likewise, replace the network interface's "model type='e1000'" with "model type='virtio'" if you can). > I get: > > virsh # start foo > error: Failed to start domain foo > error: internal error: process exited while connecting to monitor: 2015-08-12T18:24:10.651720Z qemu-system-x86_64: -device vfio-pci,host=02:00.0,id=hostdev0,bus=pci.0,addr=0x4: vfio: error, group 18 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver. > 2015-08-12T18:24:10.651752Z qemu-system-x86_64: -device vfio-pci,host=02:00.0,id=hostdev0,bus=pci.0,addr=0x4: vfio: failed to get group 18 > 2015-08-12T18:24:10.651766Z qemu-system-x86_64: -device vfio-pci,host=02:00.0,id=hostdev0,bus=pci.0,addr=0x4: Device initialization failed. > 2015-08-12T18:24:10.651781Z qemu-system-x86_64: -device vfio-pci,host=02:00.0,id=hostdev0,bus=pci.0,addr=0x4: Device 'vfio-pci' could not be initialized > > I have included dumpxml output below -- is the hostdev section wrong? > > <domain type='kvm'> > <name>foo</name> > <uuid>51f57655-11be-41bf-b925-2e6aef01f9c4</uuid> > <memory unit='KiB'>8388608</memory> > <currentMemory unit='KiB'>8388608</currentMemory> > <vcpu placement='static' current='1'>4</vcpu> > <os> > <type arch='x86_64' machine='pc-i440fx-utopic'>hvm</type> > <bootmenu enable='yes'/> > </os> > <features> > <acpi/> > <apic/> > <pae/> > <hyperv> > <relaxed state='on'/> > <vapic state='on'/> > <spinlocks state='on' retries='8191'/> > </hyperv> > </features> > <cpu mode='custom' match='exact'> > <model fallback='allow'>SandyBridge</model> > <topology sockets='1' cores='4' threads='1'/> > </cpu> > <clock offset='localtime'> > <timer name='rtc' tickpolicy='catchup'/> > <timer name='pit' tickpolicy='delay'/> > <timer name='hpet' present='no'/> > <timer name='hypervclock' present='yes'/> > </clock> > <on_poweroff>destroy</on_poweroff> > <on_reboot>restart</on_reboot> > <on_crash>restart</on_crash> > <pm> > <suspend-to-mem enabled='no'/> > <suspend-to-disk enabled='no'/> > </pm> > <devices> > <emulator>/usr/bin/kvm-spice</emulator> > <disk type='file' device='disk'> > <driver name='qemu' type='raw'/> > <source file='/vm2/foo.img'/> > <target dev='sda' bus='sata'/> > <boot order='1'/> > <address type='drive' controller='0' bus='0' target='0' unit='0'/> > </disk> > <disk type='block' device='cdrom'> > <driver name='qemu' type='raw'/> > <target dev='hdb' bus='ide'/> > <readonly/> > <boot order='2'/> > <address type='drive' controller='0' bus='0' target='0' unit='1'/> > </disk> > <controller type='usb' index='0' model='ich9-ehci1'> > <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x7'/> > </controller> > <controller type='usb' index='0' model='ich9-uhci1'> > <master startport='0'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0' multifunction='on'/> > </controller> > <controller type='usb' index='0' model='ich9-uhci2'> > <master startport='2'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x1'/> > </controller> > <controller type='usb' index='0' model='ich9-uhci3'> > <master startport='4'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x2'/> > </controller> > <controller type='pci' index='0' model='pci-root'/> > <controller type='ide' index='0'> > <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> > </controller> > <controller type='virtio-serial' index='0'> > <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> > </controller> > <controller type='sata' index='0'> > <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> > </controller> > <interface type='direct'> > <mac address='52:54:00:f0:47:f5'/> > <source dev='p5p1' mode='bridge'/> > <model type='e1000'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> > </interface> > <serial type='pty'> > <target port='0'/> > </serial> > <console type='pty'> > <target type='serial' port='0'/> > </console> > <channel type='spicevmc'> > <target type='virtio' name='com.redhat.spice.0'/> > <address type='virtio-serial' controller='0' bus='0' port='1'/> > </channel> > <input type='tablet' bus='usb'/> > <input type='mouse' bus='ps2'/> > <input type='keyboard' bus='ps2'/> > <graphics type='spice' autoport='yes'/> > <video> > <model type='cirrus' vram='16384' heads='1'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> > </video> > <hostdev mode='subsystem' type='pci' managed='yes'> > <source> > <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> > </source> > <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> > </hostdev> > <memballoon model='virtio'> > <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> > </memballoon> > </devices> > </domain> > _______________________________________________ libvirt-users mailing list libvirt-users@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/libvirt-users
