Re: trying to use vfio to pass VGA card and getting operation not permitted error

Steven DuChene <steven.duchene@xxxxxx> · Thu, 13 Nov 2014 12:36:42 -0500

Alex:
Thanks for the quick reply.
Yes, I am using qemu:args in the xml file.
I looked around via google for an example of how to use hostdev in the 
xml file for the audio or video devices.
I seem to recall the only example I could find was a hostdev section in 
the libvirt wiki for a NIC card.
Do you know of any example xml files where hostdev is used for video and 
audio devices?
--
Steven DuChene

On 11/13/2014 12:24 PM, Alex Williamson wrote:
On Thu, 2014-11-13 at 11:50 -0500, Steven DuChene wrote:
I am trying to use VFIO and pci-bind to pass a NVidia VGA card and it's
companion audio device through to a VM I am trying to start up.

I am trying this on two different hardware platforms, a HP zbook 15 and
a HP z800 workstation. Both systems are running Ubuntu 14.04.

Each time I try to start the VM on the laptop using "virsh start vmname"
I get the following errors:

error: Failed to start domain vmname
error: internal error: early end of file from monitor: possible problem:
qemu-system-x86_64: -device
vfio-pci,host=01:00.0,bus=sven.1,addr=00.0,multifunction=on,x-vga=on,rombar=0:
vfio: error opening /dev/vfio/1: Operation not permitted
qemu-system-x86_64: -device
vfio-pci,host=01:00.0,bus=sven.1,addr=00.0,multifunction=on,x-vga=on,rombar=0:
vfio: failed to get group 1
qemu-system-x86_64: -device
vfio-pci,host=01:00.0,bus=sven.1,addr=00.0,multifunction=on,x-vga=on,rombar=0:
Device initialization failed.
qemu-system-x86_64: -device
vfio-pci,host=01:00.0,bus=sven.1,addr=00.0,multifunction=on,x-vga=on,rombar=0:
Device 'vfio-pci' could not be initialized

If I start the VM from the command line on the z800 as root using the
qemu-system-x86_64 command directly it works. The VM starts and both of
the passed through devices show up in the VM. If I try to start the same
VM as the libvirt-qemu user using the same qemu-system-x86_64 command I
get the same above error (but with a different group number obviously).

My order of operations are:

blacklist the nouveau driver from loading (the closed source nvidia
driver is not loaded on either system)

In the kernel command line passed through grub on the z800 workstation I
have:

intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1

on the zbook15 laptop I have:

pci-stub.ids=10de:11fc,10de:0e0b intel_iommu=on

and on the laptop I have added the "allow_unsafe_interrupts=1" option to
the modprobe of that module.

On the z800 I manually bind the nvidia video and audio devices to the
pci-stub driver.

In both cases I see the following in the kernel dmesg:

(from the laptop)
   [    6.342603] pci-stub: add 10DE:11FC sub=FFFFFFFF:FFFFFFFF
cls=00000000/00000000
[    6.342618] pci-stub 0000:01:00.0: claimed by stub
[    6.342625] pci-stub: add 10DE:0E0B sub=FFFFFFFF:FFFFFFFF
cls=00000000/00000000
[    6.342632] pci-stub 0000:01:00.1: claimed by stub

(or from the z800 workstation)
[  115.116860] pci-stub 0000:0f:00.1: claimed by stub
[  157.126503] pci-stub 0000:0f:00.0: claimed by stub

I then use a vfio-bind script to bind both the video and audio devices
to the vfio driver.
Once I do that I see the appropriate files under /dev/vfio

(on the laptop)
drwxr-xr-x  2 root root     80 Nov 13 08:15 ./
drwxr-xr-x 18 root root   4380 Nov 13 08:14 ../
crw-------  1 root root 249, 1 Nov 13 08:15 1
crw-rw-rw-  1 root root 249, 0 Nov 13 08:14 vfio

(on the z800)
drwxr-xr-x  2 root root     80 Nov 13 10:26 ./
drwxr-xr-x 16 root root   4540 Nov 13 10:26 ../
crw-------  1 root root 247, 1 Nov 13 10:26 14
crw-rw-rw-  1 root root 247, 0 Nov 13 10:26 vfio

I have confirmed that the only devices in the iommu group is the nvidia
video and audio devices and I am attempting to pass both devices through
to the VM when I invoke it.

On the laptop I was seeing messages in the logs from apparmor each time
I tried to start the VM referring to a libvirt profile for this VM's
uuid. After looking into that for a while, I finally set the
security_driver in /etc/libvirt/qemu.conf to "none" and that stopped
those messages from showing up in the logs. This also prevented the
libvirt-UUID files from showing up for this VM in
/etc/apparmor.d/libvirt/ each time I tried to start the VM. I did try to
add the files in /dev/vfio to the apparmor profile and TEMPLATE files
but that did not seem to have any effect on the log messages.

In both cases I am actually wanting to start the VMs though openstack
which invokes the qemu-system-x86_64 process through libvirt. However if
I try to start the VM on the z800 workstation as root from the command
line using the following command:

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host -smp
2,sockets=1,cores=2,threads=1 -bios /usr/share/qemu/bios.bin -device
ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=sven.1
-device
vfio-pci,host=01:00.0,bus=sven.1,addr=00.0,multifunction=on,x-vga=on,rombar=0
-device vfio-pci,host=01:00.1,bus=sven.1,addr=00.1 -drive
file=/root/ubuntu_withdriver.raw,id=disk,format=raw -device
ide-hd,bus=ide.0,drive=disk -drive file=/root/my-seed.img,id=isocd
-device ide-cd,bus=ide.1,drive=isocd -vnc 0.0.0.0:1 -k en-us

it works. If I try to start the VM on the z800 as the libvirt-qemu user
(user id virsh and openstack use) I get the error I listed at the top of
this E-mail. I have even tried changing the ownerships of the files in
/dev/vfio to this user but that did not make any difference and I still
get the same error.

I have spent quite a lot of time on this doing lots of research and I
seem to be no closer to solving the problem.

Can any experts on using VFIO see any problems with the steps and
configurations I have done or know of any other reason why I might be
getting this error sequence?
What does your domain xml look for like assigning the device?  I suspect
you might be using a <qemu:args> block to define the assigned device so
that you can add the x-vga=on option, which unfortunately has the
side-effect that libvirt is no longer aware of the device and cannot set
file permissions and locked memory limits correctly for doing device
assignment.  If so, you might be able to assign the audio function using
a standard <hostdev> section, which should setup the permissions for the
GPU in a <qemu:args> section as well since they're in the same group.
Another option is to use a wrapper script around qemu, for instance:

  <emulator>/usr/libexec/qemu-kvm-nvidia-vga</emulator>

# cat /usr/libexec/qemu-kvm-nvidia-vga
#!/bin/sh
exec /usr/libexec/qemu-kvm \
	`echo "\$@" | sed 's|01:00.0|01:00.0,x-vga=on|g'`

You'll likely also need to update apparmor to allow libvirt to execute
this script.  For selinux you'd do something like described in this
document:

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Technical_Notes/virt.html

The wrapper script option would allow both devices in the group to be
fully managed by libvirt and would then remove your need to manually run
any sort of vfio-bind script.

However, I've now looked up the quickspecs for the zbook and see that
you're likely using an Nvidia Quadro K-series GPU and hopefully
something similar on the z800.  These can be used as secondary GPUs in
the VM and do not need the x-vga option.  Simply attach the GPU and
audio to the VM (using <hostdev>, not <qemu:arg>) in addition to one of
the emulated VGA models.  The emulated VGA will be used for boot and
install and the assigned GPU will be activated when the Nvidia driver
stack is loaded.

Caveats: a) I have no idea if Ubuntu 14.04 includes the necessary
support for GPU assignment.  b) Laptop do strange and bizarre things
with discrete GPUs supporting Optimus.  It may not work, and if it does
work, it may not output through any of the external video ports on the
laptop or the screen.  You might want to setup remote access to the VM
to access the desktop via network and not rely on a physical output
port.

Good luck,
Alex

--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html