On Fri, 08 Jun 2012 11:35:07 -0600
Alex Williamson <alex.williamson@xxxxxxxxxx> wrote:
> On Fri, 2012-06-08 at 18:58 +0200, Andreas Hartmann wrote:
> > Hello Alex,
> >
> > You can probably say, what this message on host side means:
> >
> > kernel: [ 3902.124109] vfio_dma_do_map: RLIMIT_MEMLOCK (65536) exceeded
>
> We've hit the limit of locked pages. Are you trying to run as root or a
> normal user? If the latter, you need to play with ulimits to increase
> the size.
That's what I did now. What for is this memory exactly needed? I don't
think for the complete VM, because the VM without the device passed
through works fine without it (and it comes up fine and can ssh'd).
That's why I think, it's just needed for the communication between
the device and the guest. But why so much then?
I think I didn't got it right until now ... .
>
> > The WLAN card in the VM doesn't work any more. It came up after a few
> > times of restarting the VM (with unbinding / rebinding - procedures).
>
> Do I recall correctly you reporting a message about the device not
> supporting reset for the WLAN?`
Yes.
> Unfortunately devices are mostly black
> boxes as far as VFIO is concerned, so if the device doesn't support
> reset and doesn't have it's own device specific reset and doesn't simply
> start behaving when we restore config space, there's little for vfio to
> do. We do have a bit more flexibility in performing a secondary bus
> reset on the bridge since we own everything below the bridge. We
> probably need to consider adding a group reset ioctl to take advantage
> of that.
>
> > I'll see if it is reproducible. I had to reboot to get it working again.
>
> I'm definitely curious if there's anything cumulative about the locked
> memory problem above. Thanks,
Ok, I managed to get it reproducible. I'll describe step by step, how.
- setting low memory (64k)
- start VM:
qemu-system-x86_64: vfio_dma_map(0x7fbfcf4fd170, 0x00000000febe0000, 0x10000, 0x7fbfb57b0000) = -12 (Cannot allocate memory)
Jun 9 14:11:33 host kernel: [12001.026007] vfio_dma_do_map: RLIMIT_MEMLOCK (65536) exceeded
- VM is up
- module rt2800pci in VM is loaded fine - no errors can be seen in log.
- but: device doesn't work (no beaconing)
- stop hostapd
- unload wlan stack (hardware + nl80211)
- reload wlan stack
- start hostapd
Jun 9 14:16:17 vm kernel: [ 286.088795] phy0 -> rt2x00lib_request_firmware: Info - Loading firmware file 'rt2860.bin'.
Jun 9 14:16:17 vm kernel: [ 286.090251] phy0 -> rt2x00lib_request_firmware: Info - Firmware detected - version: 0.34.
Jun 9 14:16:18 vm kernel: [ 287.194351] phy0 -> rt2800_wait_wpdma_ready: Error - WPDMA TX/RX busy [0x0000006a].
Jun 9 14:16:19 vm kernel: [ 288.294350] phy0 -> rt2800_wait_wpdma_ready: Error - WPDMA TX/RX busy [0x0000006a].
Jun 9 14:16:19 vm kernel: [ 288.294358] phy0 -> rt2800pci_set_device_state: Error - Device failed to enter state 4 (-5).
- shutdown VM (virsh shutdown VM)
- set memory to 512M
- start VM (no RLIMIT_MEMLOCK error)
- VM is up
- module rt2800pci doesn't load correctly:
Jun 9 14:24:27 vm kernel: [ 8.544858] phy0 -> rt2x00lib_request_firmware: Info - Loading firmware file 'rt2860.bin'.
Jun 9 14:24:27 vm kernel: [ 8.547870] phy0 -> rt2x00lib_request_firmware: Info - Firmware detected - version: 0.34.
Jun 9 14:24:28 vm kernel: [ 9.652364] phy0 -> rt2800_wait_wpdma_ready: Error - WPDMA TX/RX busy [0x0000006a].
Jun 9 14:24:29 vm kernel: [ 10.752363] phy0 -> rt2800_wait_wpdma_ready: Error - WPDMA TX/RX busy [0x0000006a].
Jun 9 14:24:29 vm kernel: [ 10.752371] phy0 -> rt2800pci_set_device_state: Error - Device failed to enter state 4 (-5).
I didn't manage to remove this error but with rebooting.
I tried w/ or w/o including the bridge to the bind procedure. I even
tried to get it working again by loading the module on the host. Could
it be probably a issue of rt2800pci?
Start script for VM:
---------------------------------------------------------------
#!/bin/sh
function bind() {
modprobe vfio-pci
# not necessary
#echo "1002 4385" > /sys/bus/pci/drivers/pci-stub/new_id
#echo 0000:00:14.0 > /sys/bus/pci/devices/0000:00:14.0/driver/unbind
#echo 0000:00:14.0 > /sys/bus/pci/drivers/pci-stub/bind
echo "1002 439c" > /sys/bus/pci/drivers/pci-stub/new_id
echo 0000:00:14.1 > /sys/bus/pci/devices/0000:00:14.1/driver/unbind
echo 0000:00:14.1 > /sys/bus/pci/drivers/pci-stub/bind
echo "1002 4383" > /sys/bus/pci/drivers/pci-stub/new_id
echo 0000:00:14.2 > /sys/bus/pci/devices/0000:00:14.2/driver/unbind
echo 0000:00:14.2 > /sys/bus/pci/drivers/pci-stub/bind
# not necessary
#echo "1002 439d" > /sys/bus/pci/drivers/pci-stub/new_id
#echo 0000:00:14.3 > /sys/bus/pci/devices/0000:00:14.3/driver/unbind
#echo 0000:00:14.3 > /sys/bus/pci/drivers/pci-stub/bind
# not necessary
#echo "1002 4384" > /sys/bus/pci/drivers/pci-stub/new_id
#echo 0000:00:14.4 > /sys/bus/pci/devices/0000:00:14.4/driver/unbind
#echo 0000:00:14.4 > /sys/bus/pci/drivers/pci-stub/bind
echo "1002 4399" > /sys/bus/pci/drivers/pci-stub/new_id
echo 0000:00:14.5 > /sys/bus/pci/devices/0000:00:14.5/driver/unbind
echo 0000:00:14.5 > /sys/bus/pci/drivers/pci-stub/bind
echo "1814 0601" > /sys/bus/pci/drivers/vfio-pci/new_id
echo 0000:06:07.0 > /sys/bus/pci/devices/0000:06:07.0/driver/unbind
echo 0000:06:07.0 > /sys/bus/pci/drivers/vfio-pci/bind
sleep 1
chgrp virt5 /dev/vfio/9
chmod 660 /dev/vfio/9
chgrp virt5 /dev/vfio/vfio
chmod 660 /dev/vfio/vfio
}
if [ -S /a/vm/.libvirt/qemu/lib/VM.monitor ]; then
echo "VM already running!"
exit 1
fi
bind
ulimit -l 524288
su vm -c "virsh start VM"
----------------------------------------------------------------
Stop script for VM:
---------------------------------------------------------------
#!/bin/sh
if [ ! -S /a/vm/.libvirt/qemu/lib/VM.monitor ]; then
echo "VM not running"
exit 1
fi
su vm -c "virsh shutdown VM"
# wait for VM exited ...
function wait4exit ()
{
while true ; do
sleep 5
cnt=`su vm -c "virsh list" | grep -c VM`
if [ "$cnt" -eq 0 ] ; then
break
fi
done
}
wait4exit
# 1002 4385 / 0000:00:14.0 not necessary
echo 0000:00:14.1 > /sys/bus/pci/drivers/pci-stub/unbind
echo "1002 439c" > /sys/bus/pci/drivers/pci-stub/remove_id
echo 0000:00:14.2 > /sys/bus/pci/drivers/pci-stub/unbind
echo "1002 4383" > /sys/bus/pci/drivers/pci-stub/remove_id
# rebind sound
echo 0000:00:14.2 > /sys/bus/pci/drivers/snd_hda_intel/bind
# 1002 439d / 0000:00:14.3 not necessary
# not necessary - bridge 1002 4384 / 0000:00:14.4
#echo 0000:00:14.4 > /sys/bus/pci/drivers/pci-stub/unbind
#echo "1002 4384" > /sys/bus/pci/drivers/pci-stub/remove_id
echo 0000:00:14.5 > /sys/bus/pci/drivers/pci-stub/unbind
echo "1002 4399" > /sys/bus/pci/drivers/pci-stub/remove_id
echo 0000:06:07.0 > /sys/bus/pci/drivers/vfio-pci/unbind
echo "1814 0601" > /sys/bus/pci/drivers/vfio-pci/remove_id
------------------------------------------------------------
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
