Alex Williamson wrote: > On Fri, 2012-06-08 at 19:39 +0200, Andreas Hartmann wrote: >> Andreas Hartmann wrote: >>> Hello Alex, >>> >>> You can probably say, what this message on host side means: >>> >>> kernel: [ 3902.124109] vfio_dma_do_map: RLIMIT_MEMLOCK (65536) exceeded >>> >>> The WLAN card in the VM doesn't work any more. It came up after a few >>> times of restarting the VM (with unbinding / rebinding - procedures). >>> >>> I'll see if it is reproducible. I had to reboot to get it working again. >> >> It is reproducible. And id seems not to be a problem of binding / >> unbinding, but the fact of not starting it as root user seems to be the >> problem. >> >> I never saw these problems with a root VM (and root does have the same >> value for ulimit -l). > > Yes, this is expected when running as non-root. VFIO needs to lock > pages on behalf of the user, so the user needs limits granted to be able > to do that. Otherwise a VFIO user could lock down all the memory in the > system. > >> - Is it possible to run the VM / VFIO in user context? > > Yes, this is one of the key design requirements of VFIO. > Pre-requirements are that a privileged entity has sequestered all the > devices as being owned by vfio-pci or pci-stub, the user has permissions > to /dev/vfio/<group#> and /dev/vfio/vfio (the latter is expected to be > safe to leave as 0666), and the user has limits set to lock pages > sufficient for what they need (note that the default of 64k might be > enough for some userspace driver applications). I already read about the permissions, but I wasn't aware, that I have to set the max locked memory for the user. I set it to 512 MB (for a VM with 256 MB plus some overhead) for the beginning. It most probably could be reduced. >> - What size should be used for ulimit -l? > > It should be about the size of memory assigned to the guest. Thanks for this hint! 64k is definitely not enough for my VM :-). > Once we have libvirt support, all of this should be relatively > transparent as that will take care of the limits setting. If you are aware of it, it's no problem ... . > For now, it's > a bit of a pain running it as a normal user. Not that much. It's good to become an understanding of what is going on. > If you come up with an > easy way of doing it, please share. I don't know, if it's easy, but it's straight forward and working: 1. define a user for running the VM. 2. create a VM for virsh (xml-File) libvirt has the ability to freely define parameters for qemu. That's the way I enabled libvirt to support VFIO: <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> ... <devices> ... </devices> <qemu:commandline> <!-- instead of hostdev ... --> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=06:07.0'/> <!-- <qemu:env name='QEMU_ENV' value='VAL'/> --> </qemu:commandline> </domain> 3. set max locked memory limit in /etc/security/limits.conf (PAM). 4. write a small start script which contains the bindings and the start of the VM as root with su user -c "virsh start VM" Don't forget to increase the max locked memory before virsh start if the default size is less than the desired size for the VM. 5. write a small stop script to shutdown the VM with su user -c "virsh shutdown VM" and free the devices again. The start / stop scripts could be put to sudo to enable a user without root authorization to start the VM. You can even start it with an icon with your favourite GUI if you want to (women acceptance factor). BTW: During playing around (yes, it was my fault), I accidentally tried to free the devices though the VM still has been running. Vfio wasn't that happy about this accident :-) : Jun 9 01:01:49 host kernel: [20108.859106] ------------[ cut here ]------------ Jun 9 01:01:49 host kernel: [20108.859117] kernel BUG at /rpm/BUILD/kernel-desktop-3.4.1/linux-3.4/drivers/vfio/vfio.c:574! Jun 9 01:01:49 host kernel: [20108.859125] invalid opcode: 0000 [#1] PREEMPT SMP Jun 9 01:01:49 host kernel: [20108.859131] CPU 3 Jun 9 01:01:49 host kernel: [20108.859133] Modules linked in: xfs vfio_iommu_type1 vfio_pci vfio ... [last unloaded: micr Jun 9 01:01:49 host kernel: ocode] Jun 9 01:01:49 host kernel: [20108.859226] Jun 9 01:01:49 host kernel: [20108.859231] Pid: 22178, comm: stopVM Tainted: P O 3.4.1-3.1-desktop #1 Gigabyte Technology Co., Ltd. GA-990XA-UD3/GA-990XA-UD3 Jun 9 01:01:49 host kernel: [20108.859241] RIP: 0010:[<ffffffffa077f0f3>] [<ffffffffa077f0f3>] vfio_iommu_group_notifier+0x1f3/0x300 [vfio] Jun 9 01:01:49 host kernel: [20108.859252] RSP: 0018:ffff8801f25f9d18 EFLAGS: 00010282 Jun 9 01:01:49 host kernel: [20108.859259] RAX: 00000000ffffffea RBX: ffff88022065a6c0 RCX: 0000000000000008 Jun 9 01:01:49 host kernel: [20108.859265] RDX: ffff880223f98090 RSI: ffff880223f98090 RDI: 0000000000000000 Jun 9 01:01:49 host kernel: [20108.859271] RBP: ffff88022065a718 R08: ffff88022065a6c0 R09: 000000000000000a Jun 9 01:01:49 host kernel: [20108.859277] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880223f98090 Jun 9 01:01:49 host kernel: [20108.859283] R13: ffffffffa04bb8f9 R14: 0000000000000000 R15: ffff880222906b40 Jun 9 01:01:49 host kernel: [20108.859290] FS: 00007f8613318700(0000) GS:ffff88022ecc0000(0000) knlGS:0000000000000000 Jun 9 01:01:49 host kernel: [20108.859297] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b Jun 9 01:01:49 host kernel: [20108.859303] CR2: 00007f6c0e8674f8 CR3: 000000013a066000 CR4: 00000000000407e0 Jun 9 01:01:49 host kernel: [20108.859310] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Jun 9 01:01:49 host kernel: [20108.859316] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Jun 9 01:01:49 host kernel: [20108.859322] Process stopVM (pid: 22178, threadinfo ffff8801f25f8000, task ffff880111176580) Jun 9 01:01:49 host kernel: [20108.859328] Stack: Jun 9 01:01:49 host kernel: [20108.859333] ffff8801f25f9d58 ffffffff8107471f ffff8801109b3e40 0000000000000004 Jun 9 01:01:49 host kernel: [20108.859342] ffff880223f98090 00000000ffffffff 0000000000000000 ffffffff815b5e25 Jun 9 01:01:49 host kernel: [20108.859350] ffff880221585e78 0000000000000004 ffff880223f98090 00000000ffffffff Jun 9 01:01:49 host kernel: [20108.859358] Call Trace: Jun 9 01:01:49 host kernel: [20108.859384] [<ffffffff815b5e25>] notifier_call_chain+0x45/0x60 Jun 9 01:01:49 host kernel: [20108.859395] [<ffffffff8106b886>] __blocking_notifier_call_chain+0x56/0x90 Jun 9 01:01:49 host kernel: [20108.859405] [<ffffffff81497e79>] iommu_bus_notifier+0x89/0xe0 Jun 9 01:01:49 host kernel: [20108.859414] [<ffffffff815b5e25>] notifier_call_chain+0x45/0x60 Jun 9 01:01:49 host kernel: [20108.859422] [<ffffffff8106b886>] __blocking_notifier_call_chain+0x56/0x90 Jun 9 01:01:49 host kernel: [20108.859433] [<ffffffff813bbb30>] really_probe+0xc0/0x300 Jun 9 01:01:49 host kernel: [20108.859442] [<ffffffff813bbef7>] driver_probe_device+0x47/0xb0 Jun 9 01:01:49 host kernel: [20108.859450] [<ffffffff813ba6f2>] driver_bind+0xd2/0x110 Jun 9 01:01:49 host kernel: [20108.859459] [<ffffffff811db732>] sysfs_write_file+0xd2/0x160 Jun 9 01:01:49 host kernel: [20108.859468] [<ffffffff8116b2b6>] vfs_write+0xc6/0x180 Jun 9 01:01:49 host kernel: [20108.859476] [<ffffffff8116b5ce>] sys_write+0x4e/0x90 Jun 9 01:01:49 host kernel: [20108.859484] [<ffffffff815b99f9>] system_call_fastpath+0x16/0x1b Jun 9 01:01:49 host kernel: [20108.859494] [<00007f86127a6190>] 0x7f86127a618f Jun 9 01:01:49 host kernel: [20108.859499] Code: a0 48 c7 c7 60 01 78 a0 e8 4a 01 e3 e0 8b 45 b0 85 c0 0f 84 20 ff ff ff 48 89 de 4c 89 e7 e8 b5 fb ff ff 85 c0 0f 84 0d ff ff ff <0f> 0b 0f 1f 00 48 8b 7d b8 e8 8f 87 d1 e0 49 8b 54 24 50 48 85 Jun 9 01:01:49 host kernel: [20108.859554] RIP [<ffffffffa077f0f3>] vfio_iommu_group_notifier+0x1f3/0x300 [vfio] Jun 9 01:01:49 host kernel: [20108.859563] RSP <ffff8801f25f9d18> Jun 9 01:01:49 host kernel: [20108.859573] ---[ end trace faef2013325649a8 ]--- The stopVM script segfaulted. Thanks, kind regards, Andreas -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
