Re: question about rdma migration

["Michael R. Hines" <mhines@xxxxxxxxxxxxxxxx>]

The limits are controlled by the operating system via the getrlimit/setrlimit system calls.

If you want libvirt to set the limit for QEMU after QEMU is forked, you have to change libvirt, and that requires a invoking the system calls,
but more permanently requires a plan with the libvirt community on the use of those system calls.

/*
 * Michael R. Hines
 * Platform Engineer, DigitalOcean.
 */

On 02/19/2016 04:37 PM, Roy Shterman wrote:

Yes,

I tried also running it as root user and it also didn't worked.

Do you know where libvirt (or QEMU) gets the value for process MEMLOCK? maybe i can change this value in libvirt code?

Regards,

Roy

On Fri, Feb 19, 2016 at 11:15 PM, Michael R. Hines <mhines@xxxxxxxxxxxxxxxx> wrote:

Is the QEMU process (after startup) actually running as the QEMU userid ?

/*
 * Michael R. Hines
 * Platform Engineer, DigitalOcean.
 */

On 02/19/2016 02:43 PM, Roy Shterman wrote:

First off all thank you for your answer,

I couldn't figured how to start virtual machine with increased MEMLOCK,

tried to add into /etc/security/limits.d 

qemu            soft    memlock  3221225

qemu            hard    memlock  3221225

so max locked-in-memory will be 3G, but it didn't worked.

still has MEMLOCK of 60kb per each VM.

Maybe you can spot what I'm doing wrong?

On Tue, Feb 9, 2016 at 5:16 PM, Michael R. Hines <michael@xxxxxxxxxxxx> wrote:

Hi Roy,

On 02/09/2016 03:57 AM, Roy Shterman wrote:

Hi,

I tried to understand the rdma-migration in qemu code and i have two questions about it:

1. I'm working with qemu-kvm using libvirt and i'm getting

MEMLOCK    max locked-in-memory address space     65536 65536 bytes

in qemu process so I don't understand how can you use rdma-pin-all with such low MEMLOCK.

I found a solution in libvirt to lock all vm memory in advance and to enlarge MEMLOCK.
It uses memoryBacking locking and memory tuning hard_limit of vm memory but I couldn't find a usage of this in rdma-migration code.

You're absolutey right, the RDMA migration code itself doesn't set this lock limit explicitly because there are system-wide restrictions in both appArmour,
/etc/security, as well as SELINUX that restrict applications from arbitrarily setting their maximum memory lock limits.

The other problem is CGROUPS: If someone sets a cgroup control for maximum memory and forgets about that mlock() limits, then
there will be a conflict.

So, libvirt must have a policy to deal with all of these possibilities, not just handle a special case for RDMA migration.

The only way "simple" way (without patching the problems above) to apply a higher lock limit to QEMU is to set the ulimit for libvirt
(or for QEMU if starting QEMU manually) in your environment or the command line with $ ulimit # before attempting the migration,
then the RDMA subsystem will be able to lock the memory successfully.

The other option is to use /etc/security/limits.conf and set the option for a specific libvirt process user and make sure your libvirt/qemu
are not running as root.

QEMU itself also has a "mlock" option built into the command line, but it also suffers from the same problem --- you have to find
a way (currently) to increase the limit before using the option.

2. Do you have any comparison of IOPS and bandwidth between TCP migration and rdma migration?

Yes, lots of comparisons.

http://wiki.qemu.org/Features/RDMALiveMigration
http://www.canturkisci.com/ETC/papers/IBMJRD2011/preprint.pdf

Regards,
Roy

--
libvir-list mailing list
libvir-list@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/libvir-list

--
libvir-list mailing list
libvir-list@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/libvir-list