-----Original Message-----
From: Martin Kletzander <mkletzan@xxxxxxxxxx>
Sent: Monday, August 17, 2020 4:58 PM
To: Zhong, Luyao <luyao.zhong@xxxxxxxxx>
Cc: libvir-list@xxxxxxxxxx; Zang, Rui <rui.zang@xxxxxxxxx>; Michal
Privoznik
<mprivozn@xxxxxxxxxx>
Subject: Re: [RFC PATCH] add a new 'default' option for
attribute mode
in numatune
On Tue, Aug 11, 2020 at 04:39:42PM +0800, Zhong, Luyao wrote:
On 8/7/2020 4:24 PM, Martin Kletzander wrote:
On Fri, Aug 07, 2020 at 01:27:59PM +0800, Zhong, Luyao wrote:
On 8/3/2020 7:00 PM, Martin Kletzander wrote:
On Mon, Aug 03, 2020 at 05:31:56PM +0800, Luyao Zhong wrote:
Hi Libvirt experts,
I would like enhence the numatune snippet configuration. Given a
example snippet:
<domain>
 ...
 <numatune>
  <memory mode="strict" nodeset="1-4,^3"/>  ÂÂ
<memnode cellid="0" mode="strict" nodeset="1"/>   <memnode
cellid="2" mode="preferred" nodeset="2"/>  </numatune>  ...
</domain>
Currently, attribute mode is either 'interleave', 'strict', or
'preferred', I propose to add a new 'default' option. I give
the reason as following.
Presume we are using cgroups v1, Libvirt sets cpuset.mems for all
vcpu threads according to 'nodeset' in memory element. And
translate the memnode element to qemu config options (--object
memory-backend-ram) for per numa cell, which invoking mbind()
system call at the end.[1]
But what if we want using default memory policy and request each
guest numa cell pinned to different host memory nodes? We can't
use mbind via qemu config options, because (I quoto here) "For
MPOL_DEFAULT, the nodemask and maxnode arguments must be specify
the empty set of nodes." [2]
So my solution is introducing a new 'default' option for attribute
mode. e.g.
<domain>
 ...
 <numatune>
  <memory mode="default" nodeset="1-2"/>   <memnode
cellid="0" mode="default" nodeset="1"/>   <memnode
cellid="1" mode="default" nodeset="2"/>  </numatune>  ...
</domain>
If the mode is 'default', libvirt should avoid generating qemu
command line '--object memory-backend-ram', and invokes cgroups to
set cpuset.mems for per guest numa combining with numa topology
config. Presume the numa topology is :
<cpu>
 ...
 <numa>
  <cell id='0' cpus='0-3' memory='512000' unit='KiB' /> ÂÂ
 <cell id='1' cpus='4-7' memory='512000' unit='KiB' /> ÂÂ
</numa>  ...
</cpu>
Then libvirt should set cpuset.mems to '1' for vcpus 0-3, and '2'
for vcpus 4-7.
Is this reasonable and feasible? Welcome any comments.
There are couple of problems here. The memory is not (always)
allocated by the vCPU threads. I also remember it to not be
allocated by the process, but in KVM in a way that was not affected
by the cgroup settings.
Thanks for your reply. Maybe I don't get what you mean, could you
give me more context? But what I proposed will have no effect on
other memory allocation.
Check how cgroups work. We can set the memory nodes that a process
will allocate from. However to set the node for the process
(thread) QEMU needs to be started with the vCPU threads already
spawned (albeit stopped). And for that QEMU already allocates some
memory. Moreover if extra memory was allocated after we set the
cpuset.mems it is not guaranteed that it will be allocated by the
vCPU in that NUMA cell, it might be done in the emulator instead or
the KVM module in the kernel in which case it might not be accounted
for the process actually causing the allocation (as we've already
seen with Linux). In all these cases cgroups will not do what you
want them to do. The last case might be fixed, the first ones are
by default not going to work.
That might be
fixed now,
however.
But basically what we have against is all the reasons why we
started using QEMU's command line arguments for all that.
I'm not proposing use QEMU's command line arguments, on contrary I
want using cgroups setting to support a new config/requirement. I
give a solution about if we require default memory policy and memory
numa pinning.
And I'm suggesting you look at the commit log to see why we *had* to
add these command line arguments, even though I think I managed to
describe most of them above already (except for one that _might_
already be fixed in the kernel). I understand the git log is huge
and the code around NUMA memory allocation was changing a lot, so I
hope my explanation will be enough.
Thank you for detailed explanation, I think I get it now. We can't
guarantee memory allocation matching requirement since there is a time
slot before setting cpuset.mems.
That's one of the things, although this one could be avoided (by
setting a global
cgroup before exec()).
Thanks,
Luyao
Sorry, but I think it will more likely break rather than fix stuff.
Maybe this
could be dealt with by a switch in `qemu.conf` with a huge warning
above it.
I'm not trying to fix something, I propose how to support a new
requirement just like I stated above.
I guess we should take a couple of steps back, I don't get what you
are trying to achieve. Maybe if you describe your use case it will
be easier to reach a conclusion.
Yeah, I do have a usecase I didn't mention before. It's a feature in
kernel but not merged yet, we call it memory tiering.
(https://lwn.net/Articles/802544/)
If memory tiering is enabled on host, DRAM is top tier memory, and
PMEM(persistent memory) is second tier memory, PMEM is shown as numa
node without cpu. For short, pages can be migrated between DRAM and
PMEM based on DRAM pressure and how cold/hot they are.
We could configure multiple memory migrating path. For example, node 0:
DRAM, node 1: DRAM, node 2: PMEM, node 3: PMEM we can make 0+2 to a
group, and 1+3 to a group. In each group, page is allowed to migrated
down(demotion) and up(promotion).
If **we want our VMs utilizing memory tiering and with NUMA topology**,
we need handle the guest memory mapping to host memory, that means we
need bind each guest numa node to a memory nodes group(DRAM node +
PMEM
node) on host. For example, guest node 0 -> host node 0+2.
However, only cgroups setting can make the memory tiering work, if we
use mbind() system call, demoted pages will never go back to DRAM.
That's why I propose to add 'default' option and bypass mbind in QEMU.
I hope I make myself understandable. I'll appreciate if you could give
some suggestion.
This comes around every couple of months/years and bites us in the
back no
matter what way we go (every time there is someone who wants it the
other
way).
That's why I think there could be a way for the user to specify
whether they will
likely move the memory or not and based on that we would specify `host-
nodes` and `policy` to qemu or not. I think I even suggested this
before (or
probably delegated it to someone else for a suggestion so that there
is more
discussion), but nobody really replied.
So what we need, I think, is a way for someone to set a per-domain
information
whether we should bind the memory to nodes in a changeable fashion or
not.
I'd like to have it in as well. The way we need to do that is,
probably, per-
domain, because adding yet another switch for each place in the XML
where we
can select a NUMA memory binding would be a suicide. There should
also be
no need for this to be enabled per memory-(module, node), so it
should work
fine.
