(ping)
Hi,
this has been on these lists for quite some time now. I want to start
preparing a virtio spec for virtio-mem soon.
So if you have any more comments/ideas/objections/questions, now is the
right time to post them :)
Thanks!
On 16.06.2017 16:20, David Hildenbrand wrote:
> Hi,
>
> this is an idea that is based on Andrea Arcangeli's original idea to
> host enforce guest access to memory given up using virtio-balloon using
> userfaultfd in the hypervisor. While looking into the details, I
> realized that host-enforcing virtio-balloon would result in way too many
> problems (mainly backwards compatibility) and would also have some
> conceptual restrictions that I want to avoid. So I developed the idea of
> virtio-mem - "paravirtualized memory".
>
> The basic idea is to add memory to the guest via a paravirtualized
> mechanism (so the guest can hotplug it) and remove memory via a
> mechanism similar to a balloon. This avoids having to online memory as
> "online-movable" in the guest and allows more fain grained memory
> hot(un)plug. In addition, migrating QEMU guests after adding/removing
> memory gets a lot easier.
>
> Actually, this has a lot in common with the XEN balloon or the Hyper-V
> balloon (namely: paravirtualized hotplug and ballooning), but is very
> different when going into the details.
>
> Getting this all implemented properly will take quite some effort,
> that's why I want to get some early feedback regarding the general
> concept. If you have some alternative ideas, or ideas how to modify this
> concept, I'll be happy to discuss. Just please make sure to have a look
> at the requirements first.
>
> -----------------------------------------------------------------------
> 0. Outline:
> -----------------------------------------------------------------------
> - I. General concept
> - II. Use cases
> - III. Identified requirements
> - IV. Possible modifications
> - V. Prototype
> - VI. Problems to solve / things to sort out / missing in prototype
> - VII. Questions
> - VIII. Q&A
>
> ------------------------------------------------------------------------
> I. General concept
> ------------------------------------------------------------------------
>
> We expose memory regions to the guest via a paravirtualize interface. So
> instead of e.g. a DIMM on x86, such memory is not anounced via ACPI.
> Unmodified guests (without a virtio-mem driver) won't be able to see/use
> this memory. The virtio-mem guest driver is needed to detect and manage
> these memory areas. What makes this memory special is that it can grow
> while the guest is running ("plug memory") and might shrink on a reboot
> (to compensate "unplugged" memory - see next paragraph). Each virtio-mem
> device manages exactly one such memory area. By having multiple ones
> assigned to different NUMA nodes, we can modify memory on a NUMA basis.
>
> Of course, we cannot shrink these memory areas while the guest is
> running. To be able to unplug memory, we do something like a balloon
> does, however limited to this very memory area that belongs to the
> virtio-mem device. The guest will hand back small chunks of memory. If
> we want to add memory to the guest, we first "replug" memory that has
> previously been given up by the guest, before we grow our memory area.
>
> On a reboot, we want to avoid any memory holes in our memory, therefore
> we resize our memory area (shrink it) to compensate memory that has been
> unplugged. This highly simplifies hotplugging memory in the guest (
> hotplugging memory with random memory holes is basically impossible).
>
> We have to make sure that all memory chunks the guest hands back on
> unplug requests will not consume memory in the host. We do this by
> write-protecting that memory chunk in the host and then dropping the
> backing pages. The guest can read this memory (reading from the ZERO
> page) but no longer write to it. For now, this will only work on
> anonymous memory. We will use userfaultfd WP (write-protect mode) to
> avoid creating too many VMAs. Huge pages will require more effort (no
> explicit ZERO page).
>
> As we unplug memory on a fine grained basis (and e.g. not on
> a complete DIMM basis), there is no need to online virtio-mem memory
> as online-movable. Also, memory unplug support for Windows might be
> supported that way. You can find more details in the Q/A section below.
>
>
> The important points here are:
> - After a reboot, every memory the guest sees can be accessed and used.
> (in contrast to e.g. the XEN balloon, see Q/A fore more details)
> - Rebooting into an unmodified guest will not result into random
> crashed. The guest will simply not be able to use all memory without a
> virtio-mem driver.
> - Adding/Removing memory will not require modifying the QEMU command
> line on the migration target. Migration simply works (re-sizing memory
> areas is already part of the migration protocol!). Essentially, this
> makes adding/removing memory to/from a guest way simpler and
> independent of the underlying architecture. If the guest OS can online
> new memory, we can add more memory this way.
> - Unplugged memory can be read. This allows e.g. kexec() without nasty
> modifications. Especially relevant for Windows' kexec() variant.
> - It will play nicely with other things mapped into the address space,
> e.g. also other DIMMs or NVDIMM. virtio-mem will only work on its own
> memory region (in contrast e.g. to virtio-balloon). Especially it will
> not give up ("allocate") memory on other DIMMs, hindering them to get
> unplugged the ACPI way.
> - We can add/remove memory without running into KVM memory slot or other
> (e.g. ACPI slot) restrictions. The granularity in which we can add
> memory is only limited by the granularity the guest can add memory
> (e.g. Windows 2MB, Linux on x86 128MB for now).
> - By not having to online memory as online-movable we don't run into any
> memory restrictions in the guest. E.g. page tables can only be created
> on !movable memory. So while there might be plenty of online-movable
> memory left, allocation of page tables might fail. See Q/A for more
> details.
> - The admin will not have to set memory offline in the guest first in
> order to unplug it. virtio-mem will handle this internally and not
> require interaction with an admin or a guest-agent.
>
> Important restrictions of this concept:
> - Guests without a virtio-mem guest driver can't see that memory.
> - We will always require some boot memory that cannot get unplugged.
> Also, virtio-mem memory (as all other hotplugged memory) cannot become
> DMA memory under Linux. So the boot memory also defines the amount of
> DMA memory.
> - Hibernation/Sleep+Restore while virtio-mem is active is not supported.
> On a reboot/fresh start, the size of the virtio-mem memory area might
> change and a running/loaded guest can't deal with that.
> - Unplug support for hugetlbfs/shmem will take quite some time to
> support. The larger the used page size, the harder for the guest to
> give up memory. We can still use DIMM based hotplug for that.
> - Huge huge pages are problematic, as the guest would have to give up
> e.g. 1GB chunks. This is not expected to be supported. We can still
> use DIMM based hotplug for setups that require that.
> - For any memory we unplug using this mechanism, for now we will still
> have struct pages allocated in the guest. This means, that roughly
> 1.6% of unplugged memory will still be allocated in the guest, being
> unusable.
>
>
> ------------------------------------------------------------------------
> II. Use cases
> ------------------------------------------------------------------------
>
> Of course, we want to deny any access to unplugged memory. In contrast
> to virtio-balloon or other similar ideas (free page hinting), this is
> not about cooperative memory management, but about guarantees. The idea
> is, that both concepts can coexist.
>
> So one use case is of course cloud providers. Customers can add
> or remove memory to/from a VM without having to care about how to
> online memory or in which amount to add memory in the first place in
> order to remove it again. In cloud environments, we care about
> guarantees. E.g. for virtio-balloon a malicious guest can simply reuse
> any deflated memory, and the hypervisor can't even tell if the guest is
> malicious (e.g. a harmless guest reboot might look like a malicious
> guest). For virtio-mem, we guarantee that the guest can't reuse any
> memory that it previously gave up.
>
> But also for ordinary VMs (!cloud), this avoids having to online memory
> in the guest as online-movable and therefore not running into allocation
> problems if there are e.g. many processes needing many page tables on
> !movable memory. Also here, we don't have to know how much memory we
> want to remove some-when in the future before we add memory. (e.g. if we
> add a 128GB DIMM, we can only remove that 128GB DIMM - if we are lucky).
>
> We might be able to support memory unplug for Windows (as for now,
> ACPI unplug is not supported), more details have to be clarified.
>
> As we can grow these memory areas quite easily, another use case might
> be guests that tell us they need more memory. Thinking about VMs to
> protect containers, there seems to be the general problem that we don't
> know how much memory the container will actually need. We could
> implement a mechanism (in virtio-mem or guest driver), by which the
> guest can request more memory. If the hypervisor agrees, it can simply
> give the guest more memory. As this is all handled within QEMU,
> migration is not a problem. Adding more memory will not result in new
> DIMM devices.
>
>
> ------------------------------------------------------------------------
> III. Identified requirements
> ------------------------------------------------------------------------
>
> I considered the following requirements.
>
> NUMA aware:
> We want to be able to add/remove memory to/from NUMA nodes.
> Different page-size support:
> We want to be able to support different page sizes, e.g. because of
> huge pages in the hypervisor or because host and guest have different
> page sizes (powerpc 64k vs 4k).
> Guarantees:
> There has to be no way the guest can reuse unplugged memory without
> host consent. Still, we could implement a mechanism for the guest to
> request more memory. The hypervisor then has to decide how it wants to
> handle that request.
> Architecture independence:
> We want this to work independently of other technologies bound to
> specific architectures, like ACPI.
> Avoid online-movable:
> We don't want to have to online memory in the guest as online-movable
> just to be able to unplug (at least parts of) it again.
> Migration support:
> Be able to migrate without too much hassle. Especially, to handle it
> completely within QEMU (not having to add new devices to the target
> command line).
> Windows support:
> We definitely want to support Windows guests in the long run.
> Coexistence with other hotplug mechanisms:
> Allow to hotplug DIMMs / NVDIMMs, therefore to share the "hotplug"
> address space part with other devices.
> Backwards compatibility:
> Don't break if rebooting into an unmodified guest after having
> unplugged some memory. All memory a freshly booted guest sees must not
> contain memory holes that will crash it if it tries to access it.
>
>
> ------------------------------------------------------------------------
> IV. Possible modifications
> ------------------------------------------------------------------------
>
> Adding a guest->host request mechanism would make sense to e.g. be able
> to request further memory from the hypervisor directly from the guest.
>
> Adding memory will be much easier than removing memory. We can split
> this up and first introduce "adding memory" and later add "removing
> memory". Removing memory will require userfaultfd WP in the hypervisor
> and a special fancy allocator in the guest. So this will take some time.
>
> Adding a mechanism to trade in memory blocks might make sense to allow
> some sort of memory compaction. However I expect this to be highly
> complicated and basically not feasible.
>
> Being able to unplug memory "any" memory instead of only memory
> belonging to the virtio-mem device sounds tempting (and simplifies
> certain parts), however it has a couple of side effects I want to avoid.
> You can read more about that in the Q/A below.
>
>
> ------------------------------------------------------------------------
> V. Prototype
> ------------------------------------------------------------------------
>
> To identify potential problems I developed a very basic prototype. It
> is incomplete, full of hacks and most probably broken in various ways.
> I used it only in the given setup, only on x86 and only with an initrd.
>
> It uses a fixed page size of 256k for now, has a very ugly allocator
> hack in the guest, the virtio protocol really needs some tuning and
> an async job interface towards the user is missing. Instead of using
> userfaultfd WP, I am using simply mprotect() in this prototype. Basic
> migration works (not involving userfaultfd).
>
> Please, don't even try to review it (that's why I will also not attach
> any patches to this mail :) ), just use this as an inspiration what this
> could look like. You can find the latest hack at:
>
> QEMU: https://github.com/davidhildenbrand/qemu/tree/virtio-mem
>
> Kernel: https://github.com/davidhildenbrand/linux/tree/virtio-mem
>
> Use the kernel in the guest and make sure to compile the virtio-mem
> driver into the kernel (CONFIG_VIRTIO_MEM=y). A host kernel patch is
> contained to allow atomic resize of KVM memory regions, however it is
> pretty much untested.
>
>
> 1. Starting a guest with virtio-mem memory:
> We will create a guest with 2 NUMA nodes and 4GB of "boot + DMA"
> memory. This memory is visible also to guests without virtio-mem.
> Also, we will add 4GB to NUMA node 0 and 3GB to NUMA node 1 using
> virtio-mem. We allow both virtio-mem devices to grow up to 8GB. The
> last 4 lines are the important part.
>
> --> qemu/x86_64-softmmu/qemu-system-x86_64 \
> --enable-kvm
> -m 4G,maxmem=20G \
> -smp sockets=2,cores=2 \
> -numa node,nodeid=0,cpus=0-1 -numa node,nodeid=1,cpus=2-3 \
> -machine pc \
> -kernel linux/arch/x86_64/boot/bzImage \
> -nodefaults \
> -chardev stdio,id=serial \
> -device isa-serial,chardev=serial \
> -append "console=ttyS0 rd.shell rd.luks=0 rd.lvm=0" \
> -initrd /boot/initramfs-4.10.8-200.fc25.x86_64.img \
> -chardev socket,id=monitor,path=/var/tmp/monitor,server,nowait \
> -mon chardev=monitor,mode=readline \
> -object memory-backend-ram,id=mem0,size=4G,max-size=8G \
> -device virtio-mem-pci,id=reg0,memdev=mem0,node=0 \
> -object memory-backend-ram,id=mem1,size=3G,max-size=8G \
> -device virtio-mem-pci,id=reg1,memdev=mem1,node=1
>
> 2. Listing current memory assignment:
>
> --> (qemu) info memory-devices
> Memory device [virtio-mem]: "reg0"
> addr: 0x140000000
> node: 0
> size: 4294967296
> max-size: 8589934592
> memdev: /objects/mem0
> Memory device [virtio-mem]: "reg1"
> addr: 0x340000000
> node: 1
> size: 3221225472
> max-size: 8589934592
> memdev: /objects/mem1
> --> (qemu) info numa
> 2 nodes
> node 0 cpus: 0 1
> node 0 size: 6144 MB
> node 1 cpus: 2 3
> node 1 size: 5120 MB
>
> 3. Resize a virtio-mem device: Unplugging memory.
> Setting reg0 to 2G (remove 2G from NUMA node 0)
>
> --> (qemu) virtio-mem reg0 2048
> virtio-mem reg0 2048
> --> (qemu) info numa
> info numa
> 2 nodes
> node 0 cpus: 0 1
> node 0 size: 4096 MB
> node 1 cpus: 2 3
> node 1 size: 5120 MB
>
> 4. Resize a virtio-mem device: Plugging memory
> Setting reg0 to 8G (adding 6G to NUMA node 0) will replug 2G and plug
> 4G, automatically re-sizing the memory area. You might experience
> random crashes at this point if the host kernel missed a KVM patch
> (as the memory slot is not re-sized in an atomic fashion).
>
> --> (qemu) virtio-mem reg0 8192
> virtio-mem reg0 8192
> --> (qemu) info numa
> info numa
> 2 nodes
> node 0 cpus: 0 1
> node 0 size: 10240 MB
> node 1 cpus: 2 3
> node 1 size: 5120 MB
>
> 5. Resize a virtio-mem device: Try to unplug all memory.
> Setting reg0 to 0G (removing 8G from NUMA node 0) will not work. The
> guest will not be able to unplug all memory. In my example, 164M
> cannot be unplugged (out of memory).
>
> --> (qemu) virtio-mem reg0 0
> virtio-mem reg0 0
> --> (qemu) info numa
> info numa
> 2 nodes
> node 0 cpus: 0 1
> node 0 size: 2212 MB
> node 1 cpus: 2 3
> node 1 size: 5120 MB
> --> (qemu) info virtio-mem reg0
> info virtio-mem reg0
> Status: ready
> Request status: vm-oom
> Page size: 2097152 bytes
> --> (qemu) info memory-devices
> Memory device [virtio-mem]: "reg0"
> addr: 0x140000000
> node: 0
> size: 171966464
> max-size: 8589934592
> memdev: /objects/mem0
> Memory device [virtio-mem]: "reg1"
> addr: 0x340000000
> node: 1
> size: 3221225472
> max-size: 8589934592
> memdev: /objects/mem1
>
> At any point, we can migrate our guest without having to care about
> modifying the QEMU command line on the target side. Simply start the
> target e.g. with an additional '-incoming "exec: cat IMAGE"' and you're
> done.
>
> ------------------------------------------------------------------------
> VI. Problems to solve / things to sort out / missing in prototype
> ------------------------------------------------------------------------
>
> General:
> - We need an async job API to send the unplug/replug/plug requests to
> the guest and query the state. [medium/hard]
> - Handle various alignment problems. [medium]
> - We need a virtio spec
>
> Relevant for plug:
> - Resize QEMU memory regions while the guest is running (esp. grow).
> While I implemented a demo solution for KVM memory slots, something
> similar would be needed for vhost. Re-sizing of memory slots has to be
> an atomic operation. [medium]
> - NUMA: Most probably the NUMA node should not be part of the virtio-mem
> device, this should rather be indicated via e.g. ACPI. [medium]
> - x86: Add the complete possible memory to the a820 map as reserved.
> [medium]
> - x86/powerpc/...: Indicate to which NUMA node the memory belongs using
> ACPI. [medium]
> - x86/powerpc/...: Share address space with ordinary DIMMS/NVDIMMs, for
> now this is blocked for simplicity. [medium/hard]
> - If the bitmaps become too big, migrate them like memory. [medium]
>
> Relevant for unplug:
> - Allocate memory in Linux from a specific memory range. Windows has a
> nice interface for that (at least it looks nice when reading the API).
> This could be done using fake NUMA nodes or a new ZONE. My prototype
> just uses a very ugly hack. [very hard]
> - Use userfaultfd WP (write-protect) insted of mprotect. Especially,
> have multiple userfaultfd user in QEMU at a time (postcopy).
> [medium/hard]
>
> Stuff for the future:
> - Huge pages are problematic (no ZERO page support). This might not be
> trivial to support. [hard/very hard]
> - Try to free struct pages, to avoid the 1.6% overhead [very very hard]
>
>
> ------------------------------------------------------------------------
> VII. Questions
> ------------------------------------------------------------------------
>
> To get unplug working properly, it will require quite some effort,
> that's why I want to get some basic feedback before continuing working
> on a RFC implementation + RFC virtio spec.
>
> a) Did I miss anything important? Are there any ultimate blockers that I
> ignored? Any concepts that are broken?
>
> b) Are there any alternatives? Any modifications that could make life
> easier while still taking care of the requirements?
>
> c) Are there other use cases we should care about and focus on?
>
> d) Am I missing any requirements? What else could be important for
> !cloud and cloud?
>
> e) Are there any possible solutions to the allocator problem (allocating
> memory from a specific memory area)? Please speak up!
>
> f) Anything unclear?
>
> e) Any feelings about this? Yay or nay?
>
>
> As you reached this point: Thanks for having a look!!! Highly appreciated!
>
>
> ------------------------------------------------------------------------
> VIII. Q&A
> ------------------------------------------------------------------------
>
> ---
> Q: What's the problem with ordinary memory hot(un)plug?
>
> A: 1. We can only unplug in the granularity we plugged. So we have to
> know in advance, how much memory we want to remove later on. If we
> plug a 2G dimm, we can only unplug a 2G dimm.
> 2. We might run out of memory slots. Although very unlikely, this
> would strike if we try to always plug small modules in order to be
> able to unplug again (e.g. loads of 128MB modules).
> 3. Any locked page in the guest can hinder us from unplugging a dimm.
> Even if memory was onlined as online_movable, a single locked page
> can hinder us from unplugging that memory dimm.
> 4. Memory has to be onlined as online_movable. If we don't put that
> memory into the movable zone, any non-movable kernel allocation
> could end up on it, turning the complete dimm unpluggable. As
> certain allocations cannot go into the movable zone (e.g. page
> tables), the ratio between online_movable/online memory depends on
> the workload in the guest. Ratios of 50% -70% are usually fine.
> But it could happen, that there is plenty of memory available,
> but kernel allocations fail. (source: Andrea Arcangeli)
> 5. Unplugging might require several attempts. It takes some time to
> migrate all memory from the dimm. At that point, it is then not
> really obvious why it failed, and whether it could ever succeed.
> 6. Windows does support memory hotplug but not memory hotunplug. So
> this could be a way to support it also for Windows.
> ---
> Q: Will this work with Windows?
>
> A: Most probably not in the current form. Memory has to be at least
> added to the a820 map and ACPI (NUMA). Hyper-V ballon is also able to
> hotadd memory using a paravirtualized interface, so there are very
> good chances that this will work. But we won't know for sure until we
> also start prototyping.
> ---
> Q: How does this compare to virtio-balloo?
>
> A: In contrast to virtio-balloon, virtio-mem
> 1. Supports multiple page sizes, even different ones for different
> virtio-mem devices in a guest.
> 2. Is NUMA aware.
> 3. Is able to add more memory.
> 4. Doesn't work on all memory, but only on the managed one.
> 5. Has guarantees. There is now way for the guest to reclaim memory.
> ---
> Q: How does this compare to XEN balloon?
>
> A: XEN balloon also has a way to hotplug new memory. However, on a
> reboot, the guest will "see" more memory than it actually has.
> Compared to XEN balloon, virtio-mem:
> 1. Supports multiple page sizes.
> 2. Is NUMA aware.
> 3. The guest can survive a reboot into a system without the guest
> driver. If the XEN guest driver doesn't come up, the guest will
> get killed once it touches too much memory.
> 4. Reboots don't require any hacks.
> 5. The guest knows which memory is special. And it remains special
> during a reboot. Hotplugged memory not suddenly becomes base
> memory. The balloon mechanism will only work on a specific memory
> area.
> ---
> Q: How does this compare to Hyper-V balloon?
>
> A: Based on the code from the Linux Hyper-V balloon driver, I can say
> that Hyper-V also has a way to hotplug new memory. However, memory
> will remain plugged on a reboot. Therefore, the guest will see more
> memory than the hypervisor actually wants to assign to it.
> Virtio-mem in contrast:
> 1. Supports multiple page sizes.
> 2. Is NUMA aware.
> 3. I have no idea what happens under Hyper-v when
> a) rebooting into a guest without a fitting guest driver
> b) kexec() touches all memory
> c) the guest misbehaves
> 4. The guest knows which memory is special. And it remains special
> during a reboot. Hotpplugged memory not suddenly becomes base
> memory. The balloon mechanism will only work on a specific memory
> area.
> In general, it looks like the hypervisor has to deal with malicious
> guests trying to access more memory than desired by providing enough
> swap space.
> ---
> Q: How is virtio-mem NUMA aware?
>
> A: Each virtio-mem device belongs exactly to one NUMA node (if NUMA is
> enabled). As we can resize these regions separately, we can control
> from/to which node to remove/add memory.
> ---
> Q: Why do we need support for multiple page sizes?
>
> A: If huge pages are used in the host, we can only guarantee that they
> are not accessible by the guest anymore, if the guest gives up memory
> in this granularity. We prepare for that. Also, powerpc can have 64k
> pages in the host but 4k pages in the guest. So the guest must only
> give up 64k chunks. In addition, unplugging 4k pages might be bad
> when it comes to fragmentation. My prototype currently uses 256k. We
> can make this configurable - and it can vary for each virtio-mem
> device.
> ---
> Q: What are the limitations with paravirtualized memory hotplug?
>
> A: The same as for DIMM based hotplug, but we don't run out of any
> memory/ACPI slots. E.g. on x86 Linux, only 128MB chunks can be
> hotplugged, on x86 Windows it's 2MB. In addition, of course we
> have to take care of maximum address limits in the guest. The idea
> is to communicate these limits to the hypervisor via virtio-mem,
> to give hints when trying to add/remove memory.
> ---
> Q: Why not simply unplug *any* memory like virtio-balloon does?
>
> A: This could be done and a previous prototype did it like that.
> However, there are some points to consider here.
> 1. If we combine this with ordinary memory hotplug (DIMM), we most
> likely won't be able to unplug DIMMs anymore as virtio-mem memory
> gets "allocated" on these.
> 2. All guests using virtio-mem cannot use huge pages as backing
> storage at all (as virtio-mem only supports anonymous pages).
> 3. We need to track unplugged memory for the complete address space,
> so we need a global state in QEMU. Bitmaps get bigger. We will not
> be abe to dynamically grow the bitmaps for a virtio-mem device.
> 4. Resolving/checking memory to be unplugged gets significantly
> harder. How should the guest know which memory it can unplug for a
> specific virtio-mem device? E.g. if NUMA is active, only that NUMA
> node to which a virtio-mem device belongs can be used.
> 5. We will need userfaultfd handler for the complete address space,
> not just for the virtio-mem managed memory.
> Especially, if somebody hotplugs a DIMM, we dynamically will have
> to enable the userfaultfd handler.
> 6. What shall we do if somebody hotplugs a DIMM with huge pages? How
> should we tell the guest, that this memory cannot be used for
> unplugging?
> In summary: This concept is way cleaner, but also harder to
> implement.
> ---
> Q: Why not reuse virtio-balloon?
>
> A: virtio-balloon is for cooperative memory management. It has a fixed
> page size and will deflate in certain situations. Any change we
> introduce will break backwards compatibility. virtio-balloon was not
> designed to give guarantees. Nobody can hinder the guest from
> deflating/reusing inflated memory. In addition, it might make perfect
> sense to have both, virtio-balloon and virtio-mem at the same time,
> especially looking at the DEFLATE_ON_OOM or STATS features of
> virtio-balloon. While virtio-mem is all about guarantees, virtio-
> balloon is about cooperation.
> ---
> Q: Why not reuse acpi hotplug?
>
> A: We can easily run out of slots, migration in QEMU will just be
> horrible and we don't want to bind virtio* to architecture specific
> technologies.
> E.g. thinking about s390x - no ACPI. Also, mixing an ACPI driver with
> a virtio-driver sounds very weird. If the virtio-driver performs the
> hotplug itself, we might later perform some extra tricks: e.g.
> actually unplug certain regions to give up some struct pages.
>
> We want to manage the way memory is added/removed completely in QEMU.
> We cannot simply add new device from within QEMU and expect that
> migration in QEMU will work.
> ---
> Q: Why do we need resizable memory regions?
>
> A: Migration in QEMU is special. Any device we have on our source VM has
> to already be around on our target VM. So simply creating random
> devides internally in QEMU is not going to work. The concept of
> resizable memory regions in QEMU already exists and is part of the
> migration protocol. Before memory is migrated, the memory is resized.
> So in essence, this makes migration support _a lot_ easier.
>
> In addition, we won't run in any slot number restriction when
> automatically managing how to add memory in QEMU.
> ---
> Q: Why do we have to resize memory regions on a reboot?
>
> A: We have to compensate all memory that has been unplugged for that
> area by shrinking it, so that a fresh guest can use all memory when
> initializing the virtio-mem device.
> ---
> Q: Why do we need userfaultfd?
>
> A: mprotect() will create a lot of VMAs in the kernel. This will degrade
> performance and might even fail at one point. userfaultfd avoids this
> by not creating a new VMA for every protected range. userfaultfd WP
> is currently still under development and suffers from false positives
> that make it currently impossible to properly integrate this into the
> prototype.
> ---
> Q: Why do we have to allow reading unplugged memory?
>
> A: E.g. if the guest crashes and want's to write a memory dump, it will
> blindly access all memory. While we could find ways to fixup kexec,
> Windows dumps might be more problematic. Allowing the guest to read
> all memory (resulting in reading all 0's) safes us from a lot of
> trouble.
>
> The downside is, that page tables full of zero pages might be
> created. (we might be able to find ways to optimize this)
> ---
> Q: Will this work with postcopy live-migration?
>
> A: Not in the current form. And it doesn't really make sense to spend
> time on it as long as we don't use userfaultfd. Combining both
> handlers will be interesting. It can be done with some effort on the
> QEMU side.
> ---
> Q: What's the problem with shmem/hugetlbfs?
>
> A: We currently rely on the ZERO page to be mapped when the guest tries
> to read unplugged memory. For shmem/hugetlbfs, there is no ZERO page,
> so read access would result in memory getting populated. We could
> either introduce an explicit ZERO page, or manage it using one dummy
> ZERO page (using regular usefaultfd, allow only one such page to be
> mapped at a time). For now, only anonymous memory.
> ---
> Q: Ripping out random page ranges, won't this fragment our guest memory?
>
> A: Yes, but depending on the virtio-mem page size, this might be more or
> less problematic. The smaller the virtio-mem page size, the more we
> fragment and make small allocations fail. The bigger the virtio-mem
> page size, the higher the chance that we can't unplug any more
> memory.
> ---
> Q: Why can't we use memory compaction like virtio-balloon?
>
> A: If the virtio-mem page size > PAGE_SIZE, we can't do ordinary
> page migration, migration would have to be done in blocks. We could
> later add an guest->host virtqueue, via which the guest can
> "exchange" memory ranges. However, also mm has to support this kind
> of migration. So it is not completely out of scope, but will require
> quite some work.
> ---
> Q: Do we really need yet another paravirtualized interface for this?
>
> A: You tell me :)
> ---
>
> Thanks,
>
> David
>
--
Thanks,
David
