[...]
Anyhow, please suggest a way to handle it via a single flag in the kernel --
which would be some kind of heuristic as we know from MAP_POPULATE. Having
an alternative at hand would make it easier to discuss this topic further. I
certainly *don't* want MAP_POPULATE semantics when it comes to
MADV_POPULATE, especially when it comes to shared mappings. Not useful in
QEMU now and in the future.
OK, this point is still not entirely clear to me. Elsewhere you are
saying that QEMU cannot use MAP_POPULATE because it ignores errors
and also it doesn't support sparse mappings because they apply to the
whole mmap. These are all clear but it is less clear to me why the same
semantic is not applicable for QEMU when used through madvise interface
which can handle both of those.
It's a combination of things:
a) MAP_POPULATE never was an option simply because of deferred
"prealloc=on" handling in QEMU, happening way after we created the
memmap. Further it doesn't report if there was an error, which is
another reason why it's basically useless for QEMU use cases.
b) QEMU uses manual read-write prefaulting for "preallocation", for
example, to avoid SIGBUS on hugetlbfs or shmem at runtime. There are
cases where we absolutely want to avoid crashing the VM later just
because of a user error. MAP_POPULATE does *not* do what we want for
shared mappings, because it triggers a read fault.
c) QEMU uses the same mechanism for prefaulting in RT environments,
where we want to avoid any kind of pagefault, using mlock() etc.
d) MAP_POPULATE does not apply to sparse memory mappings that I'll be
using more heavily in QEMU, also for the purpose of preallocation
with virtio-mem.
See the current QEMU code along with a comment in
https://github.com/qemu/qemu/blob/972e848b53970d12cb2ca64687ef8ff797fb6236/util/oslib-posix.c#L496
it's especially bad for PMEM ("wear on the storage backing"), which is
why we have to trust on users not to trigger preallocation/prefaulting
on PMEM, otherwise (as already expressed via bug reports) we waste a lot
of time when backing VMs on PMEM or forwarding NVDIMMs, unnecessarily
read/writing (slow) DAX.
Do I get it right that you really want to emulate the full fledged write
fault to a) limit another write fault when the content is actually
modified and b) prevent from potential errors during the write fault
(e.g. mkwrite failing on the fs data)?
Yes, for the use case of "preallocation" in QEMU. See the QEMU link.
But again, the thing that makes it more complicated is that I can come
up with some use cases that want to handle "shared mappings of ordinary
files" a little better. Or the usefaultfd-wp example I gave, where
prefaulting via MADV_POPULATE_READ can roughly half the population time.
We could make MADV_POPULATE act depending on the readability/writability of
a mapping. Use MADV_POPULATE_WRITE on writable mappings, use
MADV_POPULATE_READ on readable mappings. Certainly not perfect for use cases
where you have writable mappings that are mostly read only (as in the
example with fake-NVDIMMs I gave ...), but if it makes people happy, fine
with me. I mostly care about MADV_POPULATE_WRITE.
Yes, this is where my thinking was going as well. Essentially define
MADV_POPULATE as "Populate the mapping with the memory based on the
mapping access." This looks like a straightforward semantic to me and it
doesn't really require any deep knowledge of internals.
Now, I was trying to compare which of those would be more tricky to
understand and use and TBH I am not really convinced any of the two is
much better. Separate READ/WRITE modes are explicit which can be good
but it will require quite an advanced knowledge of the #PF behavior.
On the other hand MADV_POPULATE would require some tricks like mmap,
madvise and mprotect(to change to writable) when the data is really
written to. I am not sure how much of a deal this would be for QEMU for
example.
IIRC, at the time we enable background snapshotting, the VM is running
and we cannot temporarily mprotect(PROT_READ) without making the guest
crash. But again, uffd-wp handling is somewhat a special case because
the implementation in the kernel is really suboptimal.
The reason I chose MADV_POPULATE_READ + MADV_POPULATE_WRITE is because
it really mimics what user space currently does to get the job done.
I guess the important part to document is that "be careful when using
MADV_POPULATE_READ because it might just populate the shared zeropage"
and "be careful with MADV_POPULATE_WRITE because it will do the same as
when writing to every page: dirty the pages such that they will have to
be written back when backed by actual files".
The current MAN page entry for MADV_POPULATE_READ reads:
"
Populate (prefault) page tables readable for the whole range without
actually reading. Depending on the underlying mapping, map the shared
zeropage, preallocate memory or read the underlying file. Do not
generate SIGBUS when populating fails, return an error instead.
If MADV_POPULATE_READ succeeds, all page tables have been populated
(prefaulted) readable once. If MADV_POPULATE_READ fails, some page
tables might have been populated.
MADV_POPULATE_READ cannot be applied to mappings without read
permissions and special mappings marked with the kernel-internal
VM_PFNMAP and VM_IO.
Note that with MADV_POPULATE_READ, the process can still be killed at
any moment when the system runs out of memory.
"
So, all that being said, I am not really sure. I am not really happy
about READ/WRITE split but if a simpler interface is going to be a bad
fit for existing usecases then I believe a proper way to go is the
document the more complex interface thoroughly.
I think with the split we are better off long term without requiring
workarounds (mprotect()) to make some use cases work in the long term.
But again, if there is a good justification why a single MADV_POPULATE
make sense, I'm happy to change it. Again, for me, the most important
thing long-term is MADV_POPULATE_WRITE because that's really what QEMU
mainly uses right now for preallocation. But I can see use cases for
MADV_POPULATE_READ as well.
Thanks for your input!
--
Thanks,
David / dhildenb
