Hi, (changed the subject and added CRIU folks) On Tue, Jan 29, 2019 at 06:40:58PM -0500, Andrea Arcangeli wrote: > Hello, > > -- > > In addition to the above "NUMA remote THP vs NUMA local non-THP > tradeoff" topic, there are other developments in "userfaultfd" land that > are approaching merge readiness and that would be possible to provide a > short overview about: > > - Peter Xu made significant progress in finalizing the userfaultfd-WP > support over the last few months. That feature was planned from the > start and it will allow userland to do some new things that weren't > possible to achieve before. In addition to synchronously blocking > write faults to be resolved by an userland manager, it has also the > ability to obsolete the softdirty feature, because it can provide > the same information, but with O(1) complexity (as opposed of the > current softdirty O(N) complexity) similarly to what the Page > Modification Logging (PML) does in hardware for EPT write accesses. We (CRIU) have some concerns about obsoleting soft-dirty in favor of uffd-wp. If there are other soft-dirty users these concerns would be relevant to them as well. With soft-dirty we collect the information about the changed memory every pre-dump iteration in the following manner: * freeze the tasks * find entries in /proc/pid/pagemap with SOFT_DIRTY set * unfreeze the tasks * dump the modified pages to disk/remote host While we do need to traverse the /proc/pid/pagemap to identify dirty pages, in between the pre-dump iterations and during the actual memory dump the tasks are running freely. If we are to switch to uffd-wp, every write by the snapshotted/migrated task will incur latency of uffd-wp processing by the monitor. We'd need to see how this affects overall slowdown of the workload under migration before moving forward with obsoleting soft-dirty. > - Blake Caldwell maintained the UFFDIO_REMAP support to atomically > remove memory from a mapping with userfaultfd (which can't be done > with a copy as in UFFDIO_COPY and it requires a slow TLB flush to be > safe) as an alternative to host swapping (which of course also > requires a TLB flush for similar reasons). Notably UFFDIO_REMAP was > rightfully naked early on and quickly replaced by UFFDIO_COPY which > is more optimal to add memory to a mapping is small chunks, but we > can't remove memory with UFFDIO_COPY and UFFDIO_REMAP should be as > efficient as it gets when it comes to removing memory from a > mapping. If we are to discuss userfaultfd, I'd like also to bring the subject of COW mappings. The pages populated with UFFDIO_COPY cannot be COW-shared between related processes which unnecessarily increases memory footprint of a migrated process tree. I've posted a patch [1] a (real) while ago, but nobody reacted and I've put this aside. Maybe it's time to discuss it again :) > Thank you, > Andrea > [1] https://lwn.net/ml/linux-api/20180328101729.GB1743%40rapoport-lnx/ -- Sincerely yours, Mike.
