On 31/07/2024 18:51, David Hildenbrand wrote: > On 31.07.24 19:01, Usama Arif wrote: >> >> >> On 30/07/2024 21:25, David Hildenbrand wrote: >>> On 30.07.24 19:22, Usama Arif wrote: >>>> >>>> >>>> On 30/07/2024 17:11, David Hildenbrand wrote: >>>>> On 30.07.24 17:19, Usama Arif wrote: >>>>>> >>>>>> >>>>>> On 30/07/2024 16:14, Usama Arif wrote: >>>>>>> >>>>>>> >>>>>>> On 30/07/2024 15:35, David Hildenbrand wrote: >>>>>>>> On 30.07.24 14:45, Usama Arif wrote: >>>>>>>>> The current upstream default policy for THP is always. However, Meta >>>>>>>>> uses madvise in production as the current THP=always policy vastly >>>>>>>>> overprovisions THPs in sparsely accessed memory areas, resulting in >>>>>>>>> excessive memory pressure and premature OOM killing. >>>>>>>>> Using madvise + relying on khugepaged has certain drawbacks over >>>>>>>>> THP=always. Using madvise hints mean THPs aren't "transparent" and >>>>>>>>> require userspace changes. Waiting for khugepaged to scan memory and >>>>>>>>> collapse pages into THP can be slow and unpredictable in terms of performance >>>>>>>>> (i.e. you dont know when the collapse will happen), while production >>>>>>>>> environments require predictable performance. If there is enough memory >>>>>>>>> available, its better for both performance and predictability to have >>>>>>>>> a THP from fault time, i.e. THP=always rather than wait for khugepaged >>>>>>>>> to collapse it, and deal with sparsely populated THPs when the system is >>>>>>>>> running out of memory. >>>>>>>>> >>>>>>>>> This patch-series is an attempt to mitigate the issue of running out of >>>>>>>>> memory when THP is always enabled. During runtime whenever a THP is being >>>>>>>>> faulted in or collapsed by khugepaged, the THP is added to a list. >>>>>>>>> Whenever memory reclaim happens, the kernel runs the deferred_split >>>>>>>>> shrinker which goes through the list and checks if the THP was underutilized, >>>>>>>>> i.e. how many of the base 4K pages of the entire THP were zero-filled. >>>>>>>>> If this number goes above a certain threshold, the shrinker will attempt >>>>>>>>> to split that THP. Then at remap time, the pages that were zero-filled are >>>>>>>>> not remapped, hence saving memory. This method avoids the downside of >>>>>>>>> wasting memory in areas where THP is sparsely filled when THP is always >>>>>>>>> enabled, while still providing the upside THPs like reduced TLB misses without >>>>>>>>> having to use madvise. >>>>>>>>> >>>>>>>>> Meta production workloads that were CPU bound (>99% CPU utilzation) were >>>>>>>>> tested with THP shrinker. The results after 2 hours are as follows: >>>>>>>>> >>>>>>>>> | THP=madvise | THP=always | THP=always >>>>>>>>> | | | + shrinker series >>>>>>>>> | | | + max_ptes_none=409 >>>>>>>>> ----------------------------------------------------------------------------- >>>>>>>>> Performance improvement | - | +1.8% | +1.7% >>>>>>>>> (over THP=madvise) | | | >>>>>>>>> ----------------------------------------------------------------------------- >>>>>>>>> Memory usage | 54.6G | 58.8G (+7.7%) | 55.9G (+2.4%) >>>>>>>>> ----------------------------------------------------------------------------- >>>>>>>>> max_ptes_none=409 means that any THP that has more than 409 out of 512 >>>>>>>>> (80%) zero filled filled pages will be split. >>>>>>>>> >>>>>>>>> To test out the patches, the below commands without the shrinker will >>>>>>>>> invoke OOM killer immediately and kill stress, but will not fail with >>>>>>>>> the shrinker: >>>>>>>>> >>>>>>>>> echo 450 > /sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_none >>>>>>>>> mkdir /sys/fs/cgroup/test >>>>>>>>> echo $$ > /sys/fs/cgroup/test/cgroup.procs >>>>>>>>> echo 20M > /sys/fs/cgroup/test/memory.max >>>>>>>>> echo 0 > /sys/fs/cgroup/test/memory.swap.max >>>>>>>>> # allocate twice memory.max for each stress worker and touch 40/512 of >>>>>>>>> # each THP, i.e. vm-stride 50K. >>>>>>>>> # With the shrinker, max_ptes_none of 470 and below won't invoke OOM >>>>>>>>> # killer. >>>>>>>>> # Without the shrinker, OOM killer is invoked immediately irrespective >>>>>>>>> # of max_ptes_none value and kill stress. >>>>>>>>> stress --vm 1 --vm-bytes 40M --vm-stride 50K >>>>>>>>> >>>>>>>>> Patches 1-2 add back helper functions that were previously removed >>>>>>>>> to operate on page lists (needed by patch 3). >>>>>>>>> Patch 3 is an optimization to free zapped tail pages rather than >>>>>>>>> waiting for page reclaim or migration. >>>>>>>>> Patch 4 is a prerequisite for THP shrinker to not remap zero-filled >>>>>>>>> subpages when splitting THP. >>>>>>>>> Patches 6 adds support for THP shrinker. >>>>>>>>> >>>>>>>>> (This patch-series restarts the work on having a THP shrinker in kernel >>>>>>>>> originally done in >>>>>>>>> https://lore.kernel.org/all/cover.1667454613.git.alexlzhu@xxxxxx/. >>>>>>>>> The THP shrinker in this series is significantly different than the >>>>>>>>> original one, hence its labelled v1 (although the prerequisite to not >>>>>>>>> remap clean subpages is the same).) >>>>>>>> >>>>>>>> As shared previously, there is one issue with uffd (even when currently not active for a VMA!), where we must not zap present page table entries. >>>>>>>> >>>>>>>> Something that is always possible (assuming no GUP pins of course, which) is replacing the zero-filled subpages by shared zeropages. >>>>>>>> >>>>>>>> Is that being done in this patch set already, or are we creating pte_none() entries? >>>>>>>> >>>>>>> >>>>>>> I think thats done in Patch 4/6. In function try_to_unmap_unused, we have below which I think does what you are suggesting? i.e. point to shared zeropage and not clear pte for uffd armed vma. >>>>>>> >>>>>>> if (userfaultfd_armed(pvmw->vma)) { >>>>>>> newpte = pte_mkspecial(pfn_pte(page_to_pfn(ZERO_PAGE(pvmw->address)), >>>>>>> pvmw->vma->vm_page_prot)); >>>>>>> ptep_clear_flush(pvmw->vma, pvmw->address, pvmw->pte); >>>>>>> set_pte_at(pvmw->vma->vm_mm, pvmw->address, pvmw->pte, newpte); >>>>>>> } >>>>>> >>>>>> >>>>>> Ah are you suggesting userfaultfd_armed(pvmw->vma) will evaluate to false even if its uffd? I think something like below would work in that case. >>>>> >>>>> I remember one ugly case in QEMU with postcopy live-migration where we must not zap zero-filled pages. I am not 100% regarding THP (if it could be enabled at that point), but imagine the following >>>>> >>>>> 1) mmap(), enable THP >>>>> 2) Migrate a bunch of pages from the source during precopy (writing to >>>>> the memory). Might end up creating THPs (during fault/khugepaged) >>>>> 3) Register UFFD on the VMA >>>>> 4) Disable new THPs from forming via MADV_NOHUGEPAGE on the VMA >>>>> 5) Discard any pages that have been re-dirtied or not migrated yet >>>>> 6) Migrate-on-demand any holes using uffd >>>>> >>>>> >>>>> If we discard zero-filled pages between 2) and 3) we might get wrong uffd notifications in 6 for pages that have already been migrated). >>>>> >>>>> I'll have to check if that actually happens in that sequence in QEMU: if QEMU would disable THP right before 2) we would be safe. But I recall that it is not the case :/ >>>>> >>>>> >>>> >>>> Thanks for the example! >>>> >>>> Just to understand the issue better, as I am not very familiar with live-migration code, the problem is only for zero-filled pages that were migrated, right? If a THP is created and a subpage of it was a zero-page that was migrated and its split before VMA is armed with uffd, userfaultfd_armed(pvmw->vma) will return false when splitting and it will become pte_none. And afterwards when the destination faults on it, uffd will see that its pte_clear and will request the zero-page back from source. Uffd will then have to get the page again from source. >>> >>> Something like that, but I recall that if it detects that it already migrated the page stuff will go wrong. >>> >>> IIRC QEMU maintains receive bitmaps about which pages it already received+placed. Staring at migrate_send_rp_req_pages(), QEMU ignores the uffd notification if it finds that the page was already received and would not ask the source again. >>> >>> So if we turn some PTEs that map zeroed-pages into pte-none we could run into trouble, because we will never try requesting/placing that page again and our VM will just be stuck forever. >>> >>> I wonder if other uffd users could similarly be affected. But maybe they don't place pages into the VMA before registering uffd. >>> >>> I'll try to double-check when QEMU would disable THP. But it could also be that that behavior changed between QEMU versions. >>> >>>> >>>> If I understand the example correctly, the below diff over patch 6 should be good? i.e. just point to the empty_zero_page instead of doing pte_clear. This should still use the same amount of memory, although ptep_clear_flush means it might be slighly more expensive. >>> >>> There are rare cases where we must not use the shared zeropage (mm_forbids_zeropage()), that must be handled. >>> >>> I assume we could optimize here if uffd is not configured in. Further, what QEMU does is sense right at the beginning by temporarily registering uffd on some other VMA if it is even supported. That could be used as an indication ("ever used uffd -> don't turn PTEs none"). But again, no idea what other uffd users might be relying on :/ >>> >>> In I wonder if some applications could rely on anon memory not suddenly "vanishing" form the PTEs (for example relying on pagemap like tools/testing/selftests/mm/cow.c) would. I don't think a lot of applications would do that, though. >>> >> >> >> I had a deeper look at how QEMU handles migration and at how kernel handles THPs in other places with respect to uffd. I hope I understood the migration code correctly :) >> >> QEMU: >> There are 2 types of "migrations" [1], migration_thread and bg_migration_thread >> - migration_thread supports postcopy live migration, but doesn't use uffd. So postcopy live migration doesn't use uffd as far as I can tell looking at the function [2]. >> - bg_migration_thread [3]: this isn't really live migration, but background snapshot. There is no postcopy and it works similar to pre-copy. From what I understand, uffd is registered before any pages are migrated [4]. >> > > Ignore the second. What QEMU ends up using is precopy + postcopy. You can start precopy migration and decide at some point that you want to switch to postcopy (this is what I trigger below, let me know if you want a simple way to reproduce it). > >> So the way these 2 threads work in qemu is: its either postcopy without uffd, or precopy-style snapshot with uffd registered at the start, then the current patch in this series is good, i.e. only use zeropage if userfaultfd_armed(pvmw->vma), otherwise pte_clear. I can't see postcopy with uffd anywhere in qemu which doesn all the steps 1-6 that you mentioned above, but I might not be looking at the right place? > > "I can't see postcopy with uffd anywhere" -- most of it lives in migration/postcopy-ram.c. > > See postcopy_ram_incoming_setup() where most of the UFFD setup logic happens. > Thanks for pointing to this, I did not look at this before :( postcopy_ram_incoming_setup clears it up. >> >> Kernel: >> From a kernel perspective, if we look at khugepaged [5], it just takes into account if the vma is currently registered as uffd. So maybe this scenario can happen? >> 1) THP is enabled on VMA, UFFD is not yet registered. >> 2) dest accesses a 4K page (lets call it 4Kpage1), khugepaged collapses that page into 2M THP as the following 511 4K pages were pte_none [5], as VMA is not uffd armed yet. >> 3) UFFD is registered + MADV_NOHUGEPAGE is set. No further collapse will happen, but the THPs that were created in step 2 will remain. MADV_NOHUGEPAGE doesn't split existing THP created in 2. >> 4) dest tries to access 4Kpage2, the address right after 4Kpage1. As khugepaged collapsed it, there won't be a pagefault, and dest will read it as a zero-filled page, even if UFFD handler wanted to give some non-zero data filled page. I think anyone who wrote the uffd handler would want there to be a pagefault and give the non-zero data filled page and would not expect dest to see a zero-filled page. >> >> What I am trying to say with the above example is: >> - UFFD registeration + MADV_NOHUGEPAGE should be done by all applications before any data in the region is accessed. Using THPs and accessing data before UFFD registeration + MADV_NOHUGEPAGE can lead to unexpected behaviour and is wrong? > > The important point here is that you discard (MADV_DONTNEED) any pages you want to fault+place later lazily. That must happen after MADV_NOHUGEPAGE but before registering+running userfaultfd. > > Then you don't care if page faults / kuhgepaged gave you THPs before that. In fact, you want to happily use THPs wherever possible and not > disable them unconditionally right from the start. > >> - even in the current kernel code in other places like khugepaged, its only checked if uffd is enabled currently. It is not tracked if it was ever enabled on any VMA. > > Right, see above, this all works if you MADV_DONTNEED the pages you want to get faults to after disabling THPs. > > > I just added a bunch of quick printfs to QEMU and ran a precopy+postcopy live migration. Looks like my assumption was right: > > On the destination: > > Writing received pages during precopy # ram_load_precopy() > Writing received pages during precopy > Writing received pages during precopy > Writing received pages during precopy > Writing received pages during precopy > Writing received pages during precopy > Writing received pages during precopy > Writing received pages during precopy > Writing received pages during precopy > Writing received pages during precopy > Writing received pages during precopy > Writing received pages during precopy > Writing received pages during precopy > Writing received pages during precopy > Writing received pages during precopy > Writing received pages during precopy > Writing received pages during precopy > Writing received pages during precopy > Disabling THP: MADV_NOHUGEPAGE # postcopy_ram_prepare_discard() > Discarding pages # loadvm_postcopy_ram_handle_discard() > Discarding pages > Discarding pages > Discarding pages > Discarding pages > Discarding pages > Discarding pages > Registering UFFD # postcopy_ram_incoming_setup() > Thanks for this, yes it makes sense after you mentioned postcopy_ram_incoming_setup. postcopy_ram_incoming_setup happens in the Listen phase, which is after the discard phase, so I was able to follow in code in qemu the same sequence of events that the above prints show. > > Let me know if you need more information. > >> Thanks for pointing to mm_forbids_zeropage. Incorporating that into the code, and if I am (hopefully :)) right about qemu and kernel above, then I believe the right code should be: > > I'm afraid you are not right about the qemu code :) > Yes, and also didn't consider MADV_DONTNEED! Thanks for explaining both of these things clearly. Its clear that pte_clear won't work in this case. We don't need to clear_pte, just use zero_page for all cases. The original series from Alex did tlb flush, but looking further at the code, thats not needed. try_to_migrate() flushes tlb and installs migration entries which are not ‘present’ so will never be tlb cached. remove_migration_ptes() restores page pointers so tlb flushing is not needed. When using zeropage, we don't need make a distinction if uffd is used or not. i.e. we can just do below: if (contains_data || mm_forbids_zeropage(pvmw->vma->vm_mm)) return false; newpte = pte_mkspecial(pfn_pte(page_to_pfn(ZERO_PAGE(pvmw->address)), pvmw->vma->vm_page_prot)); set_pte_at(pvmw->vma->vm_mm, pvmw->address, pvmw->pte, newpte);
