Am 2021-08-19 um 2:00 p.m. schrieb Sierra Guiza, Alejandro (Alex): > > On 8/18/2021 2:28 PM, Ralph Campbell wrote: >> On 8/17/21 5:35 PM, Felix Kuehling wrote: >>> Am 2021-08-17 um 8:01 p.m. schrieb Ralph Campbell: >>>> On 8/12/21 11:31 PM, Alex Sierra wrote: >>>>> From: Ralph Campbell <rcampbell@xxxxxxxxxx> >>>>> >>>>> ZONE_DEVICE struct pages have an extra reference count that >>>>> complicates the >>>>> code for put_page() and several places in the kernel that need to >>>>> check the >>>>> reference count to see that a page is not being used (gup, >>>>> compaction, >>>>> migration, etc.). Clean up the code so the reference count doesn't >>>>> need to >>>>> be treated specially for ZONE_DEVICE. >>>>> >>>>> v2: >>>>> AS: merged this patch in linux 5.11 version >>>>> >>>>> v5: >>>>> AS: add condition at try_grab_page to check for the zone device type, >>>>> while >>>>> page ref counter is checked less/equal to zero. In case of device >>>>> zone, pages >>>>> ref counter are initialized to zero. >>>>> >>>>> Signed-off-by: Ralph Campbell <rcampbell@xxxxxxxxxx> >>>>> Signed-off-by: Alex Sierra <alex.sierra@xxxxxxx> >>>>> --- >>>>> arch/powerpc/kvm/book3s_hv_uvmem.c | 2 +- >>>>> drivers/gpu/drm/nouveau/nouveau_dmem.c | 2 +- >>>>> fs/dax.c | 4 +- >>>>> include/linux/dax.h | 2 +- >>>>> include/linux/memremap.h | 7 +-- >>>>> include/linux/mm.h | 13 +---- >>>>> lib/test_hmm.c | 2 +- >>>>> mm/internal.h | 8 +++ >>>>> mm/memremap.c | 68 >>>>> +++++++------------------- >>>>> mm/migrate.c | 5 -- >>>>> mm/page_alloc.c | 3 ++ >>>>> mm/swap.c | 45 ++--------------- >>>>> 12 files changed, 46 insertions(+), 115 deletions(-) >>>>> >>>> I haven't seen a response to the issues I raised back at v3 of this >>>> series. >>>> https://lore.kernel.org/linux-mm/4f6dd918-d79b-1aa7-3a4c-caa67ddc29bc@xxxxxxxxxx/ >>>> >>>> >>>> >>>> Did I miss something? >>> I think part of the response was that we did more testing. Alex added >>> support for DEVICE_GENERIC pages to test_hmm and he ran DAX tests >>> recommended by Theodore Tso. In that testing he ran into a WARN_ON_ONCE >>> about a zero page refcount in try_get_page. The fix is in the latest >>> version of patch 2. But it's already obsolete because John Hubbard is >>> about to remove that function altogether. >>> >>> I think the issues you raised were more uncertainty than known bugs. It >>> seems the fact that you can have DAX pages with 0 refcount is a feature >>> more than a bug. >>> >>> Regards, >>> Felix >> >> Did you test on a system without CONFIG_ARCH_HAS_PTE_SPECIAL defined? >> In that case, mmap() of a DAX device will call insert_page() which calls >> get_page() which would trigger VM_BUG_ON_PAGE(). >> >> I can believe it is OK for PTE_SPECIAL page table entries to have no >> struct page or that MEMORY_DEVICE_GENERIC struct pages be mapped with >> a zero reference count using insert_pfn(). > Hi Ralph, > We have tried the DAX tests with and without > CONFIG_ARCH_HAS_PTE_SPECIAL defined. > Apparently none of the tests touches that condition for a DAX device. > Of course, > that doesn't mean it could happen. > > Regards, > Alex S. > >> >> >> I find it hard to believe that other MM developers don't see an issue >> with a struct page with refcount == 0 and mapcount == 1. >> >> I don't see where init_page_count() is being called for the >> MEMORY_DEVICE_GENERIC or MEMORY_DEVICE_PRIVATE struct pages the AMD >> driver allocates and passes to migrate_vma_setup(). >> Looks like svm_migrate_get_vram_page() needs to call init_page_count() >> instead of get_page(). (I'm looking at branch >> origin/alexsierrag/device_generic >> https://github.com/RadeonOpenCompute/ROCK-Kernel-Driver.git > Yes, you're right. My bad. Thanks for catching this up. I didn't > realize I was missing > to define CONFIG_DEBUG_VM on my build. Therefore this BUG was never > caught. > It worked after I replaced get_pages by init_page_count at > svm_migrate_get_vram_page. However, I don't think this is the best way > to fix it. > Ideally, get_pages call should work for device pages with ref count > equal to 0 > too. Otherwise, we could overwrite refcounter if someone else is > grabbing the page > concurrently. I think using init_page_count in svm_migrate_get_vram_page is the right answer. This is where the page first gets allocated and initialized (data migrated into it). I think nobody should have or try to take a reference to the page before that. We should probably also add a VM_BUG_ON_PAGE(page_ref_count(page) != 0) before calling init_page_count to make sure of that. > I was thinking to add a special condition in get_pages for dev pages. > This could > also fix the insert_page -> get_page call from a DAX device. [+Theodore] I got lost trying to understand how DAX counts page references and how the PTE_SPECIAL option affects that. Theodore, can you help with this? Is there an easy way to test without CONFIG_ARCH_HAS_PTE_SPECIAL on x86, or do we need to test on a CPU architecture that doesn't support this feature? Thanks, Felix > > Regards, > Alex S. >> >> >> Also, what about the other places where is_device_private_page() is >> called? >> Don't they need to be updated to call is_device_page() instead? >> One of my goals for this patch was to remove special casing reference >> counts >> for ZONE_DEVICE pages in rmap.c, etc. > Correct, is_device_private_page is still used in rmap, memcontrol and > migrate.c files > Looks like rmap and memcontrol should be replaced by is_device_page > function. However, > I still need test to validate this. For migrate.c is used in > remove_migration_pte and > migrate_vma_insert_page, however these are specific conditions for > private device type > Thanks for raise these questions, I think we're getting close. > > Regards, > Alex S. >> >> I still think this patch needs an ACK from a FS/DAX maintainer. >>
