On 12/8/20 7:49 PM, Jason Gunthorpe wrote:
> On Tue, Dec 08, 2020 at 05:28:58PM +0000, Joao Martins wrote:
>> Much like hugetlbfs or THPs, we treat device pagemaps with
>> compound pages like the rest of GUP handling of compound pages.
>>
>> Rather than incrementing the refcount every 4K, we record
>> all sub pages and increment by @refs amount *once*.
>>
>> Performance measured by gup_benchmark improves considerably
>> get_user_pages_fast() and pin_user_pages_fast():
>>
>> $ gup_benchmark -f /dev/dax0.2 -m 16384 -r 10 -S [-u,-a] -n 512 -w
>>
>> (get_user_pages_fast 2M pages) ~75k us -> ~3.6k us
>> (pin_user_pages_fast 2M pages) ~125k us -> ~3.8k us
>>
>> Signed-off-by: Joao Martins <joao.m.martins@xxxxxxxxxx>
>> mm/gup.c | 67 ++++++++++++++++++++++++++++++++++++++++++--------------
>> 1 file changed, 51 insertions(+), 16 deletions(-)
>>
>> diff --git a/mm/gup.c b/mm/gup.c
>> index 98eb8e6d2609..194e6981eb03 100644
>> +++ b/mm/gup.c
>> @@ -2250,22 +2250,68 @@ static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end,
>> }
>> #endif /* CONFIG_ARCH_HAS_PTE_SPECIAL */
>>
>> +
>> +static int record_subpages(struct page *page, unsigned long addr,
>> + unsigned long end, struct page **pages)
>> +{
>> + int nr;
>> +
>> + for (nr = 0; addr != end; addr += PAGE_SIZE)
>> + pages[nr++] = page++;
>> +
>> + return nr;
>> +}
>> +
>> #if defined(CONFIG_ARCH_HAS_PTE_DEVMAP) && defined(CONFIG_TRANSPARENT_HUGEPAGE)
>> -static int __gup_device_huge(unsigned long pfn, unsigned long addr,
>> - unsigned long end, unsigned int flags,
>> - struct page **pages, int *nr)
>> +static int __gup_device_compound_huge(struct dev_pagemap *pgmap,
>> + struct page *head, unsigned long sz,
>> + unsigned long addr, unsigned long end,
>> + unsigned int flags, struct page **pages)
>> +{
>> + struct page *page;
>> + int refs;
>> +
>> + if (!(pgmap->flags & PGMAP_COMPOUND))
>> + return -1;
>> +
>> + page = head + ((addr & (sz-1)) >> PAGE_SHIFT);
>
> All the places that call record_subpages do some kind of maths like
> this, it should be placed inside record_subpages and not opencoded
> everywhere.
>
Makes sense.
>> + refs = record_subpages(page, addr, end, pages);
>> +
>> + SetPageReferenced(page);
>> + head = try_grab_compound_head(head, refs, flags);
>> + if (!head) {
>> + ClearPageReferenced(page);
>> + return 0;
>> + }
>> +
>> + return refs;
>> +}
>
> Why is all of this special? Any time we see a PMD/PGD/etc pointing to
> PFN we can apply this optimization. How come device has its own
> special path to do this??
>
I think the reason is that zone_device struct pages have no relationship to one other. So
you anyways need to change individual pages, as opposed to just the head page.
I made it special to avoid breaking other ZONE_DEVICE users (and gating that with
PGMAP_COMPOUND). But if there's no concerns with that, I can unilaterally enable it.
> Why do we need to check PGMAP_COMPOUND? Why do we need to get pgmap?
> (We already removed that from the hmm version of this, was that wrong?
> Is this different?) Dan?
>
> Also undo_dev_pagemap() is now out of date, we have unpin_user_pages()
> for that and no other error unwind touches ClearPageReferenced..
>
/me nods Yeap I saw that too.
> Basic idea is good though!
>
Cool, thanks!
Joao
