On Fri, Aug 27, 2021 at 07:34:54PM +0100, Joao Martins wrote:
> On 8/27/21 5:25 PM, Jason Gunthorpe wrote:
> > On Fri, Aug 27, 2021 at 03:58:13PM +0100, Joao Martins wrote:
> >
> >> #if defined(CONFIG_ARCH_HAS_PTE_DEVMAP) && defined(CONFIG_TRANSPARENT_HUGEPAGE)
> >> static int __gup_device_huge(unsigned long pfn, unsigned long addr,
> >> unsigned long end, unsigned int flags,
> >> struct page **pages, int *nr)
> >> {
> >> - int nr_start = *nr;
> >> + int refs, nr_start = *nr;
> >> struct dev_pagemap *pgmap = NULL;
> >> int ret = 1;
> >>
> >> do {
> >> - struct page *page = pfn_to_page(pfn);
> >> + struct page *head, *page = pfn_to_page(pfn);
> >> + unsigned long next = addr + PAGE_SIZE;
> >>
> >> pgmap = get_dev_pagemap(pfn, pgmap);
> >> if (unlikely(!pgmap)) {
> >> @@ -2252,16 +2265,25 @@ static int __gup_device_huge(unsigned long pfn, unsigned long addr,
> >> ret = 0;
> >> break;
> >> }
> >> - SetPageReferenced(page);
> >> - pages[*nr] = page;
> >> - if (unlikely(!try_grab_page(page, flags))) {
> >> - undo_dev_pagemap(nr, nr_start, flags, pages);
> >> +
> >> + head = compound_head(page);
> >> + /* @end is assumed to be limited at most one compound page */
> >> + if (PageHead(head))
> >> + next = end;
> >> + refs = record_subpages(page, addr, next, pages + *nr);
> >> +
> >> + SetPageReferenced(head);
> >> + if (unlikely(!try_grab_compound_head(head, refs, flags))) {
> >> + if (PageHead(head))
> >> + ClearPageReferenced(head);
> >> + else
> >> + undo_dev_pagemap(nr, nr_start, flags, pages);
> >> ret = 0;
> >> break;
> >
> > Why is this special cased for devmap?
> >
> > Shouldn't everything processing pud/pmds/etc use the same basic loop
> > that is similar in idea to the 'for_each_compound_head' scheme in
> > unpin_user_pages_dirty_lock()?
> >
> > Doesn't that work for all the special page type cases here?
>
> We are iterating over PFNs to create an array of base pages (regardless of page table
> type), rather than iterating over an array of pages to work on.
That is part of it, yes, but the slow bit here is to minimally find
the head pages and do the atomics on them, much like the
unpin_user_pages_dirty_lock()
I would think this should be designed similar to how things work on
the unpin side.
Sweep the page tables to find a proper start/end - eg even if a
compound is spread across multiple pte/pmd/pud/etc we should find a
linear range of starting PFN (ie starting page*) and npages across as
much of the page tables as we can manage. This is the same as where
things end up in the unpin case where all the contiguous PFNs are
grouped togeher into a range.
Then 'assign' that range to the output array which requires walking
over each compount_head in the range and pinning it, then writing out
the tail pages to the output struct page array.
And this approach should apply universally no matter what is under the
pte's - ie huge pages, THPs and devmaps should all be treated the same
way. Currently each case is different, like above which is unique to
device_huge.
The more we can process groups of pages in bulks the faster the whole
thing will be.
Jason
Given that all these gup
> functions already give you the boundary (end of pmd or end of pud, etc) then we just need
> to grab the ref to pgmap and head page and save the tails. But sadly we need to handle the
> base page case which is why there's this outer loop exists sadly. If it was just head
> pages we wouldn't need the outer loop (and hence no for_each_compound_head, like the
> hugetlb variant).
>
> But maybe I am being dense and you just mean to replace the outer loop with
> for_each_compound_range(). I am a little stuck on the part that I anyways need to record
> back the tail pages when iterating over the (single) head page. So I feel that it wouldn't
> bring that much improvement, unless I missed your point.
>
> Joao
>
