On Wed, Mar 05, 2025 at 07:05:24AM -0700, Christoph Hellwig wrote:
> Now that we have the buffer cache using the folio API, we can extend
> the use of folios to allocate high order folios for multi-page
> buffers rather than an array of single pages that are then vmapped
> into a contiguous range.
>
> This creates a new type of single folio buffers that can have arbitrary
> order in addition to the existing multi-folio buffers made up of many
> single page folios that get vmapped. The single folio is for now
> stashed into the existing b_pages array, but that will go away entirely
> later in the series and remove the temporary page vs folio typing issues
> that only work because the two structures currently can be used largely
> interchangeable.
>
> The code that allocates buffers will optimistically attempt a high
> order folio allocation as a fast path if the buffer size is a power
> of two and thus fits into a folio. If this high order allocation
> fails, then we fall back to the existing multi-folio allocation
> code. This now forms the slow allocation path, and hopefully will be
> largely unused in normal conditions except for buffers with size
> that are not a power of two like larger remote xattrs.
>
> This should improve performance of large buffer operations (e.g.
> large directory block sizes) as we should now mostly avoid the
> expense of vmapping large buffers (and the vmap lock contention that
> can occur) as well as avoid the runtime pressure that frequently
> accessing kernel vmapped pages put on the TLBs.
>
> Based on a patch from Dave Chinner <dchinner@xxxxxxxxxx>, but mutilated
> beyond recognition.
>
> Signed-off-by: Christoph Hellwig <hch@xxxxxx>
> ---
> fs/xfs/xfs_buf.c | 52 ++++++++++++++++++++++++++++++++++++++++++------
> 1 file changed, 46 insertions(+), 6 deletions(-)
>
> diff --git a/fs/xfs/xfs_buf.c b/fs/xfs/xfs_buf.c
> index 073246d4352f..f0666ef57bd2 100644
> --- a/fs/xfs/xfs_buf.c
> +++ b/fs/xfs/xfs_buf.c
> @@ -203,9 +203,9 @@ xfs_buf_free_pages(
>
> for (i = 0; i < bp->b_page_count; i++) {
> if (bp->b_pages[i])
> - __free_page(bp->b_pages[i]);
> + folio_put(page_folio(bp->b_pages[i]));
> }
> - mm_account_reclaimed_pages(bp->b_page_count);
> + mm_account_reclaimed_pages(howmany(BBTOB(bp->b_length), PAGE_SIZE));
>
> if (bp->b_pages != bp->b_page_array)
> kfree(bp->b_pages);
> @@ -277,12 +277,17 @@ xfs_buf_alloc_kmem(
> * For tmpfs-backed buffers used by in-memory btrees this directly maps the
> * tmpfs page cache folios.
> *
> - * For real file system buffers there are two different kinds backing memory:
> + * For real file system buffers there are three different kinds backing memory:
> *
> * The first type backs the buffer by a kmalloc allocation. This is done for
> * less than PAGE_SIZE allocations to avoid wasting memory.
> *
> - * The second type of buffer is the multi-page buffer. These are always made
> + * The second type is a single folio buffer - this may be a high order folio or
> + * just a single page sized folio, but either way they get treated the same way
> + * by the rest of the code - the buffer memory spans a single contiguous memory
> + * region that we don't have to map and unmap to access the data directly.
> + *
> + * The third type of buffer is the multi-page buffer. These are always made
> * up of single pages so that they can be fed to vmap_ram() to return a
> * contiguous memory region we can access the data through, or mark it as
> * XBF_UNMAPPED and access the data directly through individual page_address()
> @@ -295,6 +300,7 @@ xfs_buf_alloc_backing_mem(
> {
> size_t size = BBTOB(bp->b_length);
> gfp_t gfp_mask = GFP_KERNEL | __GFP_NOLOCKDEP | __GFP_NOWARN;
> + struct folio *folio;
> long filled = 0;
>
> if (xfs_buftarg_is_mem(bp->b_target))
> @@ -316,7 +322,41 @@ xfs_buf_alloc_backing_mem(
> if (size < PAGE_SIZE && is_power_of_2(size))
> return xfs_buf_alloc_kmem(bp, size, gfp_mask);
>
> - /* Make sure that we have a page list */
> + /*
> + * Don't bother with the retry loop for single PAGE allocations: vmalloc
> + * won't do any better.
> + */
> + if (size <= PAGE_SIZE)
> + gfp_mask |= __GFP_NOFAIL;
> +
> + /*
> + * Optimistically attempt a single high order folio allocation for
> + * larger than PAGE_SIZE buffers.
> + *
> + * Allocating a high order folio makes the assumption that buffers are a
> + * power-of-2 size, matching the power-of-2 folios sizes available.
> + *
> + * The exception here are user xattr data buffers, which can be arbitrarily
> + * sized up to 64kB plus structure metadata, skip straight to the vmalloc
> + * path for them instead of wasting memory here.
> + */
> + if (size > PAGE_SIZE && !is_power_of_2(size))
> + goto fallback;
> + folio = folio_alloc(gfp_mask, get_order(size));
The only thing extra that I would do here is take a leaf from the
kmalloc() call in xlog_kvmalloc() and turn off direct reclaim for
this allocation because >= 32kB allocations are considered "costly"
and so will enter the compaction code if direct reclaim is enabled.
Given that we fall back to vmalloc, clearing __GFP_DIRECT_RECLAIM
and setting __GFP_NORETRY here means that we don't burn lots of CPU
on memory compaction if there is no high order folios available for
immediate allocation. And on a busy machine, compaction is likely to
fail frequently and so this is all wasted CPU time.
This may be one of the reasons why you don't see any change in real
performance with 64kB directory blocks - we spend more time in
folio allocation because of compaction overhead than we gain back
from avoiding the use of vmapped buffers....
i.e.
if (size > PAGE_SIZE) {
if (!is_power_of_2(size))
goto fallback;
gfp_mask ~= __GFP_DIRECT_RECLAIM;
gfp_mask |= __GFP_NORETRY;
}
folio = folio_alloc(gfp_mask, get_order(size));
-Dave.
--
Dave Chinner
david@xxxxxxxxxxxxx
