On Fri, Mar 11, 2022 at 9:40 AM Benjamin Coddington <bcodding@xxxxxxxxxx> wrote:
>
> .. Something like this does the trick in my testing, but yes will have an
> impact on regular workloads:
>
> 8<------------------------------------------------------------------------
>
> Since commit 1a34c8c9a49e ("NFS: Support larger readdir buffers") has
> updated dtsize and recent improvements to the READDIRPLUS helper heuristic,
> the heuristic may not trigger until many dentries are emitted to userspace,
> which may cause many thousands of GETATTR calls for "ls -l" when the
> directory's pagecache has already been populated. This typically manifests
> as a much slower total runtime for a _second_ invocation of "ls -l" within
> the directory attribute cache timeouts.
>
> Fix this by emitting only 17 entries for any first pass through the NFS
> directory's ->iterate_shared(), which will allow userpace to prime the
> counters for the heuristic.
Here's for what it's worth. An experiment between linux to linux where
the linux server had a "small" directory structure of 57274
directories, 5727390 files in total where each directory had ~100
files each.
With this patch:
date; time tree vol1 > tree.out && date; time tree vol1 > tree.out
Wed Mar 16 12:21:30 EDT 2022
real 11m7.923s
user 0m20.507s
sys 0m39.683s
Wed Mar 16 12:32:38 EDT 2022
real 40m1.751s
user 0m23.477s
sys 0m45.663s
Without the patch:
date; time tree vol1 > tree.out && date; time tree vol1 > tree.out
Wed Mar 16 13:49:12 EDT 2022
real 10m52.909s
user 0m21.342s
sys 0m39.198s
Wed Mar 16 14:00:05 EDT 2022
real 222m56.990s
user 0m30.392s
sys 2m25.202s
>
> Signed-off-by: Benjamin Coddington <bcodding@xxxxxxxxxx>
> ---
> fs/nfs/dir.c | 9 +++++++--
> 1 file changed, 7 insertions(+), 2 deletions(-)
>
> diff --git a/fs/nfs/dir.c b/fs/nfs/dir.c
> index 7e12102b29e7..dc5fc9ba2c49 100644
> --- a/fs/nfs/dir.c
> +++ b/fs/nfs/dir.c
> @@ -1060,6 +1060,8 @@ static int readdir_search_pagecache(struct nfs_readdir_descriptor *desc)
> return res;
> }
>
> +#define NFS_READDIR_CACHE_MISS_THRESHOLD (16UL)
> +
> /*
> * Once we've found the start of the dirent within a page: fill 'er up...
> */
> @@ -1069,6 +1071,7 @@ static void nfs_do_filldir(struct nfs_readdir_descriptor *desc,
> struct file *file = desc->file;
> struct nfs_cache_array *array;
> unsigned int i;
> + bool first_emit = !desc->dir_cookie;
>
> array = kmap(desc->page);
> for (i = desc->cache_entry_index; i < array->size; i++) {
> @@ -1092,6 +1095,10 @@ static void nfs_do_filldir(struct nfs_readdir_descriptor *desc,
> desc->ctx->pos = desc->dir_cookie;
> else
> desc->ctx->pos++;
> + if (first_emit && i > NFS_READDIR_CACHE_MISS_THRESHOLD + 1) {
> + desc->eob = true;
> + break;
> + }
> }
> if (array->page_is_eof)
> desc->eof = !desc->eob;
> @@ -1173,8 +1180,6 @@ static int uncached_readdir(struct nfs_readdir_descriptor *desc)
> return status;
> }
>
> -#define NFS_READDIR_CACHE_MISS_THRESHOLD (16UL)
> -
> static bool nfs_readdir_handle_cache_misses(struct inode *inode,
> struct nfs_readdir_descriptor *desc,
> unsigned int cache_misses,
> --
> 2.31.1
>
