On Thu, Aug 29, 2024 at 8:44 PM Benjamin Coddington <bcodding@xxxxxxxxxx> wrote:
>
> On 29 Aug 2024, at 5:13, Yafang Shao wrote:
>
> > In our production environment, we noticed that some files are missing when
> > running the ls command in an NFS directory. However, we can still
> > successfully cd into the missing directories. This issue can be illustrated
> > as follows:
> >
> > $ cd nfs
> > $ ls
> > a b c e f <<<< 'd' is missing
> > $ cd d <<<< success
> >
> > I verified the issue with the latest upstream kernel, and it still
> > persists. Further analysis reveals that files go missing when the dtsize is
> > expanded. The default dtsize was reduced from 1MB to 4KB in commit
> > 580f236737d1 ("NFS: Adjust the amount of readahead performed by NFS readdir").
> > After restoring the default size to 1MB, the issue disappears. I also tried
> > setting the default size to 8KB, and the issue similarly disappears.
> >
> > Upon further analysis, it appears that there is a bad entry being decoded
> > in nfs_readdir_entry_decode(). When a bad entry is encountered, the
> > decoding process breaks without handling the error. We should revert the
> > bad entry in such cases. After implementing this change, the issue is
> > resolved.
>
> It seems like you're trying to handle a server bug of some sort. Have you
> been able to look at a wire capture to determine why there's a bad entry?
I've used tcpdump to analyze the packets but didn't find anything
suspicious. Do you have any suggestions?
Interestingly, when we increase the dtsize, the issue goes away. This
suggests that the problem might not be with the server itself, but
rather with the NFS readdir operation. The change in dtsize is as
follows,
diff --git a/fs/nfs/dir.c b/fs/nfs/dir.c
index 56a8aee..39847e1 100644
--- a/fs/nfs/dir.c
+++ b/fs/nfs/dir.c
@@ -81,7 +81,7 @@
ctx = kzalloc(sizeof(*ctx), GFP_KERNEL_ACCOUNT);
if (ctx != NULL) {
ctx->attr_gencount = nfsi->attr_gencount;
- ctx->dtsize = NFS_INIT_DTSIZE;
+ ctx->dtsize = 2 * NFS_INIT_DTSIZE; // 8K
spin_lock(&dir->i_lock);
if (list_empty(&nfsi->open_files) &&
(nfsi->cache_validity & NFS_INO_DATA_INVAL_DEFER))
--
Regards
Yafang
