On Tue, 2021-05-04 at 22:32 +0100, Daire Byrne wrote: > Hi, > > For what it's worth, I mentioned this on the associated redhat > bugzilla but I'll replicate it here - I *think* this issue (bulk > reads/writes starving getattrs etc) is one of the issues I was trying > to describe in my re-export thread: > > https://marc.info/?l=linux-nfs&m=160077787901987&w=4 > > Long story short, when we have already read lots of data into a > client's pagecache (or fscache/cachefiles), you can't reuse it again > later until you do some metadata lookups to re-validate. But if we > are also continually filling the client cache with new data (lots of > reads) as fast as possible, we starve the other processes (in my case > - knfsd re-export threads) from processing the re-validate > lookups/getattrs in a timely manner. > > We have lots of previously cached data but we can't use it for a long > time because we can't get the getattrs out and replied to quickly. > > When I was testing the client behaviour, it didn't seem like nconnect > or NFSv3/NFSv4.2 made much difference to the behaviour - metadata > lookups from another client process to the same mountpoint slowed to > a crawl when a process had reads dominating the network pipe. > > I also found that maxing out the client's network bandwidth really > showed this effect best. Either by saturating a client's physical > network link or, in the case of reads, using an ingress qdisc + htb > on the client to simulate a saturated low speed network. > > In all cases where the client's network is (read) saturated > (physically or using a qdisc), the metadata performance from another > process becomes really poor. If I mount a completely different server > on the same client, the metadata performance to that new second > server is much better despite the ongoing network saturation caused > by the continuing reads from the first server. > > I don't know if that helps much, but it was my observation when I > last looked at this. > > I'd really love to see any kind of improvement to this behaviour as > it's a real shame we can't serve cached data quickly when all the > cache re-validations (getattrs) are stuck behind bulk IO that just > seems to plow through everything else. If you use statx() instead of the regular stat call, and you specifically don't request the ctime and mtime, then the current kernel should skip the writeback. Otherwise, you're going to have to wait for the NFSv4.2 protocol changes that we're trying to push through the IETF to allow the client to be authoritative for the ctime/mtime when it holds a write delegation. > > -- Trond Myklebust Linux NFS client maintainer, Hammerspace trond.myklebust@xxxxxxxxxxxxxxx
