Patch "nfs: avoid i_lock contention in nfs_clear_invalid_mapping" has been added to the 6.1-stable tree

[Sasha Levin <sashal@xxxxxxxxxx>]

This is a note to let you know that I've just added the patch titled

    nfs: avoid i_lock contention in nfs_clear_invalid_mapping

to the 6.1-stable tree which can be found at:
    http://www.kernel.org/git/?p=linux/kernel/git/stable/stable-queue.git;a=summary

The filename of the patch is:
     nfs-avoid-i_lock-contention-in-nfs_clear_invalid_map.patch
and it can be found in the queue-6.1 subdirectory.

If you, or anyone else, feels it should not be added to the stable tree,
please let <stable@xxxxxxxxxxxxxxx> know about it.



commit 60b43e3f82992dc200765a25cbf09c49004cf6d6
Author: Mike Snitzer <snitzer@xxxxxxxxxx>
Date:   Fri Oct 18 17:15:41 2024 -0400

    nfs: avoid i_lock contention in nfs_clear_invalid_mapping
    
    [ Upstream commit 867da60d463bb2a3e28c9235c487e56e96cffa00 ]
    
    Multi-threaded buffered reads to the same file exposed significant
    inode spinlock contention in nfs_clear_invalid_mapping().
    
    Eliminate this spinlock contention by checking flags without locking,
    instead using smp_rmb and smp_load_acquire accordingly, but then take
    spinlock and double-check these inode flags.
    
    Also refactor nfs_set_cache_invalid() slightly to use
    smp_store_release() to pair with nfs_clear_invalid_mapping()'s
    smp_load_acquire().
    
    While this fix is beneficial for all multi-threaded buffered reads
    issued by an NFS client, this issue was identified in the context of
    surprisingly low LOCALIO performance with 4K multi-threaded buffered
    read IO.  This fix dramatically speeds up LOCALIO performance:
    
    before: read: IOPS=1583k, BW=6182MiB/s (6482MB/s)(121GiB/20002msec)
    after:  read: IOPS=3046k, BW=11.6GiB/s (12.5GB/s)(232GiB/20001msec)
    
    Fixes: 17dfeb911339 ("NFS: Fix races in nfs_revalidate_mapping")
    Signed-off-by: Mike Snitzer <snitzer@xxxxxxxxxx>
    Reviewed-by: Jeff Layton <jlayton@xxxxxxxxxx>
    Signed-off-by: Anna Schumaker <anna.schumaker@xxxxxxxxxx>
    Signed-off-by: Sasha Levin <sashal@xxxxxxxxxx>

diff --git a/fs/nfs/inode.c b/fs/nfs/inode.c
index 82e846611d5e1..719448d81aed2 100644
--- a/fs/nfs/inode.c
+++ b/fs/nfs/inode.c
@@ -206,12 +206,15 @@ void nfs_set_cache_invalid(struct inode *inode, unsigned long flags)
 		nfs_fscache_invalidate(inode, 0);
 	flags &= ~NFS_INO_REVAL_FORCED;
 
-	nfsi->cache_validity |= flags;
+	flags |= nfsi->cache_validity;
+	if (inode->i_mapping->nrpages == 0)
+		flags &= ~NFS_INO_INVALID_DATA;
 
-	if (inode->i_mapping->nrpages == 0) {
-		nfsi->cache_validity &= ~NFS_INO_INVALID_DATA;
-		nfs_ooo_clear(nfsi);
-	} else if (nfsi->cache_validity & NFS_INO_INVALID_DATA) {
+	/* pairs with nfs_clear_invalid_mapping()'s smp_load_acquire() */
+	smp_store_release(&nfsi->cache_validity, flags);
+
+	if (inode->i_mapping->nrpages == 0 ||
+	    nfsi->cache_validity & NFS_INO_INVALID_DATA) {
 		nfs_ooo_clear(nfsi);
 	}
 	trace_nfs_set_cache_invalid(inode, 0);
@@ -1331,6 +1334,13 @@ int nfs_clear_invalid_mapping(struct address_space *mapping)
 					 TASK_KILLABLE|TASK_FREEZABLE_UNSAFE);
 		if (ret)
 			goto out;
+		smp_rmb(); /* pairs with smp_wmb() below */
+		if (test_bit(NFS_INO_INVALIDATING, bitlock))
+			continue;
+		/* pairs with nfs_set_cache_invalid()'s smp_store_release() */
+		if (!(smp_load_acquire(&nfsi->cache_validity) & NFS_INO_INVALID_DATA))
+			goto out;
+		/* Slow-path that double-checks with spinlock held */
 		spin_lock(&inode->i_lock);
 		if (test_bit(NFS_INO_INVALIDATING, bitlock)) {
 			spin_unlock(&inode->i_lock);