Max Kellermann <mk@xxxxxxxxxx> wrote:
> > Do the NFS servers change the files that are being served - or is it
> > just WordPress pushing the changes to the NFS servers for the web
> > servers to then export?
>
> I'm not sure if I understand this question correctly. The NFS server
> (a NetApp, btw.) sees the new file contents correctly; all other web
> servers also see non-corrupt new files. Only the one web server which
> performed the update saw broken files.
I was wondering if there was missing invalidation if the web clients were
modifying the same files in parallel, but it sounds like only one place is
doing the modification, and the problem is the lack of invalidation when a
file is opened for writing.
I have a tentative patch for this - see attached.
David
---
commit 9b00af0190dfee6073aab47ee88e15c31d3c357d
Author: David Howells <dhowells@xxxxxxxxxx>
Date: Wed Apr 20 14:27:17 2022 +0100
fscache: Fix invalidation/lookup race
If an NFS file is opened for writing and closed, fscache_invalidate() will
be asked to invalidate the file - however, if the cookie is in the
LOOKING_UP state (or the CREATING state), then request to invalidate
doesn't get recorded for fscache_cookie_state_machine() to do something
with.
Fix this by making __fscache_invalidate() set a flag if it sees the cookie
is in the LOOKING_UP state to indicate that we need to go to invalidation.
Note that this requires a count on the n_accesses counter for the state
machine, which that will release when it's done.
fscache_cookie_state_machine() then shifts to the INVALIDATING state if it
sees the flag.
Without this, an nfs file can get corrupted if it gets modified locally and
then read locally as the cache contents may not get updated.
Fixes: d24af13e2e23 ("fscache: Implement cookie invalidation")
Reported-by: Max Kellermann <mk@xxxxxxxxxx>
Signed-off-by: David Howells <dhowells@xxxxxxxxxx>
Link: https://lore.kernel.org/r/YlWWbpW5Foynjllo@xxxxxxxxxxxxxxxx-ag [1]
diff --git a/fs/fscache/cookie.c b/fs/fscache/cookie.c
index 9d3cf0111709..3bb6deeb4279 100644
--- a/fs/fscache/cookie.c
+++ b/fs/fscache/cookie.c
@@ -705,7 +705,11 @@ static void fscache_cookie_state_machine(struct fscache_cookie *cookie)
spin_unlock(&cookie->lock);
fscache_init_access_gate(cookie);
fscache_perform_lookup(cookie);
- goto again;
+ spin_lock(&cookie->lock);
+ if (test_and_clear_bit(FSCACHE_COOKIE_DO_INVALIDATE, &cookie->flags))
+ __fscache_set_cookie_state(cookie,
+ FSCACHE_COOKIE_STATE_INVALIDATING);
+ goto again_locked;
case FSCACHE_COOKIE_STATE_INVALIDATING:
spin_unlock(&cookie->lock);
@@ -752,6 +756,9 @@ static void fscache_cookie_state_machine(struct fscache_cookie *cookie)
spin_lock(&cookie->lock);
}
+ if (test_and_clear_bit(FSCACHE_COOKIE_DO_INVALIDATE, &cookie->flags))
+ fscache_end_cookie_access(cookie, fscache_access_invalidate_cookie_end);
+
switch (state) {
case FSCACHE_COOKIE_STATE_RELINQUISHING:
fscache_see_cookie(cookie, fscache_cookie_see_relinquish);
@@ -1048,6 +1055,9 @@ void __fscache_invalidate(struct fscache_cookie *cookie,
return;
case FSCACHE_COOKIE_STATE_LOOKING_UP:
+ __fscache_begin_cookie_access(cookie, fscache_access_invalidate_cookie);
+ set_bit(FSCACHE_COOKIE_DO_INVALIDATE, &cookie->flags);
+ fallthrough;
case FSCACHE_COOKIE_STATE_CREATING:
spin_unlock(&cookie->lock);
_leave(" [look %x]", cookie->inval_counter);
diff --git a/include/linux/fscache.h b/include/linux/fscache.h
index e25539072463..a25804f141d3 100644
--- a/include/linux/fscache.h
+++ b/include/linux/fscache.h
@@ -129,6 +129,7 @@ struct fscache_cookie {
#define FSCACHE_COOKIE_DO_PREP_TO_WRITE 12 /* T if cookie needs write preparation */
#define FSCACHE_COOKIE_HAVE_DATA 13 /* T if this cookie has data stored */
#define FSCACHE_COOKIE_IS_HASHED 14 /* T if this cookie is hashed */
+#define FSCACHE_COOKIE_DO_INVALIDATE 15 /* T if cookie needs invalidation */
enum fscache_cookie_state state;
u8 advice; /* FSCACHE_ADV_* */
