On Mon, Jun 19, 2023 at 09:44:43PM -0700, Darrick J. Wong wrote:
> On Tue, Jun 20, 2023 at 01:24:10PM +1000, Dave Chinner wrote:
> > On Thu, May 25, 2023 at 05:44:48PM -0700, Darrick J. Wong wrote:
> > > From: Darrick J. Wong <djwong@xxxxxxxxxx>
> > >
> > > After an online repair, we need to invalidate buffers representing the
> > > blocks from the old metadata that we're replacing. It's possible that
> > > parts of a tree that were previously cached in memory are no longer
> > > accessible due to media failure or other corruption on interior nodes,
> > > so repair figures out the old blocks from the reverse mapping data and
> > > scans the buffer cache directly.
> > >
> > > Unfortunately, the current buffer cache code triggers asserts if the
> > > rhashtable lookup finds a non-stale buffer of a different length than
> > > the key we searched for. For regular operation this is desirable, but
> > > for this repair procedure, we don't care since we're going to forcibly
> > > stale the buffer anyway. Add an internal lookup flag to avoid the
> > > assert.
> > >
> > > Signed-off-by: Darrick J. Wong <djwong@xxxxxxxxxx>
> > > ---
> > > fs/xfs/scrub/reap.c | 2 +-
> > > fs/xfs/xfs_buf.c | 5 ++++-
> > > fs/xfs/xfs_buf.h | 10 ++++++++++
> > > 3 files changed, 15 insertions(+), 2 deletions(-)
> > >
> > >
> > > diff --git a/fs/xfs/scrub/reap.c b/fs/xfs/scrub/reap.c
> > > index 30e61315feb0..ca75c22481d2 100644
> > > --- a/fs/xfs/scrub/reap.c
> > > +++ b/fs/xfs/scrub/reap.c
> > > @@ -149,7 +149,7 @@ xrep_block_reap_binval(
> > > */
> > > error = xfs_buf_incore(sc->mp->m_ddev_targp,
> > > XFS_FSB_TO_DADDR(sc->mp, fsbno),
> > > - XFS_FSB_TO_BB(sc->mp, 1), 0, &bp);
> > > + XFS_FSB_TO_BB(sc->mp, 1), XBF_BCACHE_SCAN, &bp);
> >
> > Can't say I'm a big fan of XBF_BCACHE_SCAN as a name - it tells me
> > nothing about what the incore lookup is actually doing. The actual
> > lookup action that is being performed is "find any match" rather
> > than "find exact match". XBF_ANY_MATCH would be a better name, IMO.
> >
> > > if (error)
> > > return;
> > >
> > > diff --git a/fs/xfs/xfs_buf.c b/fs/xfs/xfs_buf.c
> > > index 15d1e5a7c2d3..b31e6d09a056 100644
> > > --- a/fs/xfs/xfs_buf.c
> > > +++ b/fs/xfs/xfs_buf.c
> > > @@ -481,7 +481,8 @@ _xfs_buf_obj_cmp(
> > > * reallocating a busy extent. Skip this buffer and
> > > * continue searching for an exact match.
> > > */
> > > - ASSERT(bp->b_flags & XBF_STALE);
> > > + if (!(map->bm_flags & XBM_IGNORE_LENGTH_MISMATCH))
> > > + ASSERT(bp->b_flags & XBF_STALE);
> >
> > And this becomes XBM_ANY_MATCH, too.
>
> Hmmm. I've never come up with a good name for this flag. The caller
> actually has a *specific* length in mind; it simply doesn't want to trip
> the assertions on the cached buffers that have a different length but
> won't be returned by *this* call.
>
> If the buffer cache has bufs for daddr 24 len 8 and daddr len 120, the
> scan calls xfs_buf_get as follows:
>
> daddr 24 len 1 (nothing)
> daddr 24 len 2 (nothing)
> ...
> daddr 24 len 8 (finds the first buffer)
> ...
> daddr 24 len 120 (finds the second buffer)
> ...
> daddr 24 len 132 (nothing)
>
> I don't want the scan to ASSERT 130 times, because that muddles the
> output so badly that it becomes impossible to find relevant debugging
> messages among the crap.
As I mentioned in the my response to the next patch, this is an
O(N^2) brute force search. But how do you get two buffers at the
same address into the cache in the first place?
> > > return 1;
> > > }
> > > return 0;
> > > @@ -682,6 +683,8 @@ xfs_buf_get_map(
> > > int error;
> > > int i;
> > >
> > > + if (flags & XBF_BCACHE_SCAN)
> > > + cmap.bm_flags |= XBM_IGNORE_LENGTH_MISMATCH;
> > > for (i = 0; i < nmaps; i++)
> > > cmap.bm_len += map[i].bm_len;
> > >
> > > diff --git a/fs/xfs/xfs_buf.h b/fs/xfs/xfs_buf.h
> > > index 549c60942208..d6e8c3bab9f6 100644
> > > --- a/fs/xfs/xfs_buf.h
> > > +++ b/fs/xfs/xfs_buf.h
> > > @@ -44,6 +44,11 @@ struct xfs_buf;
> > > #define _XBF_DELWRI_Q (1u << 22)/* buffer on a delwri queue */
> > >
> > > /* flags used only as arguments to access routines */
> > > +/*
> > > + * We're scanning the buffer cache; do not warn about lookup mismatches.
> >
> > The code using the flag isn't doing this - it's trying to invalidate
> > any existing buffer at the location given. It simply wants any
> > buffer at that address to be returned...
> >
> > > + * Only online repair should use this.
> > > + */
> > > +#define XBF_BCACHE_SCAN (1u << 28)
> > > #define XBF_INCORE (1u << 29)/* lookup only, return if found in cache */
> > > #define XBF_TRYLOCK (1u << 30)/* lock requested, but do not wait */
> > > #define XBF_UNMAPPED (1u << 31)/* do not map the buffer */
> > > @@ -67,6 +72,7 @@ typedef unsigned int xfs_buf_flags_t;
> > > { _XBF_KMEM, "KMEM" }, \
> > > { _XBF_DELWRI_Q, "DELWRI_Q" }, \
> > > /* The following interface flags should never be set */ \
> > > + { XBF_BCACHE_SCAN, "BCACHE_SCAN" }, \
> > > { XBF_INCORE, "INCORE" }, \
> > > { XBF_TRYLOCK, "TRYLOCK" }, \
> > > { XBF_UNMAPPED, "UNMAPPED" }
> > > @@ -114,8 +120,12 @@ typedef struct xfs_buftarg {
> > > struct xfs_buf_map {
> > > xfs_daddr_t bm_bn; /* block number for I/O */
> > > int bm_len; /* size of I/O */
> > > + unsigned int bm_flags;
> > > };
> > >
> > > +/* Don't complain about live buffers with the wrong length during lookup. */
> > > +#define XBM_IGNORE_LENGTH_MISMATCH (1U << 0)
> >
> > Which makes me wonder now: can we have two cached buffers at the
> > same address with different lengths during a repair?
>
> Let's say there's a filesystem with dirblksize 8k. A directory block
> gets crosslinked with an xattr block. Running the dir and xattr
> scrubbers each will create a new cached buffer -- an 8k buffer for the
> dir, and a 4k buffer for the xattr.
The second one will trip an assert fail because it found a non-stale
block of a different length.
This is fundamental property of the buffer cache - there is only
supposed to be a single active buffer for a given daddr in the cache
at once. Just because the production code doesn't noisily complain
about it, doesn't mean it is valid behaviour.
> Let's say that the dir block
> overwrote the attr block. Then, the xattr read verifier will fail in
> xfs_trans_buf_read, but that doesn't remove the buffer from the cache.
It won't even get to the read verifier on a debug kernel - it should
be assert failing in the compare function trying to look up the
second buffer as it's asking for a different length.
i.e. I don't see how the above "crosslinked with different lengths
instantiates two independent buffers" works at all on a debug
kernel. It should fail, it is meant to fail, it is meant to tell us
something is breaking the fundamental assumption that there is only
one active cached buffer for a given daddr at a given time....
What am I missing?
> I want the xattr repair to be able to scan the buffer cache and examine
> both buffers without throwing assertions all over the place, because
> that makes it harder to debug repair.
Right, but I don't see anything in the buffer cache changes so far
that allow that or make it in any way safe for it to occur.
Cheers,
Dave.
--
Dave Chinner
david@xxxxxxxxxxxxx
