On Fri, Jun 11, 2021 at 03:02:39PM -0400, Brian Foster wrote:
> On Thu, Jun 10, 2021 at 11:14:32AM -0400, Brian Foster wrote:
> > Hi all,
> >
> > I'm seeing what looks like at least one new generic/475 failure on
> > current for-next. (I've seen one related to an attr buffer that seems to
> > be older and harder to reproduce.). The test devices are a couple ~15GB
> > lvm devices formatted with mkfs defaults. I'm still trying to establish
> > reproducibility, but so far a failure seems fairly reliable within ~30
> > iterations.
> >
> > The first [1] looks like log recovery failure processing an EFI. The
> > second variant [2] looks like it passes log recovery, but then fails the
> > mount in the COW extent cleanup stage due to a refcountbt problem. I've
> > also seen one that looks like the same free space corruption error as
> > [1], but triggered via the COW recovery codepath in [2], so these could
> > very well be related. A snippet of the dmesg output for each failed
> > mount is appended below.
> >
> ...
>
> A couple updates..
>
> First (as noted on irc), the generic/475 failure is not new as I was
> able to produce it on vanilla 5.13.0-rc4. I'm not quite sure how far
> back that one goes, but Dave noted he's seen it on occasion for some
> time.
>
> The generic/019 failure I'm seeing does appear to be new as I cannot
> reproduce on 5.13.0-rc4. This failure looks more like silent fs
> corruption. I.e., the test or log recovery doesn't explicitly fail, but
> the post-test xfs_repair check detects corruption. Example xfs_repair
> output is appended below (note that 'xfs_repair -n' actually crashes,
> while destructive repair seems to work). Since this reproduces fairly
> reliably on for-next, I bisected it (while also navigating an unmount
> hang that I don't otherwise have data on) down to facd77e4e38b ("xfs:
> CIL work is serialised, not pipelined"). From a quick glance at that I'm
> not quite sure what the problem is there, just that it doesn't occur
> prior to that particular commit.
I suspect that there's an underlying bug in the overlapping CIL
commit record sequencing. This commit will be the first time we are
actually getting overlapping checkpoints that need ordering via the
commit record writes. Hence I suspect what is being seen here is a
subtle ordering bug that has been in that code since it was first
introduced but never exercised until now..
I haven't had any success in reproducing this yet, I'll keep trying
to see if I can get it to trigger so I can look at it in more
detail...
Cheers,
Dave.
--
Dave Chinner
david@xxxxxxxxxxxxx
