On Mon, Jul 6, 2020 at 9:52 AM Stephen John Smoogen <smooge@xxxxxxxxx> wrote:
>
> On Mon, 6 Jul 2020 at 01:19, Chris Murphy <lists@xxxxxxxxxxxxxxxxx> wrote:
> >
> > On Fri, Jul 3, 2020 at 8:40 PM Eric Sandeen <sandeen@xxxxxxxxxx> wrote:
> > >
> > > On 7/3/20 1:41 PM, Chris Murphy wrote:
> > > > SSDs can fail in weird ways. Some spew garbage as they're failing,
> > > > some go read-only. I've seen both. I don't have stats on how common it
> > > > is for an SSD to go read-only as it fails, but once it happens you
> > > > cannot fsck it. It won't accept writes. If it won't mount, your only
> > > > chance to recover data is some kind of offline scrape tool. And Btrfs
> > > > does have a very very good scrape tool, in terms of its success rate -
> > > > UX is scary. But that can and will improve.
> > >
> > > Ok, you and Josef have both recommended the btrfs restore ("scrape")
> > > tool as a next recovery step after fsck fails, and I figured we should
> > > check that out, to see if that alleviates the concerns about
> > > recoverability of user data in the face of corruption.
> > >
> > > I also realized that mkfs of an image isn't representative of an SSD
> > > system typical of Fedora laptops, so I added "-m single" to mkfs,
> > > because this will be the mkfs.btrfs default on SSDs (right?). Based
> > > on Josef's description of fsck's algorithm of throwing away any
> > > block with a bad CRC this seemed worth testing.
> > >
> > > I also turned fuzzing /down/ to hitting 2048 bytes out of the 1G
> > > image, or a bit less than 1% of the filesystem blocks, at random.
> > > This is 1/4 the fuzzing rate from the original test.
> > >
> > > So: -m single, fuzz 2048 bytes of 1G image, run btrfsck --repair,
> > > mount, mount w/ recovery, and then restore ("scrape") if all that
> > > fails, see what we get.
> >
> > What's the probability of this kind of corruption occurring in the
> > real world? If the probability is so low it can't practically be
> > computed, how do we assess the risk? And if we can't assess risk,
> > what's the basis of concern?
> >
>
> Aren't most disk failure tests 'huh it somehow happened at least once
> and I think this explains all these other failures too?' I know that
> with giant clusters you can do more testing but you also have a lot of
> things like
>
> What is the chance that a disk will die over time? 100%
> What is the chance that a disk died from this particular scenario?
> 0.00000<maybe put a digit here> %
> reword the question slightly differently.. What is the chance this
> disk died from that scenario? 100%.
Yes. Also in fuzzing there is the concept of "when to stop fuzzing"
because it's a rabbit hole, you have to come up for air at some point,
and work on other things. But you raise a good and subtle point which
is also that ext4 has a very good fsck built up over decades, they
succeed today from past failures. It's no different with Btrfs.
But also there is a bias. ext4 needs fsck to succeed in the worst
cases in order to mount the file system. Btrfs doesn't need that.
Often it can tolerate a read-only mount without any other mount
option; and optionally can be made more tolerant to errors while still
mounting read-only. This is a significant difference in recovery
strategy. An fsck is something of a risk because it is writing changes
to the file system. It is irreversible. Btrfs takes a different view,
which is to increase the chance of recovery without needing a risky
repair as the first step. Once your important data is out, now try the
repair. Good chance it works, but maybe not as good as ext4's.
--
Chris Murphy
_______________________________________________
devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
