Re: [LSF TOPIC] online repair of filesystems: what next?

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 





On 2023/3/1 04:49, Darrick J. Wong wrote:
Hello fsdevel people,

Five years ago[0], we started a conversation about cross-filesystem
userspace tooling for online fsck.  I think enough time has passed for
us to have another one, since a few things have happened since then:

1. ext4 has gained the ability to send corruption reports to a userspace
    monitoring program via fsnotify.  Thanks, Collabora!

Not familiar with the new fsnotify thing, any article to start?

I really believe we should have a generic interface to report errors, currently btrfs reports extra details just through dmesg (like the logical/physical of the corruption, reason, involved inodes etc), which is far from ideal.


2. XFS now tracks successful scrubs and corruptions seen during runtime
    and during scrubs.  Userspace can query this information.

3. Directory parent pointers, which enable online repair of the
    directory tree, is nearing completion.

4. Dave and I are working on merging online repair of space metadata for
    XFS.  Online repair of directory trees is feature complete, but we
    still have one or two unresolved questions in the parent pointer
    code.

5. I've gotten a bit better[1] at writing systemd service descriptions
    for scheduling and performing background online fsck.

Now that fsnotify_sb_error exists as a result of (1), I think we
should figure out how to plumb calls into the readahead and writeback
code so that IO failures can be reported to the fsnotify monitor.  I
suspect there may be a few difficulties here since fsnotify (iirc)
allocates memory and takes locks.

As a result of (2), XFS now retains quite a bit of incore state about
its own health.  The structure that fsnotify gives to userspace is very
generic (superblock, inode, errno, errno count).  How might XFS export
a greater amount of information via this interface?  We can provide
details at finer granularity -- for example, a specific data structure
under an allocation group or an inode, or specific quota records.

The same for btrfs.

Some btrfs specific info like subvolume id is also needed to locate the corrupted inode (ino is not unique among the full fs, but only inside one subvolume).

And something like file paths for the corrupted inode is also very helpful for end users to locate (and normally delete) the offending inode.


With (4) on the way, I can envision wanting a system service that would
watch for these fsnotify events, and transform the error reports into
targeted repair calls in the kernel.

Btrfs has two ways of repair:

- Read time repair
  This happens automatically for both invovled data and metadata, as
  long as the fs is mount RW.

- Scrub time repair
  The repair is also automatic.
  The main difference is, scrub is manually triggered by user space.
  Otherwise it can be considered as a full read of the fs (both metadata
  and data).

But the repair of btrfs only involves using the extra copies, never intended to repair things like directories. (That's still the work of btrfs-check, and the complex cross reference of btrfs is not designed to repair those problems at runtime)

Currently both repair would result a dmesg based report, while scrub has its own interface to report some very basis accounting, like how many sectors are corrupted, and how many are repaired.

A feature full and generic interface to report errors are definitely a good direction to go.

 This of course would be very
filesystem specific, but I would also like to hear from anyone pondering
other usecases for fsnotify filesystem error monitors.

Btrfs also has an internal error counters, but that's accumulated value, sometimes it's not that helpful and can even be confusing.

If we have such interface, we can more or less get rid of the internal error counters, and rely on the user space to do the history recording.


Once (3) lands, XFS gains the ability to translate a block device IO
error to an inode number and file offset, and then the inode number to a
path.  In other words, your file breaks and now we can tell applications
which file it was so they can failover or redownload it or whatever.
Ric Wheeler mentioned this in 2018's session.

Yeah, if user space deamon can automatically (at least by some policy) delete offending files, it can be a great help.

As we have hit several reports that corrupted files (no extra copy to recover from) are preventing btrfs balance, and users have to locate the file from dmesg, and then delete the file and retry balancing.

Thus such interface can greatly improve the user experience.

Thanks,
Qu


The final topic from that 2018 session concerned generic wrappers for
fsscrub.  I haven't pushed hard on that topic because XFS hasn't had
much to show for that.  Now that I'm better versed in systemd services,
I envision three ways to interact with online fsck:

- A CLI program that can be run by anyone.

- Background systemd services that fire up periodically.

- A dbus service that programs can bind to and request a fsck.

I still think there's an opportunity to standardize the naming to make
it easier to use a variety of filesystems.  I propose for the CLI:

/usr/sbin/fsscrub $mnt that calls /usr/sbin/fsscrub.$FSTYP $mnt

For systemd services, I propose "fsscrub@<escaped mountpoint>".  I
suspect we want a separate background service that itself runs
periodically and invokes the fsscrub@$mnt services.  xfsprogs already
has a xfs_scrub_all service that does that.  The services are nifty
because it's really easy to restrict privileges, implement resource
usage controls, and use private name/mountspaces to isolate the process
from the rest of the system.

dbus is a bit trickier, since there's no precedent at all.  I guess
we'd have to define an interface for filesystem "object".  Then we could
write a service that establishes a well-known bus name and maintains
object paths for each mounted filesystem.  Each of those objects would
export the filesystem interface, and that's how programs would call
online fsck as a service.

Ok, that's enough for a single session topic.  Thoughts? :)

--D

[0] https://lwn.net/Articles/754504/
[1] https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-optimize-by-default



[Index of Archives]     [XFS Filesystem Development (older mail)]     [Linux Filesystem Development]     [Linux Audio Users]     [Yosemite Trails]     [Linux Kernel]     [Linux RAID]     [Linux SCSI]


  Powered by Linux