[PATCHSET v24.0 0/7] fstests: atomic file updates

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi all,

This series creates a new FIEXCHANGE_RANGE system call to exchange
ranges of bytes between two files atomically.  This new functionality
enables data storage programs to stage and commit file updates such that
reader programs will see either the old contents or the new contents in
their entirety, with no chance of torn writes.  A successful call
completion guarantees that the new contents will be seen even if the
system fails.

The ability to swap extent mappings between files in this manner is
critical to supporting online filesystem repair, which is built upon the
strategy of constructing a clean copy of a damaged structure and
committing the new structure into the metadata file atomically.

User programs will be able to update files atomically by opening an
O_TMPFILE, reflinking the source file to it, making whatever updates
they want to make, and exchange the relevant ranges of the temp file
with the original file.  If the updates are aligned with the file block
size, a new (since v2) flag provides for exchanging only the written
areas.  Callers can arrange for the update to be rejected if the
original file has been changed.

The intent behind this new userspace functionality is to enable atomic
rewrites of arbitrary parts of individual files.  For years, application
programmers wanting to ensure the atomicity of a file update had to
write the changes to a new file in the same directory, fsync the new
file, rename the new file on top of the old filename, and then fsync the
directory.  People get it wrong all the time, and $fs hacks abound.

The reference implementation in XFS creates a new log incompat feature
and log intent items to track high level progress of swapping ranges of
two files and finish interrupted work if the system goes down.  Sample
code can be found in the corresponding changes to xfs_io to exercise the
use case mentioned above.

Note that this function is /not/ the O_DIRECT atomic file writes concept
that has also been floating around for years.  This RFC is constructed
entirely in software, which means that there are no limitations other
than the general filesystem limits.

As a side note, the original motivation behind the kernel functionality
is online repair of file-based metadata.  The atomic file swap is
implemented as an atomic inode fork swap, which means that we can
implement online reconstruction of extended attributes and directories
by building a new one in another inode and atomically swap the contents.

Subsequent patchsets adapt the online filesystem repair code to use
atomic extent swapping.  This enables repair functions to construct a
clean copy of a directory, xattr information, symbolic links, realtime
bitmaps, and realtime summary information in a temporary inode.  If this
completes successfully, the new contents can be swapped atomically into
the inode being repaired.  This is essential to avoid making corruption
problems worse if the system goes down in the middle of running repair.

This patchset also ports the old XFS extent swap ioctl interface to use
the new extent swap code.

For userspace, this series also includes the userspace pieces needed to
test the new functionality, and a sample implementation of atomic file
updates.

Question: Should we really bother with fsdevel bikeshedding?  Most
filesystems cannot support this functionality, so we could keep it
private to XFS for now.

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=atomic-file-updates

xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=atomic-file-updates

fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=atomic-file-updates

xfsdocs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-documentation.git/log/?h=atomic-file-updates
---
 common/rc              |   24 +++++++
 common/reflink         |   33 +++++++++
 common/xfs             |   19 +++++
 configure.ac           |    1 
 doc/group-names.txt    |    2 +
 include/builddefs.in   |    1 
 ltp/Makefile           |    4 +
 ltp/fsstress.c         |  168 ++++++++++++++++++++++++++++++++++++++++++++++++
 ltp/fsx.c              |  160 +++++++++++++++++++++++++++++++++++++++++++++-
 m4/package_libcdev.m4  |   21 ++++++
 src/fiexchange.h       |  101 +++++++++++++++++++++++++++++
 src/global.h           |    6 ++
 tests/generic/1200     |   55 ++++++++++++++++
 tests/generic/1200.out |    3 +
 tests/generic/1201     |   53 +++++++++++++++
 tests/generic/1201.out |    4 +
 tests/generic/1202     |   47 +++++++++++++
 tests/generic/1202.out |    2 +
 tests/generic/1203     |   58 +++++++++++++++++
 tests/generic/1203.out |    2 +
 tests/generic/1204     |  100 +++++++++++++++++++++++++++++
 tests/generic/1204.out |   86 +++++++++++++++++++++++++
 tests/generic/1205     |  116 +++++++++++++++++++++++++++++++++
 tests/generic/1205.out |   90 ++++++++++++++++++++++++++
 tests/generic/1206     |   76 ++++++++++++++++++++++
 tests/generic/1206.out |   32 +++++++++
 tests/generic/1207     |  122 +++++++++++++++++++++++++++++++++++
 tests/generic/1207.out |   48 ++++++++++++++
 tests/generic/1209     |  101 +++++++++++++++++++++++++++++
 tests/generic/1209.out |   33 +++++++++
 tests/generic/1210     |   48 ++++++++++++++
 tests/generic/1210.out |    6 ++
 tests/generic/1211     |  105 ++++++++++++++++++++++++++++++
 tests/generic/1211.out |   40 +++++++++++
 tests/generic/1212     |   58 +++++++++++++++++
 tests/generic/1212.out |    2 +
 tests/generic/1213     |  126 ++++++++++++++++++++++++++++++++++++
 tests/generic/1213.out |   48 ++++++++++++++
 tests/generic/1214     |   63 ++++++++++++++++++
 tests/generic/1214.out |    2 +
 tests/generic/1215     |   70 ++++++++++++++++++++
 tests/generic/1215.out |    2 +
 tests/generic/1216     |   53 +++++++++++++++
 tests/generic/1216.out |    6 ++
 tests/generic/1217     |   56 ++++++++++++++++
 tests/generic/1217.out |    4 +
 tests/generic/1218     |  115 +++++++++++++++++++++++++++++++++
 tests/generic/1218.out |   49 ++++++++++++++
 tests/generic/1219     |   83 ++++++++++++++++++++++++
 tests/generic/1219.out |   17 +++++
 tests/xfs/1202         |   59 +++++++++++++++++
 tests/xfs/1202.out     |   12 +++
 tests/xfs/1208         |   62 ++++++++++++++++++
 tests/xfs/1208.out     |   10 +++
 tests/xfs/1211         |   59 +++++++++++++++++
 tests/xfs/1211.out     |    7 ++
 tests/xfs/1212         |   61 +++++++++++++++++
 tests/xfs/1212.out     |    5 +
 tests/xfs/122.out      |    3 +
 tests/xfs/537          |    2 -
 60 files changed, 2798 insertions(+), 3 deletions(-)
 create mode 100644 src/fiexchange.h
 create mode 100755 tests/generic/1200
 create mode 100644 tests/generic/1200.out
 create mode 100755 tests/generic/1201
 create mode 100644 tests/generic/1201.out
 create mode 100755 tests/generic/1202
 create mode 100644 tests/generic/1202.out
 create mode 100755 tests/generic/1203
 create mode 100644 tests/generic/1203.out
 create mode 100755 tests/generic/1204
 create mode 100644 tests/generic/1204.out
 create mode 100755 tests/generic/1205
 create mode 100644 tests/generic/1205.out
 create mode 100755 tests/generic/1206
 create mode 100644 tests/generic/1206.out
 create mode 100755 tests/generic/1207
 create mode 100644 tests/generic/1207.out
 create mode 100755 tests/generic/1209
 create mode 100644 tests/generic/1209.out
 create mode 100755 tests/generic/1210
 create mode 100644 tests/generic/1210.out
 create mode 100755 tests/generic/1211
 create mode 100644 tests/generic/1211.out
 create mode 100755 tests/generic/1212
 create mode 100644 tests/generic/1212.out
 create mode 100755 tests/generic/1213
 create mode 100644 tests/generic/1213.out
 create mode 100755 tests/generic/1214
 create mode 100644 tests/generic/1214.out
 create mode 100755 tests/generic/1215
 create mode 100644 tests/generic/1215.out
 create mode 100755 tests/generic/1216
 create mode 100644 tests/generic/1216.out
 create mode 100755 tests/generic/1217
 create mode 100644 tests/generic/1217.out
 create mode 100755 tests/generic/1218
 create mode 100644 tests/generic/1218.out
 create mode 100755 tests/generic/1219
 create mode 100755 tests/generic/1219.out
 create mode 100755 tests/xfs/1202
 create mode 100644 tests/xfs/1202.out
 create mode 100755 tests/xfs/1208
 create mode 100644 tests/xfs/1208.out
 create mode 100755 tests/xfs/1211
 create mode 100644 tests/xfs/1211.out
 create mode 100755 tests/xfs/1212
 create mode 100644 tests/xfs/1212.out




[Index of Archives]     [XFS Filesystem Development (older mail)]     [Linux Filesystem Development]     [Linux Audio Users]     [Yosemite Trails]     [Linux Kernel]     [Linux RAID]     [Linux SCSI]


  Powered by Linux