[RFC PATCH 00/40] fstests: concurrent test execution

[Date Prev] [Date Next] [Thread Prev] [Thread Next] [Date Index] [Thread Index]



Hi folks,

This patchset introduces the ability to run fstests concurrently
instead of serially as the current check script does. A git branch
containing this patchset can be pulled from here:

https://git.kernel.org/pub/scm/linux/kernel/git/dgc/xfstests-dev.git check-parallel

The motivation for this is the ever growing runtime of fstests as
more tests are added, along with the extremely low resource usage of
individual tests. This means that a typical machine running fstests
is under-utilised and a single auto group test set execution takes
hours.

Yes, I know that I could scale out testing by running lots of little
VMs at once, and I already do that. The problem is that I don't get
back a complete auto test run result for hours. Want to check that a
one-line fix has not caused any regressions? That's, at minimum, an
overnight wait for the test farm to crunch through a dozen configs.

On my 64p/128GB RAM VM, 'check -g auto -s xfs -x dump' typically
takes around 230-240 minutes to run. With this patchset applied, it
runs the same set of tests in 8-10 minutes. I can run ~25 complete
auto group test sets with check-parallel in the same time it takes
check to run one.

IOWs, I can have the most common config complete a full regression
test run as fast as I can turn around a new kernel with a new change
to test. I have CPU, memory and IO to burn in my test machines, but
what I lack is instant feedback for the change I just made.
check-parallel is fast enough that it gives me pretty much instant
feedback....

Most of this patchset is preparing infrastructure for concurrent
test execution and fixing bugs in tests that I've found whilst
getting concurrent execution working reliably. The infrastructure
changes center around:

- getting rid of killall - there's nothing quite like one test
  killing the processes of 15 other tests at the same time...
- cleaning up background process instantiation and reaping, which is
  a lot more important when an interrupt needs to kill hundreds of
  processes instead of just a couple.
- isolating error reporting (e.g. dmesg filtering) so that one test
  failure doesn't trigger lots of other false failure detections
- sanitising the use of loopback devices
- avoiding the use of fixed device names - multiple tests need to
  use dm-flakey, dm-error, etc devices at the same time, so each
  test needs a unique device name
- marking tests that are unreliable when outside perturbations
  occur. These include tests that expect to find delalloc extents,
  write specific offset patterns in memory then sync them to create
  specific layouts, etc. e.g. If another test runs sync(1), then
  those write patterns no longer produce the expected output.
- taming tests that weren't designed for high CPU count machines
- replacing `udevadm settle` calls because udev is -never- idle when
  there are tens to hundreds of block devices and filesystems being
  rapidly set up and torn down.
- converting sync(1) and sync(2) to syncfs(2) to avoid ihaving
  hundreds of concurrent superblock list traversals lock-stepping
  with multiple mount/unmounts every second.

There are lots of little other things, but those are the main test
and test infrastructure changes. Some of these are big - the
fsstress execution rework touches 105 files, but it now means that
every single fsstress execution in fstests is controlled by 4 helper
functions:

	_run_fsstress()		- run fsstress synchronously
	_run_fsstress_bg	- run fsstress in background
	_wait_for_fsstress	- wait for background fsstress
	_kill_fsstress		- kill+wait for background fsstress

The test infrastructure also automatically handles cleanup of
fsstress processes when the test is interrupted, so tests using
fsstress don't need a custom _cleanup() function just to call
_kill_fsstress(). This is something that should have been done a
long time ago, but now it is critical for being able to manage
multiple independent concurrent fsstress invocations sanely.

There are some tests that just can't be run reliably in a concurrent
environment - if there is outside interference in, say, page cache
flushing then the tests fail. These tests have been added to the
"unreliable_in_parallel" test group with a comment explaining why
they are unreliable. The check-parallel script automatically
excludes this test group.

The only remaining set of tests that are somewhat flakey are the
tests that exercise quotas. Quota tests randomly fail for various
reasons. Sometimes they don't detect EDQUOT conditions. Sometimes
repquota emits weird "device not found" errors. Sometimes grace
periods don't start, sometimes they don't time out, or time out and
then don't trigger EDQUOT. I don't know why these weird things are
happening yet, and until I properly integrate test group selection
into check-parallel I can't really isolate the quota tests to focus
on them alone.

There are several patches that speed up test runtime. There were
several tests that were taking 12-15 minutes to run each, and these
made up >95% of the entire check-parallel runtime. In general, these
tests have been made to run with more concurrency to speed them up.
the result is that the longest individual test runtime has dropped
to around 7 minutes, and the elapsed runtime for check-parallel has
dropped to 8-10 minutes.

Hence there are 39 patches that are doing prep work on tests and
infrastructure to make tests run reliabily in concurrent test
contexts. The last patch is the check-parallel runner script that
runs the tests concurrently.

The check-parallel script is still very rudimentary. I hard coded
the tests it runs (generic+xfs auto tests) and the concurrency so
that I could run explicit sets of tests with './check --exact-order
<list>'. It is pointed at a mounted directory, and it creates all
the infrastructure it needs to run the tests within that directory.

This enabled me to break up the tests across a set of identical
runner process contexts. Each runner does:

- create runner directory
- create test and scratch image files
- create loopback devices for test and scratch devices
- sets up results directories
- execute check in it's own private mount namespace so it can't see
  any of the mounts that other runners are using.
- tears down loopback devices
- reports test failures.

If you run with the same directory over and over again, then it
reuses the same runner infrastructure and test and scratch image
files. The results directories are not overwritten as they are
date-stamped, hence using the same mount point automatically creates
a result archive for later data mining.

A typical target directory (/mnt/xfs) looks like:

/mnt/xfs/runner-0/...
/mnt/xfs/runner-1/...
....
/mnt/xfs/runner-63/...

And each runner directory:

log
results-2024-11-19-11:22:25/...
results-2024-11-19-12:36:28/...
.....
results-2024-11-27-13:32:42/...
scratch/
scratch.img
test/
test.img

The log file is the check output for that runner and should look
familiar:

SECTION       -- xfs
FSTYP         -- xfs (debug)
PLATFORM      -- Linux/x86_64 test1 6.12.0-dgc+ #297 SMP PREEMPT_DYNAMIC Wed Nov 27 08:13:06 AEDT 2024
MKFS_OPTIONS  -- -f -m rmapbt=1 /dev/loop10
MOUNT_OPTIONS -- -o context=system_u:object_r:root_t:s0 /dev/loop10 /mnt/xfs/runner-10/scratch

generic/387        309s
generic/551        49s
generic/289        2s
xfs/274        3s
generic/677        3s
generic/143        5s
generic/304        1s
generic/390        4s
generic/427        4s
xfs/103        1s
generic/252        3s
xfs/045        1s
generic/374        2s
generic/002        1s
generic/534        1s
generic/039        1s
generic/595       [not run] No encryption support for xfs
xfs/122       [not run] Could not compile test program (see end of /mnt/xfs/runner-10/results-2024-11-27-13:32:42/xfs/xfs/122.full)
xfs/556       [not run] xfs_scrub not found
Ran: generic/387 generic/551 generic/289 xfs/274 generic/677 generic/143 generic/304 generic/390 generic/427 xfs/103 generic/252 xfs/045 generic/374 generic/002 generic/534 generic/039 generic/595 xfs/122 xfs/556
Not run: generic/595 xfs/122 xfs/556
Passed all 19 tests

SECTION       -- xfs
=========================
Ran: generic/387 generic/551 generic/289 xfs/274 generic/677 generic/143 generic/304 generic/390 generic/427 xfs/103 generic/252 xfs/045 generic/374 generic/002 generic/534 generic/039 generic/595 xfs/122 xfs/556
Not run: generic/595 xfs/122 xfs/556
Passed all 19 tests

Doing something with all the log files from a run can be done with
"/mnt/xfs/*/log". e.g. grep, vi, etc.

The results directory contains the same check.{full,log,time} and
test results directories as per a normal check invocation, so
there's little difference in checking/analysing results with a
parallel execution run.

Because the tests run in private mount namespaces, it's easy to see
what check-parallel is running at any point in time using 'pstree -N
mnt'. Here's a run that is hung on an unmount not completing:

$ pstree -N mnt
[4026531841]
bash
bash───pstree
[0]
sudo───sudo───check-parallel─┬─check-parallel───nsexec───check───311───fsync-tester
                             ├─check-parallel───nsexec───check───467───open_by_handle
                             ├─check-parallel───nsexec───check───338
                             ├─check-parallel───nsexec───check───421
                             ├─check-parallel───nsexec───check───441
                             ├─check-parallel───nsexec───check───232
                             ├─check-parallel───nsexec───check───477───open_by_handle
                             ├─check-parallel───nsexec───check───420
                             ├─check-parallel───nsexec───check───426───open_by_handle
                             ├─check-parallel───nsexec───check───756───open_by_handle
                             ├─check-parallel───nsexec───check───231
                             ├─check-parallel───nsexec───check───475───475.fsstress───475.fsstress───{475.fsstress}
                             ├─check-parallel───nsexec───check───388───388.fsstress───388.fsstress───{388.fsstress}
                             ├─check-parallel───nsexec───check───259───sync
                             ├─check-parallel───nsexec───check───622───sync
                             ├─check-parallel───nsexec───check───318───sync
                             ├─check-parallel───nsexec───check───753───umount
                             ├─check-parallel───nsexec───check───086
                             ├─check-parallel───nsexec───check───648───648.fsstress───648.fsstress───{648.fsstress}
                             ├─check-parallel───nsexec───check───391
                             ├─check-parallel───nsexec───check───315───sync
                             └─check-parallel───nsexec───check───183───bulkstat_unlink

All the other processes are in sync or dropping caches and stuck
waiting for the superblock s_umount lock that the unmount holds.
Finding where it is stuck:

$ pgrep [u]mount
1081885
$ sudo cat /proc/1081885/stack
[<0>] xfs_ail_push_all_sync+0x9c/0xf0
[<0>] xfs_unmount_flush_inodes+0x41/0x70
[<0>] xfs_unmountfs+0x59/0x190
[<0>] xfs_fs_put_super+0x3b/0x90
[<0>] generic_shutdown_super+0x77/0x160
[<0>] kill_block_super+0x1b/0x40
[<0>] xfs_kill_sb+0x12/0x30
[<0>] deactivate_locked_super+0x38/0x100
[<0>] deactivate_super+0x41/0x50
[<0>] cleanup_mnt+0x9f/0x160
[<0>] __cleanup_mnt+0x12/0x20
[<0>] task_work_run+0x89/0xb0
[<0>] resume_user_mode_work+0x4f/0x60
[<0>] syscall_exit_to_user_mode+0x76/0xb0
[<0>] do_syscall_64+0x74/0x130
[<0>] entry_SYSCALL_64_after_hwframe+0x76/0x7e
$

Yup, known bug - a shutdown vs XFS_ISTALE inode cluster freeing
issue that leaves pinned, stale inodes in the AIL.

The point I'm making here is that running tests concurrently doesn't
change anything material in how you'd go about discovering and
diagnosing failures. The only difference is in what that initial
failure might look like. e.g. Failures that result in dmesg output
will cause a huge number of tests to all fail with "check dmesg for
failure output" reports. Hence there is a bit of sifting to find
which test triggered the dmesg output, but failure detection still
works just fine.

The check-parallel script is really only at proof-of-concept stage.
It is sufficient to run tests in parallel, but that's about it. most
of the work so far has gone into making the generic and XFS tests
run reliably in parallel. I have not run any other tests, and
haven't done full conversions of other test directories or C code in
src/. e.g. the sync -> syncfs was only done for the common dir and
the generic and xfs tests dirs

I do not plan to do these conversions for ext4/btrfs/overlay/etc any
time soon as these conversions are mostly about improving execution
time, not test correctness. Hence they are not a priority for me -
the priority is further developing the concurrent execution
environment. i.e. check-parallel.

To that end, I need to factor all the test selection and exclusion
code out of check, and do the same with the actual test list runner
loop. That way I can reuse all the existing code from within the
check-parallel context rather than having to call check to do all of
that work itself. I would like to get check-parallel to the point
where it is mostly just s/check/check-parallel/ on the command line
to move from serial to concurrent test execution.

I also want to try to integrate the config section stuff into
check-parallel. This is more a case of defining what devices the
config needs to create (i.e. as loop devices) rather than what
devices it should be using. I think I can do this just by defining a
different set of environment variables (e.g. NEED_SCRATCHDEV,
NEED_LOGDEV, etc) and triggering the loop device creation from these
variables.

In a way, the fact that check-parallel bootstraps it's own runtime
environment almost makes it entirely zero-config. As it stands right
now, you should be able to pull this patchset, create your base test
directory (make sure you have at least 100GB of free disk space),
run './check-parallel <test_dir> -x dump' and it Should Just Work.

At this point, the test infrastructure problems are largely solved.
My focus is now on further development of the check-parallel script
and integrating it tightly into the existing check infrastructure
rather than open-coding test lists and configuration information.
This will probably take a bit of time, so I'd like to get the bug
fixes, improvements and infrastructure changes underway so I'm not
left carrying a huge patchset for months....

----------------------------------------------------------------
Dave Chinner (40):
      xfs/448: get rid of assert-on-failure
      fstests: cleanup fsstress process management
      fuzzy: don't use killall
      fstests: per-test dmflakey instances
      fstests: per-test dmerror instances
      fstests: per-test dmhuge instances
      fstests: per-test dmthin instances
      fstests: per-test dmdust instances
      fstests: per-test dmdelay instances
      fstests: fix DM device creation/removal vs udev races
      fstests: use syncfs rather than sync
      fstests: clean up mount and unmount operations
      fstests: clean up loop device instantiation
      fstests: xfs/227 is really slow
      fstests: mark tests that are unreliable when run in parallel
      fstests: use udevadm wait in preference to settle
      xfs/442: rescale load so it's not exponential
      xfs/176: fix broken setup code
      xfs/177: remove unused slab object count location checks
      fstests: remove uses of killall where possible
      generic/127: reduce runtime
      quota: system project quota files need to be shared
      dmesg: reduce noise from other tests
      fstests: stop using /tmp directly
      fstests: scale some tests for high CPU count sanity
      generic/310: cleanup killing background processes
      filter: handle mount errors from CONFIG_BLK_DEV_WRITE_MOUNTED=y
      filters: add a filter that accepts EIO instead of other errors
      generic/085: general cleanup for reliability and debugging
      fstests: don't use directory stacks
      fstests: clean up a couple of dm-flakey tests
      fstests: clean up termination of various tests
      vfstests: some tests require the testdir to be shared
      xfs/629: single extent files should be within tolerance
      xfs/076: fix broken mkfs filtering
      fstests: capture some failures to seqres.full
      fstests: always use fail-at-unmount semantics for XFS
      generic/062: don't leave debug files in $here on failure
      fstests: quota grace periods unreliable under load
      fstests: check-parallel

 check                                         |   12 -
 check-parallel                                |  205 ++++++++++++++++++
 common/btrfs                                  |    4
 common/config                                 |   36 ++-
 common/dmdelay                                |   24 +-
 common/dmdust                                 |   14 -
 common/dmerror                                |   74 +++---
 common/dmflakey                               |   60 ++---
 common/dmhugedisk                             |   21 +
 common/dmlogwrites                            |    4
 common/dmthin                                 |   12 -
 common/encrypt                                |    2
 common/filter                                 |   17 +
 common/fuzzy                                  |   37 +--
 common/log                                    |    2
 common/metadump                               |   32 +-
 common/overlay                                |   10
 common/populate                               |    8
 common/preamble                               |    1
 common/quota                                  |   37 ---
 common/rc                                     |  166 +++++++++++---
 common/repair                                 |    2
 common/report                                 |    2
 common/verity                                 |    2
 common/xfs                                    |    2
 doc/group-names.txt                           |    1
 doc/requirement-checking.txt                  |    6
 ltp/fsstress.c                                |   28 ++
 src/aio-dio-regress/aio-last-ref-held-by-io.c |    5
 src/dmerror                                   |    6
 tests/btrfs/004                               |   11
 tests/btrfs/007                               |    3
 tests/btrfs/012                               |    4
 tests/btrfs/028                               |    6
 tests/btrfs/049                               |    4
 tests/btrfs/057                               |    4
 tests/btrfs/060                               |   14 -
 tests/btrfs/061                               |   13 -
 tests/btrfs/062                               |   13 -
 tests/btrfs/063                               |   13 -
 tests/btrfs/064                               |   13 -
 tests/btrfs/065                               |   14 -
 tests/btrfs/066                               |   14 -
 tests/btrfs/067                               |   14 -
 tests/btrfs/068                               |   14 -
 tests/btrfs/069                               |   13 -
 tests/btrfs/070                               |   13 -
 tests/btrfs/071                               |   13 -
 tests/btrfs/072                               |   14 -
 tests/btrfs/073                               |   13 -
 tests/btrfs/074                               |   13 -
 tests/btrfs/078                               |   12 -
 tests/btrfs/100                               |    4
 tests/btrfs/101                               |    4
 tests/btrfs/136                               |    6
 tests/btrfs/160                               |    3
 tests/btrfs/192                               |   12 -
 tests/btrfs/195                               |    2
 tests/btrfs/212                               |   16 -
 tests/btrfs/232                               |    4
 tests/btrfs/252                               |    5
 tests/btrfs/261                               |    2
 tests/btrfs/284                               |    4
 tests/btrfs/286                               |    2
 tests/btrfs/291                               |    5
 tests/btrfs/320                               |    6
 tests/btrfs/332                               |    4
 tests/ext4/004                                |    4
 tests/ext4/057                                |   14 -
 tests/ext4/058                                |    3
 tests/ext4/307                                |    4
 tests/generic/013                             |   22 -
 tests/generic/015                             |    2
 tests/generic/019                             |   13 -
 tests/generic/029                             |    2
 tests/generic/030                             |    2
 tests/generic/032                             |    2
 tests/generic/034                             |    2
 tests/generic/039                             |    2
 tests/generic/040                             |    2
 tests/generic/041                             |    2
 tests/generic/042                             |    4
 tests/generic/048                             |    2
 tests/generic/049                             |    4
 tests/generic/050                             |    3
 tests/generic/051                             |   24 --
 tests/generic/054                             |    2
 tests/generic/055                             |    6
 tests/generic/057                             |    2
 tests/generic/059                             |    2
 tests/generic/062                             |    2
 tests/generic/065                             |    2
 tests/generic/066                             |    5
 tests/generic/067                             |   17 -
 tests/generic/068                             |    7
 tests/generic/070                             |   12 -
 tests/generic/073                             |    2
 tests/generic/076                             |   10
 tests/generic/076.out                         |    1
 tests/generic/081                             |    8
 tests/generic/083                             |   13 -
 tests/generic/083.out                         |    1
 tests/generic/084                             |   12 -
 tests/generic/085                             |   21 +
 tests/generic/090                             |    4
 tests/generic/092                             |    2
 tests/generic/098                             |    4
 tests/generic/099                             |    8
 tests/generic/101                             |    2
 tests/generic/104                             |    2
 tests/generic/106                             |    2
 tests/generic/107                             |    2
 tests/generic/108                             |    9
 tests/generic/109                             |    5
 tests/generic/117                             |    4
 tests/generic/127                             |   67 +++--
 tests/generic/127.out                         |    6
 tests/generic/135                             |   24 --
 tests/generic/150                             |    2
 tests/generic/151                             |    2
 tests/generic/152                             |    2
 tests/generic/157                             |    3
 tests/generic/158                             |    3
 tests/generic/159                             |    2
 tests/generic/160                             |    2
 tests/generic/162                             |    4
 tests/generic/163                             |    4
 tests/generic/164                             |    4
 tests/generic/165                             |    4
 tests/generic/166                             |    6
 tests/generic/167                             |    4
 tests/generic/168                             |    7
 tests/generic/170                             |    7
 tests/generic/171                             |    6
 tests/generic/172                             |    6
 tests/generic/173                             |    6
 tests/generic/174                             |    6
 tests/generic/204                             |    8
 tests/generic/232                             |    5
 tests/generic/232.out                         |    1
 tests/generic/247                             |    2
 tests/generic/250                             |    2
 tests/generic/251                             |    5
 tests/generic/252                             |    5
 tests/generic/265                             |    2
 tests/generic/266                             |    2
 tests/generic/267                             |    2
 tests/generic/268                             |    2
 tests/generic/269                             |    8
 tests/generic/270                             |   10
 tests/generic/271                             |    2
 tests/generic/272                             |    2
 tests/generic/273                             |    2
 tests/generic/274                             |    6
 tests/generic/275                             |    6
 tests/generic/276                             |    2
 tests/generic/278                             |    2
 tests/generic/279                             |    2
 tests/generic/281                             |    2
 tests/generic/282                             |    2
 tests/generic/283                             |    2
 tests/generic/306                             |    2
 tests/generic/310                             |   24 +-
 tests/generic/315                             |    2
 tests/generic/317                             |    2
 tests/generic/318                             |    2
 tests/generic/321                             |    4
 tests/generic/323                             |    7
 tests/generic/325                             |    2
 tests/generic/328                             |    4
 tests/generic/329                             |    5
 tests/generic/330                             |    2
 tests/generic/331                             |    7
 tests/generic/332                             |    4
 tests/generic/333                             |    6
 tests/generic/334                             |    6
 tests/generic/335                             |    2
 tests/generic/336                             |    9
 tests/generic/341                             |    2
 tests/generic/342                             |    2
 tests/generic/343                             |    2
 tests/generic/347                             |    2
 tests/generic/348                             |    2
 tests/generic/353                             |    2
 tests/generic/361                             |    8
 tests/generic/373                             |    4
 tests/generic/374                             |    4
 tests/generic/376                             |    2
 tests/generic/382                             |    2
 tests/generic/387                             |    2
 tests/generic/388                             |   24 --
 tests/generic/390                             |   11
 tests/generic/391                             |    2
 tests/generic/395                             |    2
 tests/generic/409                             |   11
 tests/generic/410                             |   11
 tests/generic/411                             |   11
 tests/generic/416                             |    2
 tests/generic/422                             |    4
 tests/generic/425                             |    2
 tests/generic/441                             |    2
 tests/generic/459                             |    8
 tests/generic/461                             |   17 -
 tests/generic/464                             |   10
 tests/generic/474                             |    4
 tests/generic/475                             |   17 -
 tests/generic/476                             |   15 -
 tests/generic/479                             |    2
 tests/generic/480                             |    2
 tests/generic/482                             |   12 -
 tests/generic/483                             |    2
 tests/generic/484                             |    3
 tests/generic/489                             |    2
 tests/generic/502                             |    2
 tests/generic/505                             |    2
 tests/generic/506                             |    2
 tests/generic/507                             |    2
 tests/generic/508                             |    2
 tests/generic/510                             |    2
 tests/generic/520                             |    2
 tests/generic/526                             |    2
 tests/generic/527                             |    2
 tests/generic/530                             |    2
 tests/generic/531                             |    8
 tests/generic/535                             |    2
 tests/generic/546                             |    2
 tests/generic/547                             |    5
 tests/generic/556                             |    2
 tests/generic/560                             |    7
 tests/generic/561                             |   25 +-
 tests/generic/563                             |   24 +-
 tests/generic/564                             |   12 -
 tests/generic/579                             |    8
 tests/generic/585                             |    4
 tests/generic/589                             |   11
 tests/generic/590                             |    9
 tests/generic/599                             |    2
 tests/generic/601                             |    7
 tests/generic/603                             |   14 -
 tests/generic/604                             |    2
 tests/generic/610                             |    2
 tests/generic/620                             |    1
 tests/generic/628                             |    4
 tests/generic/629                             |    4
 tests/generic/631                             |    2
 tests/generic/632                             |    2
 tests/generic/640                             |    2
 tests/generic/642                             |   10
 tests/generic/648                             |   25 --
 tests/generic/650                             |   19 -
 tests/generic/670                             |    2
 tests/generic/671                             |    2
 tests/generic/672                             |    2
 tests/generic/673                             |    2
 tests/generic/674                             |    2
 tests/generic/675                             |    2
 tests/generic/677                             |    2
 tests/generic/683                             |    2
 tests/generic/684                             |    2
 tests/generic/685                             |    2
 tests/generic/686                             |    2
 tests/generic/687                             |    2
 tests/generic/688                             |    2
 tests/generic/690                             |    2
 tests/generic/691                             |    6
 tests/generic/694                             |    2
 tests/generic/695                             |    2
 tests/generic/698                             |    4
 tests/generic/699                             |    8
 tests/generic/703                             |    2
 tests/generic/704                             |    2
 tests/generic/707                             |    7
 tests/generic/716                             |    2
 tests/generic/717                             |    2
 tests/generic/718                             |    2
 tests/generic/719                             |    2
 tests/generic/721                             |    2
 tests/generic/722                             |   15 -
 tests/generic/725                             |    2
 tests/generic/726                             |    2
 tests/generic/727                             |    2
 tests/generic/730                             |    2
 tests/generic/731                             |    2
 tests/generic/732                             |    4
 tests/generic/735                             |    2
 tests/generic/738                             |    2
 tests/generic/741                             |    2
 tests/generic/743                             |    4
 tests/generic/744                             |   10
 tests/generic/745                             |    4
 tests/generic/746                             |   18 -
 tests/generic/747                             |    4
 tests/generic/749                             |    4
 tests/generic/750                             |   10
 tests/generic/751                             |    1
 tests/generic/753                             |   17 -
 tests/overlay/019                             |   48 ++--
 tests/overlay/021                             |    8
 tests/overlay/058                             |   12 -
 tests/xfs/006                                 |    7
 tests/xfs/011                                 |   11
 tests/xfs/013                                 |   22 -
 tests/xfs/014                                 |   11
 tests/xfs/016                                 |    4
 tests/xfs/017                                 |   13 -
 tests/xfs/017.out                             |    1
 tests/xfs/032                                 |    2
 tests/xfs/049                                 |   39 ++-
 tests/xfs/050                                 |    5
 tests/xfs/051                                 |   10
 tests/xfs/052                                 |    2
 tests/xfs/057                                 |    9
 tests/xfs/070                                 |    9
 tests/xfs/073                                 |   36 +--
 tests/xfs/074                                 |   23 +-
 tests/xfs/076                                 |    8
 tests/xfs/077                                 |    2
 tests/xfs/078                                 |   20 -
 tests/xfs/079                                 |   17 -
 tests/xfs/104                                 |   14 -
 tests/xfs/110                                 |    2
 tests/xfs/118                                 |    4
 tests/xfs/119                                 |    2
 tests/xfs/128                                 |    2
 tests/xfs/133                                 |    2
 tests/xfs/134                                 |    2
 tests/xfs/137                                 |    2
 tests/xfs/141                                 |   13 -
 tests/xfs/148                                 |   29 +-
 tests/xfs/149                                 |    8
 tests/xfs/154                                 |    1
 tests/xfs/158                                 |    4
 tests/xfs/161                                 |    2
 tests/xfs/167                                 |   17 -
 tests/xfs/168                                 |    8
 tests/xfs/176                                 |   10
 tests/xfs/177                                 |   15 -
 tests/xfs/186                                 |    4
 tests/xfs/195                                 |    2
 tests/xfs/201                                 |    4
 tests/xfs/212                                 |    2
 tests/xfs/216                                 |   23 +-
 tests/xfs/217                                 |   24 +-
 tests/xfs/227                                 |   59 +++--
 tests/xfs/231                                 |    4
 tests/xfs/232                                 |   10
 tests/xfs/234                                 |    2
 tests/xfs/236                                 |    2
 tests/xfs/237                                 |   13 -
 tests/xfs/239                                 |    2
 tests/xfs/240                                 |    7
 tests/xfs/241                                 |    2
 tests/xfs/243                                 |   11
 tests/xfs/246                                 |    2
 tests/xfs/250                                 |   19 +
 tests/xfs/259                                 |   13 -
 tests/xfs/264                                 |    4
 tests/xfs/265                                 |    2
 tests/xfs/270                                 |    2
 tests/xfs/272                                 |    2
 tests/xfs/274                                 |    2
 tests/xfs/276                                 |    2
 tests/xfs/289                                 |    4
 tests/xfs/291                                 |    4
 tests/xfs/297                                 |   12 -
 tests/xfs/300                                 |    8
 tests/xfs/305                                 |    8
 tests/xfs/309                                 |    2
 tests/xfs/312                                 |    2
 tests/xfs/313                                 |    2
 tests/xfs/314                                 |    2
 tests/xfs/315                                 |    4
 tests/xfs/316                                 |    2
 tests/xfs/317                                 |    2
 tests/xfs/318                                 |    4
 tests/xfs/319                                 |    2
 tests/xfs/320                                 |    2
 tests/xfs/321                                 |    2
 tests/xfs/322                                 |    2
 tests/xfs/323                                 |    2
 tests/xfs/324                                 |    2
 tests/xfs/325                                 |    4
 tests/xfs/326                                 |    4
 tests/xfs/327                                 |    2
 tests/xfs/420                                 |    2
 tests/xfs/421                                 |    2
 tests/xfs/423                                 |    2
 tests/xfs/438                                 |    2
 tests/xfs/440                                 |    8
 tests/xfs/442                                 |   15 -
 tests/xfs/448                                 |    6
 tests/xfs/495                                 |    2
 tests/xfs/501                                 |    2
 tests/xfs/502                                 |    2
 tests/xfs/507                                 |    2
 tests/xfs/511                                 |    2
 tests/xfs/513                                 |   48 +---
 tests/xfs/519                                 |    2
 tests/xfs/520                                 |    2
 tests/xfs/521                                 |    8
 tests/xfs/527                                 |    5
 tests/xfs/528                                 |   10
 tests/xfs/530                                 |   11
 tests/xfs/538                                 |    4
 tests/xfs/541                                 |    2
 tests/xfs/544                                 |    2
 tests/xfs/553                                 |    4
 tests/xfs/558                                 |    7
 tests/xfs/601                                 |    2
 tests/xfs/606                                 |   14 -
 tests/xfs/607                                 |    6
 tests/xfs/609                                 |   20 -
 tests/xfs/610                                 |   20 -
 tests/xfs/613                                 |   44 +--
 tests/xfs/613.out                             |    1
 tests/xfs/617                                 |    2
 tests/xfs/629                                 |    6
 tests/xfs/630                                 |    2
 tests/xfs/631                                 |    9
 tests/xfs/790                                 |    2
 tests/xfs/791                                 |    2
 tests/xfs/792                                 |    2
 tests/xfs/802                                 |    7
 423 files changed, 1827 insertions(+), 1629 deletions(-)





[Index of Archives]     [Linux Filesystems Development]     [Linux NFS]     [Linux NILFS]     [Linux USB Devel]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]

  Powered by Linux