Hi all,
[original patchset cover letter from Dave]
common/populate operations are slow. They are not coded for
performance, and do things in very slow ways. Mainly doing loops to
create/remove files and forcing a task to be created and destroy for
every individual operation. We can only fork a few thousand
processes a second, whilst we can create or remove tens of thousands
of files a second. Hence population speed is limited by fork/exit
overhead, not filesystem speed. I also changed it to run all the
creation steps in parallel, which means they run as fast as the
filesystem can handle them rather than as fast as a single CPU can
handle them.
patch 1 and patch 3 address these issues for common/populate and
xfs/294. I may update a bunch of other tests that use loop { touch
file } to create thousands of files to speed them up as well.
The other patch in this series (patch 2) fixes the problem with
populating an Xfs btree format directory, which currently fails
because the removal step that creates sparse directory data also
causes the dabtree index to get smaller and free blocks, taking the
inode from btree to extent format and hence failing the populate
checks.
More details are in the commit messages for change.
[further changes from Darrick]
This series moves the FMT_BTREE creation bugfix to be first in line,
since it's a bug fix. Next, I convert the directory and xattr creation
loops into separate programs to reduce the execve overhead. This alone
is sufficient for a 10x reduction in runtime without substantially
altering what gets written to disk and comes out in the xfs fsck fuzz
tests.
The last patch in this series starts parallelizing things, but I've left
most of that out since the parallelization patches make it harder to
reliably generate a filesystem image where we can fuzz a two-level inobt
and still mount the fs to run online fsck.
If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.
This is an extraordinary way to destroy everything. Enjoy!
Comments and questions are, as always, welcome.
--D
fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=fix-populate-slowness
---
common/populate | 89 ++++++++++++++++++++++++++++++-------------------------
src/popattr.py | 62 ++++++++++++++++++++++++++++++++++++++
src/popdir.pl | 72 ++++++++++++++++++++++++++++++++++++++++++++
3 files changed, 182 insertions(+), 41 deletions(-)
create mode 100755 src/popattr.py
create mode 100755 src/popdir.pl
