On Tue, Jan 17, 2023 at 04:42:02PM -0800, Darrick J. Wong wrote:
> Hi all,
>
> [original patchset cover letter from Dave]
>
> common/populate operations are slow. They are not coded for
> performance, and do things in very slow ways. Mainly doing loops to
> create/remove files and forcing a task to be created and destroy for
> every individual operation. We can only fork a few thousand
> processes a second, whilst we can create or remove tens of thousands
> of files a second. Hence population speed is limited by fork/exit
> overhead, not filesystem speed. I also changed it to run all the
> creation steps in parallel, which means they run as fast as the
> filesystem can handle them rather than as fast as a single CPU can
> handle them.
>
> patch 1 and patch 3 address these issues for common/populate and
> xfs/294. I may update a bunch of other tests that use loop { touch
> file } to create thousands of files to speed them up as well.
>
> The other patch in this series (patch 2) fixes the problem with
> populating an Xfs btree format directory, which currently fails
> because the removal step that creates sparse directory data also
> causes the dabtree index to get smaller and free blocks, taking the
> inode from btree to extent format and hence failing the populate
> checks.
>
> More details are in the commit messages for change.
>
> [further changes from Darrick]
>
> This series moves the FMT_BTREE creation bugfix to be first in line,
> since it's a bug fix. Next, I convert the directory and xattr creation
> loops into separate programs to reduce the execve overhead. This alone
> is sufficient for a 10x reduction in runtime without substantially
> altering what gets written to disk and comes out in the xfs fsck fuzz
> tests.
>
> The last patch in this series starts parallelizing things, but I've left
> most of that out since the parallelization patches make it harder to
> reliably generate a filesystem image where we can fuzz a two-level inobt
> and still mount the fs to run online fsck.
>
> If you're going to start using this mess, you probably ought to just
> pull from my git trees, which are linked below.
>
> This is an extraordinary way to destroy everything. Enjoy!
> Comments and questions are, as always, welcome.
>
> --D
>
> fstests git tree:
> https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=fix-populate-slowness
> ---
> common/populate | 89 ++++++++++++++++++++++++++++++-------------------------
> src/popattr.py | 62 ++++++++++++++++++++++++++++++++++++++
> src/popdir.pl | 72 ++++++++++++++++++++++++++++++++++++++++++++
> 3 files changed, 182 insertions(+), 41 deletions(-)
> create mode 100755 src/popattr.py
> create mode 100755 src/popdir.pl
>
The patchset looks good to me:
Reviewed-by: Andrey Albershteyn <aalbersh@xxxxxxxxxx>
--
- Andrey
