Re: [PATCHSET v2 0/4] fstests: filesystem population fixes

[Date Prev] [Date Next] [Thread Prev] [Thread Next] [Date Index] [Thread Index]



On Tue, Jan 17, 2023 at 04:42:02PM -0800, Darrick J. Wong wrote:
> Hi all,
> 
> [original patchset cover letter from Dave]
> 
> common/populate operations are slow. They are not coded for
> performance, and do things in very slow ways. Mainly doing loops to
> create/remove files and forcing a task to be created and destroy for
> every individual operation. We can only fork a few thousand
> processes a second, whilst we can create or remove tens of thousands
> of files a second. Hence population speed is limited by fork/exit
> overhead, not filesystem speed. I also changed it to run all the
> creation steps in parallel, which means they run as fast as the
> filesystem can handle them rather than as fast as a single CPU can
> handle them.
> 
> patch 1 and patch 3 address these issues for common/populate and
> xfs/294.  I may update a bunch of other tests that use loop { touch
> file } to create thousands of files to speed them up as well.
> 
> The other patch in this series (patch 2) fixes the problem with
> populating an Xfs btree format directory, which currently fails
> because the removal step that creates sparse directory data also
> causes the dabtree index to get smaller and free blocks, taking the
> inode from btree to extent format and hence failing the populate
> checks.
> 
> More details are in the commit messages for change.
> 
> [further changes from Darrick]
> 
> This series moves the FMT_BTREE creation bugfix to be first in line,
> since it's a bug fix.  Next, I convert the directory and xattr creation
> loops into separate programs to reduce the execve overhead.  This alone
> is sufficient for a 10x reduction in runtime without substantially
> altering what gets written to disk and comes out in the xfs fsck fuzz
> tests.
> 
> The last patch in this series starts parallelizing things, but I've left
> most of that out since the parallelization patches make it harder to
> reliably generate a filesystem image where we can fuzz a two-level inobt
> and still mount the fs to run online fsck.
> 
> If you're going to start using this mess, you probably ought to just
> pull from my git trees, which are linked below.
> 
> This is an extraordinary way to destroy everything.  Enjoy!
> Comments and questions are, as always, welcome.
> 
> --D
> 
> fstests git tree:
> https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=fix-populate-slowness
> ---
>  common/populate |   89 ++++++++++++++++++++++++++++++-------------------------
>  src/popattr.py  |   62 ++++++++++++++++++++++++++++++++++++++
>  src/popdir.pl   |   72 ++++++++++++++++++++++++++++++++++++++++++++
>  3 files changed, 182 insertions(+), 41 deletions(-)
>  create mode 100755 src/popattr.py
>  create mode 100755 src/popdir.pl
> 

The patchset looks good to me:

Reviewed-by: Andrey Albershteyn <aalbersh@xxxxxxxxxx>

-- 
- Andrey




[Index of Archives]     [Linux Filesystems Development]     [Linux NFS]     [Linux NILFS]     [Linux USB Devel]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]

  Powered by Linux