On Fri, Dec 13, 2019 at 3:47 AM Theodore Y. Ts'o <tytso@xxxxxxx> wrote: > > I'd like to have a discussion at LSF/MM about making it easier and > more accessible for file system developers to run benchmarks as part > of their development processes. > > My interest in this was sparked a few weeks ago, when there was a > click-bait article published on Phoronix, "The Disappointing Direction > Of Linux Performance From 4.16 To 5.4 Kernels"[1], wherein the author > published results which seem to indicate a radical decrease in > performance in a pre-5.4 kernel, which showed the 5.4(-ish) kernel > performance four times worse on a SQLite test. > > [1] https://www.phoronix.com/scan.php?page=article&item=linux-416-54&num=1 > > I tried to reproduce this, and trying to replicate the exact > benchmark, I decided to try using the Phoronix Test Suite (PTS). > Somewhat to my surprise, it was well documented[2], straightforward to > set up, and a lot of care was put into being able to get repeatable > results from running a large set of benchmarks. And so I added > support[3] for running to my gce-xfstests test automation framework. > Very nice :) You should post an [ANNOUNCE] every now and then. I rarely check upstream of xfstests-bld, because it just-works ;-) > [2] https://www.phoronix-test-suite.com/documentation/phoronix-test-suite.html > [3] https://github.com/tytso/xfstests-bld/commit/b8236c94caf0686b1cfacb1348b5a46fa1f52f48 > > Fortunately, using a controlled set kernel configs it I could find no > evidence of a massive performance regression a few days before 5.4 was > released by Linus. These results were reproduced by Jan Kara using mmtests. > > Josef Bacik added a fio benchmark to xfstests in late 2017[4], and > this was discussed at the 2018 LSF/MM. Unfortunately, there doesn't > seem to have been any additional work to add benchmarking > functionality to xfstests. > > [4] https://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git/commit/?id=e0d95552fdb2948c63b29af4a8169a2027f84a1d > > In addition to using xfstests, I have started using PTS to as a way to > sanity check patch submissions to ext4. I've also started I suppose you have access to a dedicated metal in the cloud for running your performance regression tests? Or at least a dedicated metal per execution. I have not looked into GCE, so don't know how easy it is and how expensive to use GCE this way. Is there any chance of Google donating this sort of resource for a performance regression test bot? > investigating using mmtests as well; mmtests isn't quite as polished > and well documented, but has better support for running running > monitoring scripts (e.g., iostat, perf, systemtap, etc.) in parallel > with running benchmarks as workloads. > > I'd like to share what I've learned, and also hopefully learn what > other file system developers have been using to automate measuring > file system performance as a part of their development workflow, > especially if it has been packaged up so other people can more easily > replicate their findings. > Trying to say this carefully, hopefully without starting a mud tossing war - It is sometimes useful to compare performance benchmark on different filesystems on the same benchmark/hardware. Not in order to prove that this filesystem is "better" than the other (we are all for diversity), but because it can sometimes point our attention to core issues. This simple question [1] I posted about a huge difference in fio randomrw benchmark on xfs vs. ext4 has led to patches being posted to address issues in both xfs and ext4 [2][3][4] and to discover bugs in other filesystems as well. Thanks, Amir. [1] https://lore.kernel.org/linux-xfs/CAOQ4uxi0pGczXBX7GRAFs88Uw0n1ERJZno3JSeZR71S1dXg+2w@xxxxxxxxxxxxxx/ [2] https://lore.kernel.org/linux-xfs/20190404165737.30889-1-amir73il@xxxxxxxxx/ [3] https://lore.kernel.org/linux-xfs/20190829131034.10563-1-jack@xxxxxxx/ [4] https://lore.kernel.org/linux-ext4/20190603132155.20600-1-jack@xxxxxxx/
