On Sun, Oct 08, 2017 at 10:25:10PM -0400, Josef Bacik wrote: > On Mon, Oct 09, 2017 at 11:51:37AM +1100, Dave Chinner wrote: > > On Fri, Oct 06, 2017 at 05:09:57PM -0400, Josef Bacik wrote: > > > Hello, > > > > > > One thing that comes up a lot every LSF is the fact that we have no general way > > > that we do performance testing. Every fs developer has a set of scripts or > > > things that they run with varying degrees of consistency, but nothing central > > > that we all use. I for one am getting tired of finding regressions when we are > > > deploying new kernels internally, so I wired this thing up to try and address > > > this need. > > > > > > We all hate convoluted setups, the more brain power we have to put in to setting > > > something up the less likely we are to use it, so I took the xfstests approach > > > of making it relatively simple to get running and relatively easy to add new > > > tests. For right now the only thing this framework does is run fio scripts. I > > > chose fio because it already gathers loads of performance data about it's runs. > > > We have everything we need there, latency, bandwidth, cpu time, and all broken > > > down by reads, writes, and trims. I figure most of us are familiar enough with > > > fio and how it works to make it relatively easy to add new tests to the > > > framework. > > > > > > I've posted my code up on github, you can get it here > > > > > > https://github.com/josefbacik/fsperf > > > > > > All (well most) of the results from fio are stored in a local sqlite database. > > > Right now the comparison stuff is very crude, it simply checks against the > > > previous run and it only checks a few of the keys by default. You can check > > > latency if you want, but while writing this stuff up it seemed that latency was > > > too variable from run to run to be useful in a "did my thing regress or improve" > > > sort of way. > > > > > > The configuration is brain dead simple, the README has examples. All you need > > > to do is make your local.cfg, run ./setup and then run ./fsperf and you are good > > > to go. > > > > Why re-invent the test infrastructure? Why not just make it a > > tests/perf subdir in fstests? > > > > Probably should have led with that shouldn't I have? There's nothing keeping me > from doing it, but I didn't want to try and shoehorn in a python thing into > fstests. I need python to do the sqlite and the json parsing to dump into the > sqlite database. > > Now if you (and others) are not opposed to this being dropped into tests/perf > then I'll work that up. But it's definitely going to need to be done in python. > I know you yourself have said you aren't opposed to using python in the past, so > if that's still the case then I can definitely wire it all up. I have no problems with people using python for stuff like this but, OTOH, I'm not the fstests maintainer anymore :P > > > The plan is to add lots of workloads as we discover regressions and such. We > > > don't want anything that takes too long to run otherwise people won't run this, > > > so the existing tests don't take much longer than a few minutes each. I will be > > > adding some more comparison options so you can compare against averages of all > > > previous runs and such. > > > > Yup, that fits exactly into what fstests is for... :P > > > > Integrating into fstests means it will be immediately available to > > all fs developers, it'll run on everything that everyone already has > > setup for filesystem testing, and it will have familiar mkfs/mount > > option setup behaviour so there's no new hoops for everyone to jump > > through to run it... > > > > TBF I specifically made it as easy as possible because I know we all hate trying > to learn new shit. Yeah, it's also hard to get people to change their workflows to add a whole new test harness into them. It's easy if it's just a new command to an existing workflow :P > I figured this was different enough to warrant a separate > project, especially since I'm going to add block device jobs so Jens can test > block layer things. If we all agree we'd rather see this in fstests then I'm > happy to do that too. Thanks, I'm not fussed either way - it's a good discussion to have, though. If I want to add tests (e.g. my time-honoured fsmark tests), where should I send patches? Cheers, Dave. -- Dave Chinner david@xxxxxxxxxxxxx
