On Wed, Mar 09, 2022 at 10:41:53AM -0800, Luis Chamberlain wrote: > On Tue, Mar 08, 2022 at 11:40:18AM -0500, Theodore Ts'o wrote: > > One of my team members has been working with Darrick to set up a set > > of xfs configs[1] recommended by Darrick, and she's stood up an > > automated test spinner using gce-xfstests which can watch a git branch > > and automatically kick off a set of tests whenever it is updated. > > I think its important to note, as we would all know, that contrary to > most other subsystems, in so far as blktests and fstests is concerned, > simply passing a test once does not mean there is no issue given that > some test can fail with a failure rate of 1/1,000 for instance. > FWIW we (the btrfs team) have been running nightly runs of fstests against our devel branch for over a year and tracking the results. This allowed us to get down to 0 failures because we could identify flakey tests and fix them or simply disable them. Then this means when we do have one of those 1/1,000 failures in one of our configs (I copied Ted's approach and test all the various feature combos) we know what set of parameters the failure was on and can go run that test in a loop to reproduce the problem. We like this approach because it's not a "wait a week to see if something failed", we know the day after some new thing was merged if it caused a problem. If it's more subtle then we still find it because a test will start failing at some point. It's a nice balance in how long we have to wait for results and allows us to be a lot more sure in merging new code without hemming and hawing for months. Thanks, Josef
