On Thu 10-03-22 14:41:30, Luis Chamberlain wrote: > On Thu, Mar 10, 2022 at 01:51:22PM -0500, Josef Bacik wrote: > > On Wed, Mar 09, 2022 at 05:28:28PM -0800, Luis Chamberlain wrote: > > > On Wed, Mar 09, 2022 at 04:19:21PM -0500, Josef Bacik wrote: > > > > On Wed, Mar 09, 2022 at 11:00:49AM -0800, Luis Chamberlain wrote: > > > > > > That's great! > > > > > > But although this runs nightly, it seems this runs fstest *once* to > > > ensure if there are no regressions. Is that right? > > > > > > > Yup once per config, so 8 full fstest runs. > > From my experience that is not enough to capture all failures given > lower failure rates on tests other than 1/1, like 1/42 or > 1/300. So minimum I'd go for 500 loops of fstests per config. > This does mean this is not possible nightly though, yes. 5 days > on average. And so much more work is needed to bring this down > further. Well, yes, 500 loops have better chance of detecting rare bugs. But if you did only say 100 loops, you are likely to detect the bug just 5 days later on average. Sure that makes finding the bug somewhat harder (you generally need to investigate larger time span to find the bug) but testing costs are lower... It is a tradeoff. Honza -- Jan Kara <jack@xxxxxxxx> SUSE Labs, CR
