On Sun, Jul 03, 2022 at 08:56:54AM +0300, Amir Goldstein wrote: > > That is true for some use cases, but unfortunately, the flaky > fstests are way too valuable and too hard to replace or improve, > so practically, fs developers have to run them, but not everyone does. > > Zorro has already proposed to properly tag the non deterministic tests > with a specific group and I think there is really no other solution. The non-deterministic tests are not the sole, or even the most likely cause of flaky tests. Or put another way, even if we used a deterministic pseudo-random numberator seed for some of the curently "non-determinstic tests" (and I believe we are for many of them already anyway), it's not going to be make the flaky tests go away. That's because with many of these tests, we are running multiple threads either in the fstress or fsx, or in the antogonist workload that is say, running the space utilization to full to generate ENOSPC errors, and then deleting a bunch of files to trigger as many ENOSPC hitter events as possible. > The only question is whether we remove them from the 'auto' group > (I think we should). I wouldn't; if someone wants to exclude the non-determistic tests, once they are tagged as belonging to a group, they can just exclude that group. So there's no point removing them from the auto group IMHO. > filesystem developers that will run ./check -g auto -g soak > will get the exact same test coverage as today's -g auto > and the "commoners" that run ./check -g auto will enjoy blissful > determitic test results, at least for the default config of regularly > tested filesystems (a.k.a, the ones tested by kernet test bot).? First of all, there are a number of tests today which are in soak or long_rw which are not in auto, so "-g auto -g soak" will *not* result in the "exact same test coverage". Secondly, as I've tested above, deterministic tests does not necessasrily mean determinsitic test results --- unless by "determinsitic tests" you mean "completely single-threaded tests", which would eliminate a large amount of useful test coverage. Cheers, - Ted
