> It's optimized for developers, and for our use cases. I'm sure > kdevops is much more general, since it can work for hardware-based > test machines, as well as many other cloud stacks, and it's also > optimized for the QA department --- not surprising, since where > kdevops has come from. > [...] > > We also have a very different philosophy about how to use expunge > files. In paticular, if there is test which is only failing 0.5% of > the time, I don't think it makes sense to put that test into an > expunge file. > > In general, we are only placing tests into expunge files when > it causes the system under test to crash, or it takes *WAAAY* too > long, or it's a clear test bug that is too hard to fix for real, so we > just suppress the test for that config for now. (Example: tests in > xfstests for quota don't understand clustered allocation.) > > So we want to run the tests, even if we know it will fail, and have a > way of annotating that a test is known to fail for a particular kernel > version, or if it's a flaky test, what the expected flake percentage > is for that particular test. For flaky tests, we'd like to be able > automatically retry running the test, and so we can flag when a flaky > test has become a hard failure, or a flaky test has radically changed > how often it fails. We haven't implemented all of this yet, but this > is something that we're exploring the design space at the moment. > > More generally, I think competition is a good thing, and for areas > where we are still exploring the best way to automate tests, not just > from a QA department's perspective, but from a file system developer's > perspective, having multiple systems where we can explore these ideas > can be a good thing. > I very much agree with Ted on that point. As a user and big fan of both kdevops and fstests-bld I wouldn't want to have to choose one over the other, not even to choose a unified expunge list. I think we are still at a point where this diversity makes our ecosystem stronger rather than causing duplicate work. To put it in more blunt terms, the core test suite, fstests, is not very reliable. Neither kdevops nor fstests-bld address all the reliability issue (and they contribute some of their own). So we need the community to run both to get better and more reliable filesystem test coverage. Nevertheless, we should continue to share as much experience and data points as we can during this co-opetition stage in order to improve both systems. Thanks, Amir.
