On Thu, May 22, 2014 at 12:45:05PM +0200, chrubis@xxxxxxx wrote: > Right, sorting out right testcases is not easy task. But given that the > runtime is not that long we can do rough selection and end up with > something that runs in about 10 or 20 minutes and covers all possibly > relevant testcases. Something to keep in mind is that for at least some file system developers, such as myself, we end up running certain tests multiple times, so we can get test coverage for different file system mount options, block sizes, file system features, etc. Currently, I have ten or so different configurations for my full test run. So if the bulk of the 20 minutes are tests for which there is duplicate coverage with xfstests, or which is stuff that is only testing behaviour implemented in the VFS, that's actually 3-4 extra hours of testing as far as I'm concerned. Hence, the longer and the longer the tests take, the less often I will run them --- and the effect is not linear. So failing to eliminate tests which are not relevant is not cost-free. In some cases, the costs of winnowing out non-relevant tests may not be worth the effort. If it's only an extra 5 seconds, I probably wouldn't care. Although if you multiply the time saved by the number of file system developers, and the number of full test runs that they likely will be running during the development cycle, spending a few minutes to disable a non-relevant test starts becoming net-positive. > To be frank the reputation is one thing I would really like to fix. > Nothing is more frustrating than spending years fixing code and still > being ignored because of the past you haven't been even involved in. Well, to the extent that many years ago, I was employed by one of the companies had a well-meaning, but ultimately counterproductive contribution to LTP, by putting relatively junior test engineers that didn't understand the kernel code all that deeply, and was given the mandate to increase the code coverage percentages as the only figure of merit, I was actually somewhat involved, if only indirectly. Unfortunately, I wasn't able to effect change by advocating strongly enough for a more productive approach (which would have required putting in far more senior, and thus more expensive and harder to allocate engineering talent), although I did help eventually help by arguing that ceasing support for what I considered to be a deeply dysfunctional effort as the best thing that particular company could have done at that point. :-( So I really have to commend Xiaoguang Wang for having done a large amount of analysis when submitting the report about the LTP test failure. That's significantly different from my previous experience interacting with previous people working on the LTP project, and it's precisely that sort of work and analysis which I'm sure will help turn around LTP's reputation. Cheers, - Ted -- To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
