On Sun, Oct 21, 2018 at 05:52:21PM -0500, Jayashree Mohan wrote: > > How long does each test case take to run? > All the tests would touch 4 files at most or write 32-64KB of data to > a file, starting from a empty file system image (hence very minimal > running time). We are not going to bring CrashMonkey in the loop - we > will port the generated tests to xfstest using dm-flakey (like > generic/498 [1], which was submitted in response to a bug found by > CrashMonkey in btrfs). Hence, each test should take the same time as > that of current crash-consistency tests in the xfstest suite (for > example, similar to generic/034 which takes about a second to run). > > > And note, by the way, that > > by default we automatically run fsck on the test device after each > > test. So number one, if you use the test device, you don't need to > > worry about running fsck explicitly; the xfstests check script will do > > that, and fail the test if the file system is corrupted --- and number > > two, this will influence whether which groups each test should be > > assigned. > > Noted. Since we will be writing the out-file (checker) manually, will > ensure that checks only the content/metadata. > > > See the file xfstests-dev/tests/generic/group to see how groups get > > assigned to tests. I suppose all of the crashmonkey tests should be > > assigned to a new group, say, "crashmonkey". Whether or not they > > should get assigned to the "auto" or "quick" group is a different > > question. Note that if running these tests will signicantly increase > > the test run time of smoke tests and even the full "automatic" > > regression tests, there may be some resistence in adding all of these > > tests to the "auto" or "quick" groups. Or even if you do, many file > > system developers may choose to exclude all tests from the > > "crashmonkey" group because if a 15 minute smoke test suddenly gets > > extended to take 6 hours, developers are wont to get.... cranky. :-) > > It makes sense to add it to a new group as you suggest, and > considering a second to run each test, it should take around 5 minutes > to run this batch of CrashMonkey tests. Once the tests cases are > ready, we can give you a better estimate of total time spent on the > newly added tests. Add the time between tests - fsck checks, scrub, etc, and that can easily add another 10s per test. Hence I'd strongly encourage you to batch similar tests into a single xfstest test so that we're not needlessly adding 300x15s to every test run because of the per-test external overhead..... Cheers, Dave. -- Dave Chinner david@xxxxxxxxxxxxx
