On Sat, May 12, 2018 at 08:51:08PM +0800, Eryu Guan wrote: > On Mon, May 07, 2018 at 08:46:34AM +1000, Dave Chinner wrote: > > From: Dave Chinner <dchinner@xxxxxxxxxx> > > > > Currently a test passes even if it leaves a corrupt filesystem > > behind, or a splat in the system logs that should not be there. > > It seems that test does fail when post-test fsck fails or dmesg check > reports failure, but just after the test runtime being recorded & > reported, which makes the test looks like a PASS. Yup, that's exactly what I said - the test fails, but the reporting indicates the test passed before it reports failure. And because these failures match the "test pass" signature, my results post-processing script was saying these tests passed, not had a "post-test failure". The post-test failure messages don't mention the test directly, either, so it's a bit of a wart when it comes to processing the output for regression test comparisons... Hence I thought I'd fix it so they didn't look like a test pass. > But the test summary > does report it as a failure, e.g. (I added "echo BUG: > /dev/kmsg" to > generic/444 manually) Right, but I don't look at those for comparing run-to-run regression test results - having a timestamp that is wildly different is a regression that needs investigation, too, even though we are only looking at correctness in the test harness... > generic/443 0s ... 0s > generic/444 0s ... 0s > _check_dmesg: something found in dmesg (see /root/workspace/xfstests/results//xfs_2k_crc/generic/444.dmesg) > generic/445 0s ... 0s > Ran: generic/443 generic/444 generic/445 > Failures: generic/444 > Failed 1 of 3 tests > > And the return value of check is also non-zero. Yes, that is also true, but it is also not useful to tell me what test out of the 700+ I've just run failed. > But the whole rework looks good! Just need to reword the patch summary > and description? Ok. Cheers, Dave. -- Dave Chinner david@xxxxxxxxxxxxx -- To unsubscribe from this list: send the line "unsubscribe fstests" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
