On Tue, 2018-02-27 at 09:59 -0500, Laine Stump wrote: > > So should I just drop this one, or is it still somewhat valuable > > to have any sort of test suite coverage for PCI controller options? > > I was thinking that having this test is better than not having this > test. But then I thought about what would happen if there was a > regression in just a single one of these validations - the negative test > would still "succeed" (i.e. "succeed in detecting a failure") because it > would hit the next disallowed attribute. As a matter of fact, the test > would continue to "succeed" until there was a regression in the > validation of every single attribute, so that no error would be > triggered. So having a negative test that has multiple examples of > failures actually gives us a false sense of security - we believe that > it's verifying we're catching incorrect config, but it won't actually > notify us until *all* of the bad config is missed by validation. > > > So, I think each negative test should have exactly one piece of > incorrect data. That necessarily means that it's only testing for proper > validation of a single aspect of a single attribute. But making this > generally useful with the current test apparatus would mean a huge > explosion in the number of test files, and I don't think that's > practical. But if we're only testing for one out of a thousand > validations, there's really not much point in it. Okay, I'll drop it and respin then. -- Andrea Bolognani / Red Hat / Virtualization -- libvir-list mailing list libvir-list@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/libvir-list
