On Tue, 2016-12-06 at 10:28 -0500, Matthew Miller wrote: > > > Idea #2: Do not block on optical media issues for Final release for > > certain flavors/image types (Server, netinst) > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > +1 to this one. But is there likely to be a case where it fails on just > those? I guess this primarily reduces what you need to _test_. So, > yeah, +1. The nexus between 'test coverage' and 'blocker status' is a tricky one. On the one hand, we've generally held, ever since introducing the release criteria, that it's possible for something that's not covered in regular testing to block the release. This has always been a thing that could happen, and we've explicitly acknowledged the scenario at various times. On the other hand, a standard question when a blocker is proposed very late is 'why wasn't this tested earlier?', and it always seems like a good question at the time. So we could *in theory* say that we don't require testing of every release-blocking ISO as burned to an optical disc, but we would block on any one of them being broken if someone happens to find that it is. But we'd have to own that decision and not just refuse to grant blocker status on the grounds that 'it wasn't tested early enough', etc. As for the issue of how likely failures are in different images, it is possible for bugs to affect some ISOs but not others, as the code paths by which the different images are built are quite different. It's certainly possible for an issue to affect lives but not traditional installer images, or vice versa. It's much less likely that one live image would be broken (in terms of actual bootability) but another would not. I think it's less likely, but *possible*, for DVD to work but netinst not (or vice versa) - I think we once actually had a case where one of them had isohybrid run on it but the other didn't (though that of course would affect *USB* boot, not optical media). I think it's very unlikely for one netinst to work but another not. So an alternative to kparal's scheme would be to try and consider this, and say we test: * Workstation live * Everything netinst * Server DVD and consider those to be representative of the broad 'types' of ISOs in terms of the compose process. That way we don't have to test Workstation or Server netinsts, or the KDE live, on optical media. I do have a kinda old-fashioned attachment to the idea that a real human being should boot and install, in at least *some* way, each of the release-blocking media we ship. But that might just be a personal bias. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net http://www.happyassassin.net _______________________________________________ test mailing list -- test@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to test-leave@xxxxxxxxxxxxxxxxxxxxxxx
