> So after some further consideration I think I'm on board with this, and > it turned out not to be too hard to implement. I have updated the draft > test case: > > https://fedoraproject.org/wiki/User:Adamwill/Draft_Testcase_Boot_default_install > > so it now basically says 'you can run any kind of install that boots > from the medium under test and uses the default repo, and if it > satisfies the Expected Results, we're good'. > > Does this look better to you now? Yes, thanks. > I think this is going a bit too far. I'm OK with not requiring complete > coverage at Alpha or Beta, but I think explicitly moving optical boot > to the Final criteria is too much. I think if we do find out that > optical boot is broken for a release-blocking medium at Alpha or Beta > stage that should still be a blocker. At least for now, until the > project as a whole decides we don't care about optical media any more. OK. However, unless you're strongly opposed, I'd like to start discussing exactly this (requiring optical media to work only for Final, not before) and have it agreed on for F26. From my POV, there's little gain in making sure optical works for Alpha/Beta. Optical is important for conservative audience with very old machines, and is probably in a similar position to i386. Also for giveaways. All of that is important for Final, but isn't really for Alpha and Beta. I don't think we have too many users which regularly install the latest and greatest betaversions from optical drives. (Also, the risk of breaking just optical boot on bare metal seems pretty low, even though it happened in the past). > > So I went with a halfway house. I've updated the matrix draft too: > > https://fedoraproject.org/wiki/User:Adamwill/Draft_Installation_test_matrix > > so it now lists the milestone as 'Alpha / Final' and has an admon that > explicitly explains the 'Expected coverage' for the section, saying we > expect a reasonable sample of tests for Alpha and Beta, and full > coverage for Final. Do you think that's sufficient? Just to have an idea, say that we have both VM* columns fully covered by OpenQA, but for DVD columns with have just a single BIOS+UEFI result for Workstation Live. Would that be sufficient for Beta? What do we consider "a reasonable coverage"? > > Note that with the current wikitcms code, testcase_stats considers the > milestone for 'Alpha / Final' rows to be 'Final'...because it works > like this: > > for mile in ('Alpha', 'Beta', 'Final', 'Optional', 'Tier1', 'Tier2', > 'Tier3'): > if mile in cells[0]: > milestone = mile > > obviously that would be very trivial to tweak, I just thought I'd > mention it. I might tweak it to prefer the first milestone it finds > rather than the last, I'll just have to look and see if that would > cause any unexpected effects in other cases (I don't think we have > many, though). Yeah, tweaking that would probably make sense. _______________________________________________ test mailing list -- test@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to test-leave@xxxxxxxxxxxxxxxxxxxxxxx
