On Fri, Oct 7, 2016 at 2:08 PM, Adam Williamson <adamwill@xxxxxxxxxxxxxxxxx> wrote: > Hi folks! Sending this to devel@ as well as test@ as there's been some > relevant discussion there recently. We've been kicking around a couple > of issues lately: > > 1. Exactly what do we need to test and block on, in terms of writing > images to USB sticks? > > 2. 'Default boot and install' table was supposed to require bare metal > optical media testing at release time, but this is not clear, and > openQA fills in results despite not testing real media on bare metal. > > So here's my proposal to address both topics. It involves changes to > the Boot_default_install test: > > https://fedoraproject.org/wiki/User:Adamwill/Draft_Testcase_Boot_default_install > https://fedoraproject.org/w/index.php?title=User:Adamwill/Draft_Testcase_Boot_default_install&diff=476645&oldid=476644 > > and the Installation matrix template page: > > https://fedoraproject.org/wiki/User:Adamwill/Draft_Installation_test_matrix > https://fedoraproject.org/w/index.php?title=User:Adamwill/Draft_Installation_test_matrix&diff=476646&oldid=476643 > > to summarize the changes - we improve the test case's wording to give > explicit instructions for all three media types we care about (virtual > disc attached to a VM, real optical disc in a real machine, real USB > stick in a real machine), we ditch the USB test matrices entirely, and > we give the 'Default boot and install' tables extra result columns so > they can now reflect results for VM, CD/DVD and USB testing (for BIOS > and UEFI in all three cases, for x86_64). > > I didn't get too finicky about the wording for the virt case, as that's > basically there to get filled in by openQA anyway. For the CD/DVD case, > the test case instructs you to write the disc according to the > 'official instructions', with a URL that is supposed to be always > provided by the docs project for that purpose. For the USB case, the > test case instructs you to write the stick according to > https://fedoraproject.org/wiki/How_to_create_and_use_Live_USB , telling > you to use the 'most prominently-recommended method'; the intent being > that, as things stand, we would only block on Fedora Media Writer > issues. I didn't explicitly cover dd (e.g. with separate FMW and dd > columns) for two reasons: > > 1. It should pretty reliably be the case that if FMW works, dd works > 2. Some people are going to test plain dd anyway, it's just such a > convenient 'expert' method that I can't imagine we won't have people > using it throughout the test period > > I didn't cover livecd-iso-to-disk because it seems like, overall, the > feedback to cmurf's "F26 proposal: Make Fedora Media Writer the > officially supported USB install media creator" was fairly positive and > I think justifies not guaranteeing testing for it. > > To be clear, this is a trade-off between 'in an ideal world we'd test > everything' and 'but we really don't have time'. Each additional method > adds 12 mandatory tests (6 images, both UEFI and BIOS variants of the > test). We can discount the virt tests, as openQA will do those, so in > the proposal, there are 24 mandatory tests for humans; if we added dd > as its own thing there'd be 36, if we added litd as well there'd be 48. > > One note is that this doesn't do much to ensure that FMW is working at > release time in all expected environments (probably at least the two > previous stable Fedora releases, macOS, and Windows). On the other > hand, neither does the current matrix; we explicitly list FMW, but we > don't specify what environments it should be run in. We could consider > adding a USB table with a slightly different focus, the idea being to > test the *writing tool* rather than test that the *image* can be > written and works properly. We could make that a follow-up proposal. If > we had such a table, it could maybe cover testing livecd-iso-to-disk's > advanced features as optional tests, or something. > > At present I'm not proposing any changes to the *release criteria* (we > can maybe consider that later). This is just a proposal for what we > will actually guarantee testing of via the validation process. Looks and sounds good to me. -- Chris Murphy _______________________________________________ test mailing list -- test@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to test-leave@xxxxxxxxxxxxxxxxxxxxxxx
