Hi folks! Sending this to devel@ as well as test@ as there's been some relevant discussion there recently. We've been kicking around a couple of issues lately: 1. Exactly what do we need to test and block on, in terms of writing images to USB sticks? 2. 'Default boot and install' table was supposed to require bare metal optical media testing at release time, but this is not clear, and openQA fills in results despite not testing real media on bare metal. So here's my proposal to address both topics. It involves changes to the Boot_default_install test: https://fedoraproject.org/wiki/User:Adamwill/Draft_Testcase_Boot_default_install https://fedoraproject.org/w/index.php?title=User:Adamwill/Draft_Testcase_Boot_default_install&diff=476645&oldid=476644 and the Installation matrix template page: https://fedoraproject.org/wiki/User:Adamwill/Draft_Installation_test_matrix https://fedoraproject.org/w/index.php?title=User:Adamwill/Draft_Installation_test_matrix&diff=476646&oldid=476643 to summarize the changes - we improve the test case's wording to give explicit instructions for all three media types we care about (virtual disc attached to a VM, real optical disc in a real machine, real USB stick in a real machine), we ditch the USB test matrices entirely, and we give the 'Default boot and install' tables extra result columns so they can now reflect results for VM, CD/DVD and USB testing (for BIOS and UEFI in all three cases, for x86_64). I didn't get too finicky about the wording for the virt case, as that's basically there to get filled in by openQA anyway. For the CD/DVD case, the test case instructs you to write the disc according to the 'official instructions', with a URL that is supposed to be always provided by the docs project for that purpose. For the USB case, the test case instructs you to write the stick according to https://fedoraproject.org/wiki/How_to_create_and_use_Live_USB ;, telling you to use the 'most prominently-recommended method'; the intent being that, as things stand, we would only block on Fedora Media Writer issues. I didn't explicitly cover dd (e.g. with separate FMW and dd columns) for two reasons: 1. It should pretty reliably be the case that if FMW works, dd works 2. Some people are going to test plain dd anyway, it's just such a convenient 'expert' method that I can't imagine we won't have people using it throughout the test period I didn't cover livecd-iso-to-disk because it seems like, overall, the feedback to cmurf's "F26 proposal: Make Fedora Media Writer the officially supported USB install media creator" was fairly positive and I think justifies not guaranteeing testing for it. To be clear, this is a trade-off between 'in an ideal world we'd test everything' and 'but we really don't have time'. Each additional method adds 12 mandatory tests (6 images, both UEFI and BIOS variants of the test). We can discount the virt tests, as openQA will do those, so in the proposal, there are 24 mandatory tests for humans; if we added dd as its own thing there'd be 36, if we added litd as well there'd be 48. One note is that this doesn't do much to ensure that FMW is working at release time in all expected environments (probably at least the two previous stable Fedora releases, macOS, and Windows). On the other hand, neither does the current matrix; we explicitly list FMW, but we don't specify what environments it should be run in. We could consider adding a USB table with a slightly different focus, the idea being to test the *writing tool* rather than test that the *image* can be written and works properly. We could make that a follow-up proposal. If we had such a table, it could maybe cover testing livecd-iso-to-disk's advanced features as optional tests, or something. At present I'm not proposing any changes to the *release criteria* (we can maybe consider that later). This is just a proposal for what we will actually guarantee testing of via the validation process. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net http://www.happyassassin.net _______________________________________________ test mailing list -- test@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to test-leave@xxxxxxxxxxxxxxxxxxxxxxx
