> Hi folks! Sending this to devel@ as well as test@ as there's been some > relevant discussion there recently. We've been kicking around a couple > of issues lately: > > 1. Exactly what do we need to test and block on, in terms of writing > images to USB sticks? > > 2. 'Default boot and install' table was supposed to require bare metal > optical media testing at release time, but this is not clear, and > openQA fills in results despite not testing real media on bare metal. > > So here's my proposal to address both topics. It involves changes to > the Boot_default_install test: > > https://fedoraproject.org/wiki/User:Adamwill/Draft_Testcase_Boot_default_install > https://fedoraproject.org/w/index.php?title=User:Adamwill/Draft_Testcase_Boot_default_install&diff=476645&oldid=476644 > > and the Installation matrix template page: > > https://fedoraproject.org/wiki/User:Adamwill/Draft_Installation_test_matrix > https://fedoraproject.org/w/index.php?title=User:Adamwill/Draft_Installation_test_matrix&diff=476646&oldid=476643 > > to summarize the changes - we improve the test case's wording to give > explicit instructions for all three media types we care about (virtual > disc attached to a VM, real optical disc in a real machine, real USB > stick in a real machine), we ditch the USB test matrices entirely, and > we give the 'Default boot and install' tables extra result columns so > they can now reflect results for VM, CD/DVD and USB testing (for BIOS > and UEFI in all three cases, for x86_64). It's a lot of empty fields, which can't be automated by definition. The new matrix version seems to be more time consuming that the previous version. I wonder what we can do to cut down the time spent on this. Here are a few ideas: a) could we decouple boot from install? I know we've had a few problems in the past that occurred only on bare metal and not in VMs. But do I remember correctly that all of those were related to booting the installer/live system, and not the installation? What we do here is that we repeat the default installation over and over again, e.g. Workstation Live using BIOS and UEFI (which results in a slightly different partition layout), from different sources (DVD in VM, DVD on bare metal, USB). But do those environments (VM vs bare metal, DVD vs USB) actually matter, is there a good reason to repeat them that many times? What if we reduce all of this to: Default install =============== Result Workstation Live [ ] Workstation netinst [ ] Everything netinst [ ] Server netinst [ ] Server DVD [ ] KDE Live [ ] which would be satisfiable using ANY environment and it would just check that anaconda can use that particular installation source correctly? The BIOS vs UEFI partitioning is already covered thoroughly in the "Guided storage configuration" table. The currently proposed "Default boot and install" section would stay the same, but be renamed to "Default boot" and would just check that the image *boots* into the installer in that particular environment. b) do we need to insist on performing the *default* install? If the previous idea doesn't fly, this is another way to improve testing time. For example, we're installing the Server package set 12 times in the currently proposed table ((Server DVD + Server netinst) * environments). Is that really necessary? What if we did it once (in any environment), and then could do just a minimal install the remaining 11 times? Because installation takes most of the time here, esp. from optical media. And I can't imagine how the environments could differ in such a way that would break default installation set just in a particular environment but not a different installation set in the same environment (esp. when heavily pushing dd-like based approach). So again we would have a table like this: Default package set =================== Result Workstation netinst [ ] Server netinst [ ] Server DVD [ ] which would be satisfiable in any environment (we already have "Package sets" section in which this would fit nicely), and in the "Default boot and install" table you could choose a different package set for installation. We could go even further and allow arbitrary partitioning to be used in "Default boot and install", so that it can be combined with our partitioning test cases. Again, that would allow us to test multiple features together and save time, while not sacrificing the core thing to test here (running the particular medium in a particular environment). > > I didn't get too finicky about the wording for the virt case, as that's > basically there to get filled in by openQA anyway. For the CD/DVD case, > the test case instructs you to write the disc according to the > 'official instructions', with a URL that is supposed to be always > provided by the docs project for that purpose. For the USB case, the > test case instructs you to write the stick according to > https://fedoraproject.org/wiki/How_to_create_and_use_Live_USB , telling > you to use the 'most prominently-recommended method'; the intent being > that, as things stand, we would only block on Fedora Media Writer > issues. I didn't explicitly cover dd (e.g. with separate FMW and dd > columns) for two reasons: > > 1. It should pretty reliably be the case that if FMW works, dd works > 2. Some people are going to test plain dd anyway, it's just such a > convenient 'expert' method that I can't imagine we won't have people > using it throughout the test period > > I didn't cover livecd-iso-to-disk because it seems like, overall, the > feedback to cmurf's "F26 proposal: Make Fedora Media Writer the > officially supported USB install media creator" was fairly positive and > I think justifies not guaranteeing testing for it. All of that sounds fine to me. > > To be clear, this is a trade-off between 'in an ideal world we'd test > everything' and 'but we really don't have time'. Each additional method > adds 12 mandatory tests (6 images, both UEFI and BIOS variants of the > test). We can discount the virt tests, as openQA will do those, so in > the proposal, there are 24 mandatory tests for humans; if we added dd > as its own thing there'd be 36, if we added litd as well there'd be 48. > > One note is that this doesn't do much to ensure that FMW is working at > release time in all expected environments (probably at least the two > previous stable Fedora releases, macOS, and Windows). On the other > hand, neither does the current matrix; we explicitly list FMW, but we > don't specify what environments it should be run in. We could consider > adding a USB table with a slightly different focus, the idea being to > test the *writing tool* rather than test that the *image* can be > written and works properly. We could make that a follow-up proposal. If > we had such a table, it could maybe cover testing livecd-iso-to-disk's > advanced features as optional tests, or something. Makes sense. > > At present I'm not proposing any changes to the *release criteria* (we > can maybe consider that later). This is just a proposal for what we > will actually guarantee testing of via the validation process. If we go full steam ahead with optical disk testing as proposed (in the past, the "default boot and install" test case said you can choose either USB or DVD, now we have separate fields for both implying we need to test both) I'd suggest optical media being only in the Final criteria. _______________________________________________ test mailing list -- test@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to test-leave@xxxxxxxxxxxxxxxxxxxxxxx
