On Mon, 2016-10-10 at 10:25 -0400, Kamil Paral wrote: > > Hi folks! Sending this to devel@ as well as test@ as there's been some > > relevant discussion there recently. We've been kicking around a couple > > of issues lately: > > > > 1. Exactly what do we need to test and block on, in terms of writing > > images to USB sticks? > > > > 2. 'Default boot and install' table was supposed to require bare metal > > optical media testing at release time, but this is not clear, and > > openQA fills in results despite not testing real media on bare metal. > > > > So here's my proposal to address both topics. It involves changes to > > the Boot_default_install test: > > > > https://fedoraproject.org/wiki/User:Adamwill/Draft_Testcase_Boot_default_install > > https://fedoraproject.org/w/index.php?title=User:Adamwill/Draft_Testcase_Boot_default_install&diff=476645&oldid=476644 > > > > and the Installation matrix template page: > > > > https://fedoraproject.org/wiki/User:Adamwill/Draft_Installation_test_matrix > > https://fedoraproject.org/w/index.php?title=User:Adamwill/Draft_Installation_test_matrix&diff=476646&oldid=476643 > > > > to summarize the changes - we improve the test case's wording to give > > explicit instructions for all three media types we care about (virtual > > disc attached to a VM, real optical disc in a real machine, real USB > > stick in a real machine), we ditch the USB test matrices entirely, and > > we give the 'Default boot and install' tables extra result columns so > > they can now reflect results for VM, CD/DVD and USB testing (for BIOS > > and UEFI in all three cases, for x86_64). It's a lot of empty fields, which can't be automated by definition. Yep, yep it is. > The new matrix version seems to be more time consuming that the > previous version. In fact, the count of 'empty spaces' is identical in both versions. In the old version there were 24 empty spaces that humans were *supposed* to be filling in (though this wasn't always really happening): 12 in the "Default boot and install" table, and 12 in the USB table (it's a 4x3 table). In the new version, there are...24 empty spaces for humans to fill in in the "Default boot and install (x86_64)" table. The coverage is slightly *different*, but the amount of tests humans are intended to run has not in fact changed. We lose the fudge factor of openQA filling in the spaces I really intended humans to test on real machines with real shiny discs, but that's kind of the *point* of the change, so it's not a reason to not do it... > I wonder what we can do to cut down the time spent on this. Here are > a few ideas: > > a) could we decouple boot from install? > > I know we've had a few problems in the past that occurred only on > bare metal and not in VMs. But do I remember correctly that all of > those were related to booting the installer/live system, and not the > installation? What we do here is that we repeat the default > installation over and over again, e.g. Workstation Live using BIOS > and UEFI (which results in a slightly different partition layout), > from different sources (DVD in VM, DVD on bare metal, USB). But do > those environments (VM vs bare metal, DVD vs USB) actually matter, is > there a good reason to repeat them that many times? What if we reduce > all of this to: > > Default install > =============== > Result > Workstation Live [ ] > Workstation netinst [ ] > Everything netinst [ ] > Server netinst [ ] > Server DVD [ ] > KDE Live [ ] > > which would be satisfiable using ANY environment and it would just > check that anaconda can use that particular installation source > correctly? The BIOS vs UEFI partitioning is already covered > thoroughly in the "Guided storage configuration" table. > The currently proposed "Default boot and install" section would stay > the same, but be renamed to "Default boot" and would just check that > the image *boots* into the installer in that particular environment. I think this is going a bit too far. There are, IIRC, at least two known places beyond just initial boot where things can go wrong in ways related to the install medium. One is install target selection. anaconda has code for filtering out 'protected' devices which is intended (in part) to prevent you trying to install to the USB stick you're running the install from. That code can go wrong in various ways, including over-aggressive filtering, and the bugs can be media-specific (e.g. there's no problem in a VM or from a real optical disc, but it goes wrong when booting from a USB stick). The other is use of the on-medium repository. This is mainly for the Server DVD image. When you install from that image the install should use the packages from the medium itself as the base repository (and not go out and get them from the network); I believe we've seen a case before where the Server DVD written to USB would boot OK, and even install OK, but it didn't actually use the USB stick as a repository, it acted like a network install image. When written to a DVD it worked OK. That was a long time ago, though, and I don't have the bug reference handy. This is why the test case has this text: "If the image under test is intended to act as a package source, it must be the default repository and installation must in fact use the media as the package source." We *could* reduce the test case to 'boot, select a target disk and start the installation process if using a DVD image', I guess. Would this sufficiently address your concerns? Of course it's *possible* that some weird bug could show up which made bootloader configuration not work properly in some media-dependent way, or something. But I don't recall us running into such a bug before. > > > > At present I'm not proposing any changes to the *release criteria* (we > > can maybe consider that later). This is just a proposal for what we > > will actually guarantee testing of via the validation process. > > If we go full steam ahead with optical disk testing as proposed (in > the past, the "default boot and install" test case said you can > choose either USB or DVD, now we have separate fields for both > implying we need to test both) I'd suggest optical media being only > in the Final criteria. That was my intent, but it's a bit hard to express with the way we lay out the wiki tables, unfortunately (hard to specify different milestones for different environments). I can add an admon note above the table, I guess, explaining precisely what's actually required for each milestone. How does that sound? Thanks for the feedback! -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net http://www.happyassassin.net _______________________________________________ test mailing list -- test@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to test-leave@xxxxxxxxxxxxxxxxxxxxxxx
