> > The new matrix version seems to be more time consuming that the > > previous version. > > In fact, the count of 'empty spaces' is identical in both versions. In > the old version there were 24 empty spaces that humans were *supposed* > to be filling in (though this wasn't always really happening): 12 in > the "Default boot and install" table, and 12 in the USB table (it's a > 4x3 table). In the new version, there are...24 empty spaces for humans > to fill in in the "Default boot and install (x86_64)" table. Yes, their number is the same, but I meant the amount of time required. The USB test cases allowed us to combine them with other test cases (e.g. partitioning) or perform a minimal install, while the new test case doesn't (a default install is mandatory). > > a) could we decouple boot from install? > > > > I know we've had a few problems in the past that occurred only on > > bare metal and not in VMs. But do I remember correctly that all of > > those were related to booting the installer/live system, and not the > > installation? What we do here is that we repeat the default > > installation over and over again, e.g. Workstation Live using BIOS > > and UEFI (which results in a slightly different partition layout), > > from different sources (DVD in VM, DVD on bare metal, USB). But do > > those environments (VM vs bare metal, DVD vs USB) actually matter, is > > there a good reason to repeat them that many times? What if we reduce > > all of this to: > > > > Default install > > =============== > > Result > > Workstation Live [ ] > > Workstation netinst [ ] > > Everything netinst [ ] > > Server netinst [ ] > > Server DVD [ ] > > KDE Live [ ] > > > > which would be satisfiable using ANY environment and it would just > > check that anaconda can use that particular installation source > > correctly? The BIOS vs UEFI partitioning is already covered > > thoroughly in the "Guided storage configuration" table. > > > The currently proposed "Default boot and install" section would stay > > the same, but be renamed to "Default boot" and would just check that > > the image *boots* into the installer in that particular environment. > > I think this is going a bit too far. There are, IIRC, at least two > known places beyond just initial boot where things can go wrong in ways > related to the install medium. > > One is install target selection. anaconda has code for filtering out > 'protected' devices which is intended (in part) to prevent you trying > to install to the USB stick you're running the install from. That code > can go wrong in various ways, including over-aggressive filtering, and > the bugs can be media-specific (e.g. there's no problem in a VM or from > a real optical disc, but it goes wrong when booting from a USB stick). > > The other is use of the on-medium repository. This is mainly for the > Server DVD image. When you install from that image the install should > use the packages from the medium itself as the base repository (and not > go out and get them from the network); I believe we've seen a case > before where the Server DVD written to USB would boot OK, and even > install OK, but it didn't actually use the USB stick as a repository, > it acted like a network install image. When written to a DVD it worked > OK. That was a long time ago, though, and I don't have the bug > reference handy. > > This is why the test case has this text: > > "If the image under test is intended to act as a package source, it > must be the default repository and installation must in fact use the > media as the package source." OK, you're right that there are some cases which could bypass our testing. That's why I assumed this idea might not fly, even though I'd personally have no problem to focus on "reasonable coverage" instead of full coverage. But there might be better approaches to handle this than my "a)" proposal. >From all that you described above, it seems that none of the important features to be tested should be negatively affected if we allow arbitrary partitioning or a different package set (provided the default installation source is used). Or even other, like VNC or basic video mode. What needs to stay fixed is just the install media + environment + install source. Which essentially my "b)" proposal in the original email. That would allow us to be more effective when going through "Default boot and install" table while (hopefully) not sacrificing any test coverage. What do you think about that? > > We *could* reduce the test case to 'boot, select a target disk and > start the installation process if using a DVD image', I guess. Would > this sufficiently address your concerns? That would definitely speed things up. But it feels to me a bit ugly to kill installations during execution as an official test approach. Would the idea written above serve better? It's not that fast and easy (the installations should still finish and you should verify the system boots), but since it can be combined with other test cases, it should not effectively slow us down (too much). > > Of course it's *possible* that some weird bug could show up which made > bootloader configuration not work properly in some media-dependent way, > or something. But I don't recall us running into such a bug before. > > > > > > > At present I'm not proposing any changes to the *release criteria* (we > > > can maybe consider that later). This is just a proposal for what we > > > will actually guarantee testing of via the validation process. > > > > If we go full steam ahead with optical disk testing as proposed (in > > the past, the "default boot and install" test case said you can > > choose either USB or DVD, now we have separate fields for both > > implying we need to test both) I'd suggest optical media being only > > in the Final criteria. > > That was my intent, but it's a bit hard to express with the way we lay > out the wiki tables, unfortunately (hard to specify different > milestones for different environments). I can add an admon note above > the table, I guess, explaining precisely what's actually required for > each milestone. How does that sound? I'd put "Alpha/Final" into the Milestone in the "Default boot and install" table. And the testcase would contain an admon bar linking to a Final criterion saying "installation must work from optical media". That could be obvious enough (at least for us, initiated testers) and follow the same pattern we use for other such mixed-milestone test cases. Wdyt? _______________________________________________ test mailing list -- test@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to test-leave@xxxxxxxxxxxxxxxxxxxxxxx
