On Fri, 2014-11-07 at 13:02 -0500, Kamil Paral wrote: > I have a proposal regarding installation matrix [1] related to > environments. Adam did a truly stellar job in his latest results table > restructuring, he eliminated a lot of duplicated testing and > simplified the tables a lot (for example, there's no point in testing > additional HTTP repos in anaconda on all three major architectures, > because that functionality it architecture independent). We don't have > limitless resources and must keep focused on areas where it's likely > for a bug to occur, instead of testing everything. Here's a proposal > of some last touches to installation tables: > > 1. Testcase_Anaconda_User_Interface_Basic_Video_Driver was changed to > a generic x86 test. In this case, it is problematic, because BIOS and > UEFI fallback video driver are completely different code paths. I have > a personal experience from that past when one of those worked and the > other didn't. We should distinguish BIOS and UEFI in this case. To fix > this, I see two viable options: > a) move the test case to Miscellaneous section, so that the table in > User interface section doesn't get more complicated > b) keep the test case in User interface section, but create a separate > table for it (right under the first table). This new table would have > x86 BIOS and x86 UEFI columns. Again, the purpose is to keep the > original table simple (avoid adding columns which are not required for > anything else). The tests in that table are really covering somewhat different things, now I look at them. Graphical, and Text actually exercise two of anaconda's UIs (we seem to be missing a test for cmdline, unless one of the other test cases hits it in passing). The others are different ways of accessing those same UIs: the VNC tests are different ways to get at graphical, and serial_console is another way to get at text. It makes sense for Graphical and Text to not be split between i686, x86_64 and UEFI, for sure. And it makes sense for Basic Video Driver to have probably x86 BIOS / x86 UEFI split, indeed. We should consider the other tests in deciding the right thing to do - what's the appropriate split for VNC and serial console? > 2. Testcase_Install_to_Previous_KVM and > Testcase_Install_to_Current_KVM differentiate BIOS and UEFI. I always > forget whether this is about the host or the guest (or both, or which > combination). But I think host differentiation doesn't make any > difference - libvirt should work the same, I assume. It makes sense to > differentiate the guests. Unfortunately I believe the UEFI mode for > libvirt/kvm [2] is not available under an open source license, > therefore it's not packaged and not trivial to use. Which means not > that many people probably use it. I wonder whether we really need to > keep this separated. I would gladly merge this together to a single > x86 column and split up once OVMF is available under a better license. > If we keep it separate, we should provide better instructions for the > UEFI part. Well, the licensing issue is a bit subtle. OVMF itself is freely licensed (BSD or MIT IIRC). The bit that has a license issue is the FAT driver used by EDK II, on which OVMF is built. It has this extra term in its license: Additional terms: In addition to the forgoing, redistribution and use of the code is conditioned upon the FAT 32 File System Driver and all derivative works thereof being used for and designed only to read and/or write to a file system that is directly managed by Intel's Extensible Firmware Initiative (EFI) Specification v. 1.0 and later and/or the Unified Extensible Firmware Interface (UEFI) Forum's UEFI Specifications v.2.0 and later (together the "UEFI Specifications"); only as necessary to emulate an implementation of the UEFI Specifications; and to create firmware, applications, utilities and/or drivers. which obviously makes it non-free. I believe the *reason* is something to do with Microsoft patent(s) believed to be related to FAT, and I don't think it's prudent to say anything more specific than that. There therefore isn't a legal issue with redistributing OVMF builds, and there is a commonly-used repository which contains them, updated daily: https://www.kraxel.org/repos/ they just can't go into Fedora, because of Fedora's policies about non-freely-licensed software. It's pretty easy to set up a UEFI KVM using the packages from that repo, I have one here and just spent all morning using it to test UEFI multi-boot. We certainly *can* block Fedora releases on things that have to do with software that's not part of Fedora's repositories, if we want to - viz the Windows or OS X boot criteria, for e.g. It'd be interesting to see, if a blocker was proposed, if we actually want to block on UEFI KVM guest operation, but I think at least it's worth having it in the matrix. > PS: I know nothing about Xen, which is also in the table, so I have no > idea whether it's a similar story or not. We have a commitment from Oracle to provide the testing for that criterion - basically we agreed to have the criterion so long as they make sure the testing gets done. Konrad Wilk has been doing that testing ever since we added the criterion, and he usually does it promptly and thoroughly, so we haven't really had any trouble with it. > 3. Testcase_Anaconda_save_traceback_to_bugzilla, > Testcase_Anaconda_updates.img_via_URL, > Testcase_Anaconda_user_creation, > Testcase_Anaconda_updates.img_via_installation_source and > Testcase_Anaconda_traceback_debug_mode all differentiate BIOS and > UEFI. I think that's a waste of our energy. None of these testcases > should IMO be affected by firmware. We should make them generic x86 > tests. We could go even further and make them completely > arch-independent (at least some of them), but I'm a bit careful around > ARM, so not completely sure about that. ACK. Honestly I think this is just because they used to be in the miscellaneous tests table (some of them still are, I think) and that thing was freaking huge, and I kinda ran out of steam when I was doing the revision and just left some of 'em alone). It's probably worth noting the meaning of a box being available or grey isn't *entirely* solidly nailed down. It's never been strictly the case that we require every white box for the appropriate release level to be filled - back when all we had was i386 and x86_64, we just had a box for each for every test, and it was kind of a case of 'fill in as many as we can'. Since we started getting ARM and UEFI it's got a bit more complex and I think to an extent we're starting to think 'if a box is there it really needs to be filled'. In practice when creating and revising the tables (as the person who does it most of the time) I tend to be slightly fuzzy between 'the boxes should express the range of all environments for which we really need this test to happen' or 'the boxes should express the range of environments in which it's *possible* to do this test, even if we don't really need to do it in every one of them'. I think we've been trending towards the former interpretation, though, and that's probably not a bad idea, for the sake of clarity and reducing the sheer weight of tests. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net http://www.happyassassin.net -- test mailing list test@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
