Re: Installation validation test change proposal: merge USB tests into 'default boot and install', add more environment columns

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Fri, Oct 7, 2016 at 2:08 PM, Adam Williamson
<adamwill@xxxxxxxxxxxxxxxxx> wrote:
> Hi folks! Sending this to devel@ as well as test@ as there's been some
> relevant discussion there recently. We've been kicking around a couple
> of issues lately:
>
> 1. Exactly what do we need to test and block on, in terms of writing
> images to USB sticks?
>
> 2. 'Default boot and install' table was supposed to require bare metal
> optical media testing at release time, but this is not clear, and
> openQA fills in results despite not testing real media on bare metal.
>
> So here's my proposal to address both topics. It involves changes to
> the Boot_default_install test:
>
> https://fedoraproject.org/wiki/User:Adamwill/Draft_Testcase_Boot_default_install
> https://fedoraproject.org/w/index.php?title=User:Adamwill/Draft_Testcase_Boot_default_install&diff=476645&oldid=476644
>
> and the Installation matrix template page:
>
> https://fedoraproject.org/wiki/User:Adamwill/Draft_Installation_test_matrix
> https://fedoraproject.org/w/index.php?title=User:Adamwill/Draft_Installation_test_matrix&diff=476646&oldid=476643
>
> to summarize the changes - we improve the test case's wording to give
> explicit instructions for all three media types we care about (virtual
> disc attached to a VM, real optical disc in a real machine, real USB
> stick in a real machine), we ditch the USB test matrices entirely, and
> we give the 'Default boot and install' tables extra result columns so
> they can now reflect results for VM, CD/DVD and USB testing (for BIOS
> and UEFI in all three cases, for x86_64).
>
> I didn't get too finicky about the wording for the virt case, as that's
> basically there to get filled in by openQA anyway. For the CD/DVD case,
> the test case instructs you to write the disc according to the
> 'official instructions', with a URL that is supposed to be always
> provided by the docs project for that purpose. For the USB case, the
> test case instructs you to write the stick according to
> https://fedoraproject.org/wiki/How_to_create_and_use_Live_USB , telling
> you to use the 'most prominently-recommended method'; the intent being
> that, as things stand, we would only block on Fedora Media Writer
> issues. I didn't explicitly cover dd (e.g. with separate FMW and dd
> columns) for two reasons:
>
> 1. It should pretty reliably be the case that if FMW works, dd works
> 2. Some people are going to test plain dd anyway, it's just such a
> convenient 'expert' method that I can't imagine we won't have people
> using it throughout the test period
>
> I didn't cover livecd-iso-to-disk because it seems like, overall, the
> feedback to cmurf's "F26 proposal: Make Fedora Media Writer the
> officially supported USB install media creator" was fairly positive and
> I think justifies not guaranteeing testing for it.
>
> To be clear, this is a trade-off between 'in an ideal world we'd test
> everything' and 'but we really don't have time'. Each additional method
> adds 12 mandatory tests (6 images, both UEFI and BIOS variants of the
> test). We can discount the virt tests, as openQA will do those, so in
> the proposal, there are 24 mandatory tests for humans; if we added dd
> as its own thing there'd be 36, if we added litd as well there'd be 48.
>
> One note is that this doesn't do much to ensure that FMW is working at
> release time in all expected environments (probably at least the two
> previous stable Fedora releases, macOS, and Windows). On the other
> hand, neither does the current matrix; we explicitly list FMW, but we
> don't specify what environments it should be run in. We could consider
> adding a USB table with a slightly different focus, the idea being to
> test the *writing tool* rather than test that the *image* can be
> written and works properly. We could make that a follow-up proposal. If
> we had such a table, it could maybe cover testing livecd-iso-to-disk's
> advanced features as optional tests, or something.
>
> At present I'm not proposing any changes to the *release criteria* (we
> can maybe consider that later). This is just a proposal for what we
> will actually guarantee testing of via the validation process.


Looks and sounds good to me.

-- 
Chris Murphy
_______________________________________________
test mailing list -- test@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to test-leave@xxxxxxxxxxxxxxxxxxxxxxx




[Index of Archives]     [Fedora Desktop]     [Fedora SELinux]     [Photo Sharing]     [Yosemite Forum]     [KDE Users]

  Powered by Linux