On Tue, 2016-10-18 at 07:54 -0400, Kamil Paral wrote: <snip> > OK, you're right that there are some cases which could bypass our > testing. That's why I assumed this idea might not fly, even though > I'd personally have no problem to focus on "reasonable coverage" > instead of full coverage. But there might be better approaches to > handle this than my "a)" proposal. > > From all that you described above, it seems that none of the > important features to be tested should be negatively affected if we > allow arbitrary partitioning or a different package set (provided the > default installation source is used). Or even other, like VNC or > basic video mode. What needs to stay fixed is just the install media > + environment + install source. Which essentially my "b)" proposal in > the original email. That would allow us to be more effective when > going through "Default boot and install" table while (hopefully) not > sacrificing any test coverage. What do you think about that? <snip> > That would definitely speed things up. But it feels to me a bit ugly > to kill installations during execution as an official test approach. > Would the idea written above serve better? It's not that fast and > easy (the installations should still finish and you should verify the > system boots), but since it can be combined with other test cases, it > should not effectively slow us down (too much). <snip> So after some further consideration I think I'm on board with this, and it turned out not to be too hard to implement. I have updated the draft test case: https://fedoraproject.org/wiki/User:Adamwill/Draft_Testcase_Boot_default_install so it now basically says 'you can run any kind of install that boots from the medium under test and uses the default repo, and if it satisfies the Expected Results, we're good'. Does this look better to you now? > > > If we go full steam ahead with optical disk testing as proposed (in > > > the past, the "default boot and install" test case said you can > > > choose either USB or DVD, now we have separate fields for both > > > implying we need to test both) I'd suggest optical media being only > > > in the Final criteria. > > > > That was my intent, but it's a bit hard to express with the way we lay > > out the wiki tables, unfortunately (hard to specify different > > milestones for different environments). I can add an admon note above > > the table, I guess, explaining precisely what's actually required for > > each milestone. How does that sound? > > I'd put "Alpha/Final" into the Milestone in the "Default boot and > install" table. And the testcase would contain an admon bar linking > to a Final criterion saying "installation must work from optical > media". That could be obvious enough (at least for us, initiated > testers) and follow the same pattern we use for other such mixed- > milestone test cases. Wdyt? I think this is going a bit too far. I'm OK with not requiring complete coverage at Alpha or Beta, but I think explicitly moving optical boot to the Final criteria is too much. I think if we do find out that optical boot is broken for a release-blocking medium at Alpha or Beta stage that should still be a blocker. At least for now, until the project as a whole decides we don't care about optical media any more. So I went with a halfway house. I've updated the matrix draft too: https://fedoraproject.org/wiki/User:Adamwill/Draft_Installation_test_matrix so it now lists the milestone as 'Alpha / Final' and has an admon that explicitly explains the 'Expected coverage' for the section, saying we expect a reasonable sample of tests for Alpha and Beta, and full coverage for Final. Do you think that's sufficient? Note that with the current wikitcms code, testcase_stats considers the milestone for 'Alpha / Final' rows to be 'Final'...because it works like this: for mile in ('Alpha', 'Beta', 'Final', 'Optional', 'Tier1', 'Tier2', 'Tier3'): if mile in cells[0]: milestone = mile obviously that would be very trivial to tweak, I just thought I'd mention it. I might tweak it to prefer the first milestone it finds rather than the last, I'll just have to look and see if that would cause any unexpected effects in other cases (I don't think we have many, though). -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net http://www.happyassassin.net _______________________________________________ test mailing list -- test@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to test-leave@xxxxxxxxxxxxxxxxxxxxxxx
