On Wed, 2016-01-06 at 09:11 -0500, Kamil Paral wrote: > > Hi folks! So at the meeting this week we agreed to add test cases for > > N-1 upgrades. I've adjusted the templates used to generate the upgrade > > test cases to allow for this, but now I notice there's kind of a naming > > conflict. > > > > All the current upgrade test cases are named upgrade_dnf_previous_foo , > > with 'previous' meaning 'upgrade from the previous release relative to > > the one being tested'. But the FedoraVersion template that's used in > > the upgrade test case templates also uses the term 'previous', only it > > uses it to mean 'previous stable release'. In other words, right now > > for the upgrade test case names 'previous' means 'Fedora 23', but for > > the FedoraVersion template it means 'Fedora 22'. > > > > The way I implemented source release choice in the upgrade test case > > template is to let you set a value that'll be passed through to the > > FedoraVersion template. So we're in the kinda odd position right now > > that our 'previous' test cases pass the value 'current' to > > FedoraVersion, and to add N-1 test cases we'd have to name them > > something like 'upgrade_dnf_previous2_foo' but have them pass the value > > 'previous'. > > > > That seems like it'd be confusing for anyone who came along in a year > > or two and wanted to figure out what was going on, so I'm proposing > > that instead, we rename the existing upgrade tests to follow the > > FedoraVersion template naming. We'd have 'upgrade_dnf_current_foo' test > > cases which test upgrade from current stable to release under test (23 > > -> 24 at present), and 'upgrade_dnf_previous_foo' test cases which > > would test upgrade from previous stable to release under test (22 -> 24 > > at present), and the names of the test cases would match the values > > they use for the template params. > > Honestly, I've always found it confusing to refer to last stable > release as "previous" in the test case name. So I welcome this > change, because I think it makes more sense and is more in line with > our terminology. > > > > > The only slight drawback I can see is that it'd lead to a slightly odd > > effect in testcase_stats: as the *newly added* test cases would have > > the same name as the *current* test cases currently have, > > testcase_stats would group them together - so testcase_stats would show > > a full series of 'upgrade_dnf_previous' results for F24, but in fact > > the first few of those would be F23->F24 upgrade tests, while the rest > > would be F22->F24 upgrade tests. This doesn't really seem like a big > > problem, though, we're very early in the cycle and testcase_stats is > > just an aid for testers, it's OK for it to have this kind of wrinkle. > > I think this is not important at this moment. Hi again! So since no-one objected and kparal approved, I've gone ahead and implemented this change now. The 'previous' test cases now cover upgrades from N-1 (so, F22 at present) and there are 'current' test cases covering upgrades from N (F23 at present). I added warning notices to each set of test cases explaining the rename, and updated the installation test matrix, so the next validation event will include both sets of tests. I'll also send a patch to update openQA's wiki reporting configuration. If anyone notices any issues with the changes, please yell, or if it's simple, just fix it! :) Thanks. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net http://www.happyassassin.net -- test mailing list test@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe: http://lists.fedoraproject.org/admin/lists/test@xxxxxxxxxxxxxxxxxxxxxxx
