Hi folks! So at the meeting this week we agreed to add test cases for N-1 upgrades. I've adjusted the templates used to generate the upgrade test cases to allow for this, but now I notice there's kind of a naming conflict. All the current upgrade test cases are named upgrade_dnf_previous_foo , with 'previous' meaning 'upgrade from the previous release relative to the one being tested'. But the FedoraVersion template that's used in the upgrade test case templates also uses the term 'previous', only it uses it to mean 'previous stable release'. In other words, right now for the upgrade test case names 'previous' means 'Fedora 23', but for the FedoraVersion template it means 'Fedora 22'. The way I implemented source release choice in the upgrade test case template is to let you set a value that'll be passed through to the FedoraVersion template. So we're in the kinda odd position right now that our 'previous' test cases pass the value 'current' to FedoraVersion, and to add N-1 test cases we'd have to name them something like 'upgrade_dnf_previous2_foo' but have them pass the value 'previous'. That seems like it'd be confusing for anyone who came along in a year or two and wanted to figure out what was going on, so I'm proposing that instead, we rename the existing upgrade tests to follow the FedoraVersion template naming. We'd have 'upgrade_dnf_current_foo' test cases which test upgrade from current stable to release under test (23 -> 24 at present), and 'upgrade_dnf_previous_foo' test cases which would test upgrade from previous stable to release under test (22 -> 24 at present), and the names of the test cases would match the values they use for the template params. The only slight drawback I can see is that it'd lead to a slightly odd effect in testcase_stats: as the *newly added* test cases would have the same name as the *current* test cases currently have, testcase_stats would group them together - so testcase_stats would show a full series of 'upgrade_dnf_previous' results for F24, but in fact the first few of those would be F23->F24 upgrade tests, while the rest would be F22->F24 upgrade tests. This doesn't really seem like a big problem, though, we're very early in the cycle and testcase_stats is just an aid for testers, it's OK for it to have this kind of wrinkle. So does anyone have any objections to changing things so we have 'upgrade_dnf_current_foo' test cases which are for F23->F24, and 'upgrade_dnf_previous_foo' test cases which are for F22->F24? If no-one sees any issues I'll go ahead and do that by the end of the week, and make appropriate adjustments to the matrices and the openQA wiki reporting code. As discussed at the meeting, the N-1 test cases would be marked 'Optional' for now, as we want to talk to FPC about packaging guidelines for N-1 upgrades before we make them part of the release criteria. Thanks folks! -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net http://www.happyassassin.net -- test mailing list test@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe: http://lists.fedoraproject.org/admin/lists/test@xxxxxxxxxxxxxxxxxxxxxxx
