On Sat, Nov 11, 2017 at 4:04 AM, Adam Williamson <adamwill@xxxxxxxxxxxxxxxxx> wrote: > Hi folks! > > So it's been an open question exactly how we would do release > validation for the Modular Server release. For Beta, we wound up trying > to co-ordinate it via mailing list and IRC; I thought doing it this way > would be sufficient and save time trying to set up a parallel stream of > wiki validation events. > > However, I don't think it really turned out that well; we didn't have a > great overview of test coverage at the Beta go/no-go meeting, and just > wound up shipping it more or less on a handwave. And people keep asking > about where they can find and report results for the Modular Server > composes. > > So, I went ahead and spent the last couple of days enhancing all the > release validation bits to handle having release validation events for > Modular composes, in a way which hopefully won't screw anything up. > > The way it *should* work, if I got it all right, is this: > > The fedmsg consumer which automatically creates release validation > events will now also create events for Fedora-Modular 27 composes, > following the exact same rules it follows for creating regular > validation events. I have also created an initial Modular validation > event, for the Beta-1.5 compose we signed off as the Modular Server > Beta. And I have set things up so there's a new set of 'Current' > redirect pages for the Modular events, with 'Current_Modular' in their > name: > > https://fedoraproject.org/wiki/Test_Results:Current_Modular_Summary > https://fedoraproject.org/wiki/Test_Results:Current_Modular_Installation_Test > https://fedoraproject.org/wiki/Test_Results:Current_Modular_Server_Test > https://fedoraproject.org/wiki/Test_Results:Current_Modular_Base_Test > > At present those links will redirect you to the Beta 1.5 pages. You'll > notice the page names follow the same basic scheme as for non-modular > events, just with "Fedora Modular" replacing "Fedora". This similarly > applies to all the category names, so there is now a "Fedora Modular 27 > Beta Test Results" category and so on. > > You may note the contents of the Installation and Base results pages > are a bit different, effectively I've cut them down to the bits that > are relevant to a compose which is intended to deliver only Server > bits. This is achieved by using separate matrix template pages: > > https://fedoraproject.org/wiki/Template:Base_modular_test_matrix > https://fedoraproject.org/wiki/Template:Installation_modular_test_matrix > > I'm usually the only one who edits the matrix templates anyway, but > thought I'd note it just in case. Naturally, if you want to change the > result pages for modular events, you edit the modular template pages, > and vice versa. Server_modular_test_matrix exists but is just a > redirect to Server_test_matrix , as we don't want those tables to > differ at all. For Modular events, only Installation, Server and Base > pages are created; Desktop and Cloud are not needed or relevant. > > The consumer should create a new event for a nightly compose as soon as > one appears that has significant package differences from the Beta 1.5 > compose. Events for modular composes should be announced by email just > as non-modular events are; the text of the email is of course adjusted > slightly to distinguish between modular and non-modular events. > > openQA results should get forwarded to the wiki just as they do for > non-modular composes (I have forwarded the results for Beta 1.5 to the > wiki to test the mechanism, for future composes/events it should happen > automatically). > > All the relval sub-commands now support operating on modular > composes/events. So you can use relval to create modular events > (though, again, normally the fedmsg consumer will do this for us > automatically, no-one needs to do it manually), report results for > modular composes, run the image size checks on modular composes, and do > the user-stats and testcase-stats analysis for modular composes. For > each sub-command, a new argument `--modular` is available which tells > it to operate on a modular compose/event/set of events. For report- > results, you can run just `relval report-results --modular` to report > results for the current Modular validation event. > > You can find testcase_stats for 27 Modular at > https://www.happyassassin.net/testcase_stats/27modular/ . This link > will be included in announcement emails for Modular validation events. > > Updates are available for EPEL 7, Fedora 25, Fedora 26 and Fedora 27 > containing the updated relval, python-wikitcms and fedfind packages. > I've just submitted these, so they should reach updates-testing > shortly. If you want to use relval to report results for modular > composes, you'll want to update to the new versions (fedfind 3.8.0, > python-wikitcms 2.2.0, relval 2.2.0). > > Ultimately, for the Final phase of the Fedora 27 Modular Server > release, we should be able to do release validation just as we do for > regular releases. > > For now I've implemented all of this with two assumptions: we're only > going to have this split 'regular / modular server' schedule for F27, > and the modular composes won't have any bits we care about besides > server bits. If either of those assumptions turns out to be wrong, > we'll have to revisit this again. > > Please do drop me a line if you spot anything wrong/weird/broken- > looking in any of this modular handling. Thanks! Hi Adam, just to provide you with a feedback - it looks great to me. I did a raw check of the matrices, stats etc. and it all seems to be consistent with the way we do regular releases. So, thanks a lot for your work. Regards, Jan > -- > Adam Williamson > Fedora QA Community Monkey > IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net > http://www.happyassassin.net -- Jan Kuřík Platform & Fedora Program Manager Red Hat Czech s.r.o., Purkynova 99/71, 612 45 Brno, Czech Republic _______________________________________________ test mailing list -- test@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to test-leave@xxxxxxxxxxxxxxxxxxxxxxx
