On Wed, 2017-04-05 at 15:38 -0700, Adam Williamson wrote: > Hi folks! Time for another proposal :) > > We currently kinda bundle the checks for correct artwork and 'release > identification' (stuff that identifies the release, like 'Welcome to > Fedora 26!' etc.) into two test cases that also include functionality > testing: > > https://fedoraproject.org/wiki/QA:Testcase_base_startup > https://fedoraproject.org/wiki/QA:Testcase_base_initial_setup > > This is a bit awkward, and also is a bit of a problem for automation. > We could actually automate the functional parts of those tests in > openQA (I've just finished adding openQA tests that install without a > user account and check gnome-initial-setup and initial-setup work OK), > but we can't really automate the 'release identification' / artwork > checks. So we can't have openQA report wiki passes for these tests, so > long as they're combined like this. > > So I'm proposing we separate out a new 'artwork and release > identification' test case. I've written a draft of it: > > https://fedoraproject.org/wiki/User:Adamwill/Draft_Testcase_artwork_release_identification > > We would remove the relevant bits from the other two test cases. > > I'm a bit in two minds about what to do with the matrices. I'm thinking > we could either add this test case to the Server, Cloud and Desktop > pages, or we could put it in the Base page (as the current tests are). > Looking at this, it occurs to me that not just these checks but most of > the other Base tests would actually be relevant to things beyond the > 'environments' currently in the Base page (Workstation, Server, KDE, > ARM and Cloud). I'm thinking perhaps we could rejig the page a bit, and > add another table with non-blocking columns for other desktops and > stuff. > > Thoughts? Thanks! Thanks for the feedback so far. So I've moved the test case itself to a 'production' name: https://fedoraproject.org/wiki/QA:Testcase_base_artwork_release_identification and created a draft of a revised Base matrix: https://fedoraproject.org/wiki/User:Adamwill/Draft_Base_test_matrix It looks a lot bigger, but don't be scared! Two of the tables are completely optional. The ARM table is really just the old 'ARM' column from the single table split out into its own table and with a split between the two ARM release-blocking environments, Xfce and Minimal - it seems reasonable that we could have different results for those, so an 'ARM' column really didn't seem to make much sense. I added the new test to each table, with the milestone set to 'Alpha / Final'. What do people think of this (especially ARM folks - paging pwhalen)? Does it look reasonable? Thanks! If this draft looks OK, I'll put it into production soon, and remove the artwork-y bits from the startup and initial_setup tests. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net http://www.happyassassin.net _______________________________________________ test mailing list -- test@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to test-leave@xxxxxxxxxxxxxxxxxxxxxxx
