On Tue, 2011-11-22 at 14:06 +0100, Nils Philippsen wrote: > On Mon, 2011-11-21 at 10:32 -0800, Adam Williamson wrote: > > > red_alert (sandro mathys): critpath packages should have detailed test plans > > Hm. The list of (implicitly labeled) critpath packages seems to have > proliferated recently: a few days ago I submitted an update for > sane-backends and then found out that it's on critpath, probably by way > of critical-path-gnome -> control-center -> colord -> > sane-backends-libs. On the one hand it would probably be a good idea to > notify maintainers of such packages being implicitly pulled in (in this > case when BR: colord-devel was added to control-center). On the other > hand, sane-backends is a particularly nasty case and a detailed test > plan would probably start with this: > > 0. Have a lot of scanners handy, or at least models affected by the > changes. > > Not even I get past that point in most cases: I have one USB scanner in > the office, and two rather ancient SCSI models at home. The only thing I > can meaningfully test is these models/their drivers and the > HW-independent parts of the package, and I don't expect more from the > testers. With the package being critpath now, I fear that updates for > sane-backends might never see the light of day, depending on the HW > affected, unless Bruno's proposal (two weeks of wait for critpath > without needed and no negative karma) or enough (proven)testers are > content with only cursory testing :-/. > > Don't get me wrong: I'd really like sane-backends to get as much testing > as possible before turning updates loose on the unsuspecting public. I > just don't see how it can be done right at the moment. This touches on a couple of issues. I know part of FESCo's response to the proventesters question was to look at ways to slim down critpath. It also hits the question of things which are critpath indirectly (via dependencies). Part of the idea of having a test plan for all components identified as critpath is to help identify those which are indirect critpath packages, and make it clear what needs to be the case for those packages to pass critpath validation - usually only that they don't cause the actual critical path function to fail. In the sane_backends case, for instance, in order to validate the critical path, you only have to check the update doesn't somehow stop GNOME (and possibly control-center) running. You don't have to make sure it doesn't break scanning. Providing a test plan is actually a good way to do this, if you take a look at https://fedoraproject.org/wiki/User:Adamwill/Draft_QA_SOP_package_test_plan_creation and https://fedoraproject.org/wiki/QA:SOP_test_case_creation : with the scheme explained there for categorizing test cases, you can both associate test cases with packages, and mark them as critical path or not. So the 'intended' way to do things in this case would be to write a bunch of test cases that actually test sane-backends, and put them in the 'Category:Package_sane-backends_test_cases' category - but *not* to put them in the 'Category:Critical_path_test_cases' category. You would also add an existing test case for basic GNOME functionality - say, https://fedoraproject.org/wiki/QA:Testcase_base_startup - to the 'Category:Package_sane-backends_test_cases' category, and make sure it was in the 'Category:Critical_path_test_cases' category too. The result of that would be that, when a sane-backends update appeared, the Bodhi page for it would list all the sane-backends test cases and also the 'base_startup' test case, and identify the 'base_startup' test case as being a critical path test case - thus indicating that the way to really validate whether sane-backends *works* is to do the detailed, specific tests, but the way to validate whether it breaks the critical path is just to see if the system boots to a working desktop. Then you'd need Bodhi 2.0 so you'd be able to provide feedback of "This update doesn't break critical path" or "This update actually works for scanning", rather than simply numeric feedback, as is the case now. This may be a bit complex, I admit, and we haven't really completed the 'intended design' for many packages to show how it's supposed to work. (That 'base_startup' test case should be associated with just about every critpath package in existence, for instance, and it should be in the critical path category too, which it currently isn't). But all the bits are more or less there already, and just need hooking up. And we need Bodhi 2.0 (yes, I know that's my catchphrase). -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora http://www.happyassassin.net -- devel mailing list devel@xxxxxxxxxxxxxxxxxxxxxxx https://admin.fedoraproject.org/mailman/listinfo/devel
