On Tue, 2016-07-12 at 23:57 +0530, Siddhesh Poyarekar wrote: > On Tue, Jul 12, 2016 at 09:49:14AM -0700, Adam Williamson wrote: > > > > To be clear, the idea would be to have general-purpose instructions for > > basic functionality testing of each package, not requiring new 'how to > > test' text to be written for every individual package update, > > specifically tailored to the changes in that update. > > It is a good idea in general, but lack of such instructions is not an > excuse for not performing the one implicit set of test cases in most > updates: verifying fixes to bug reports attached to the update. If a > tester does not know how to do that, then they should not be giving > karma at all, regardless of whether or not there is documentation on > sanity tests for the package. This isn't really correct, because there is no simple relationship between 'bugs claimed to be fixed actually are fixed' and 'update should be released'. Both of these are possible: 1) an update which fixes the bugs it claims to fix, but should *NOT* be released 2) an update which does not fix all the bugs it claims to fix, but *SHOULD* be released An example of 1) is an update which claims to fix a minor bug, and does, but creates a *major* bug. e.g., fixes a typo in the package description, but causes the app not to run at all. This update should be given -1 karma (negative response to 'Is the update generally functional?'), not +1. An example of 2) is an update which claims to provides a critical security fix and a trivial bug fix, and *does* fix the security issue, but the trivial bug fix doesn't work. This update should be given +1 karma (positive response to 'Is the update generally functional?'), not -1. This is in fact *why*, in Bodhi 2.0, there are separate feedback entries for each individual bug listed by the update - so testers can separate 'does or does not fix bug X' feedback from 'update generally works' feedback. It is possible (in my opinion) for a tester to reasonably provide 'Is the update generally functional?' feedback without actually checking all or even any of the claimed bug fixes, and I've done this myself quite often. It can quite often be difficult (if the bug is in a very complex use case) or impossible (if the bug is specific to e.g. a piece of hardware you don't own) to check bug fixes. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net http://www.happyassassin.net -- test mailing list test@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe: https://lists.fedoraproject.org/admin/lists/test@xxxxxxxxxxxxxxxxxxxxxxx
