On Sun, 2016-07-10 at 21:30 +0530, Sayan Chowdhury wrote: > Hi, > > I recently packaged and pushed an update for fedmsg-meta-fedora-infrastructure > to bodhi and exactly 40 secs[1] later I got a +1 to the update. I am sure that > testing a package surely takes more than 40 secs. This makes me really curious > that are the packages really being tested before giving out the karma. > > After going through messages in datagrepper[2][3], I found that few people are > giving out karma in one go (4-5 packages under a minute). If these packages > really are not-tested and the karma are given out randomly then I am sure that > this sure going to affect the release, infrastructure and our users. > > Does anybody know what is going on? So I've been discussing this with various people in the last few days, and one specific idea has come out of that which I'd like to float. We've been hesitant to suggest this before as we thought packagers might not like the idea, but we figured it can't hurt to at least suggest it. The idea is this: there could be a requirement for all packages to provide at least *some* kind of 'how to test' information. Looked at from the perspective of a new tester, the current system is quite difficult to handle when it comes to upgrades of packages which aren't obviously part of the critical path (e.g. kernel) or a well- known GUI application package (e.g. firefox). How do we *expect* a new tester to respond when they come to an update for fedmsg-meta-fedora-infrastructure , in Bodhi or fedora-easy-karma? It's very difficult - probably impossible - for them to know or work out what they should actually do to test this package. Of course, writing instructions for every single package is a lot of work, but right now we have test cases for very few packages, and it would definitely help if we could significantly increase that number. What do people think about this idea? To be clear, the idea would be to have general-purpose instructions for basic functionality testing of each package, not requiring new 'how to test' text to be written for every individual package update, specifically tailored to the changes in that update. The way the system works at present is that Bodhi will show all wiki pages in a specific category based on the package name. For package 'foo', Bodhi's web interface and fedora-easy-karma will list all wiki pages in the category "Category:Package_foo_test_cases" . These are expected to be typical 'test case' pages, though there isn't actually any technical *requirement* for them to be, or enforcement of that. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net http://www.happyassassin.net -- test mailing list test@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe: https://lists.fedoraproject.org/admin/lists/test@xxxxxxxxxxxxxxxxxxxxxxx
