On Tue, 2011-11-01 at 13:59 +0000, Matthew Garrett wrote: > It's a common complaint that it's too difficult to get updates to > critpath packages through the update system at the moment. We've been > looking into trying to make that easier without just dropping the > critpath requirements, and one thing we looked at was whether the > requirement for positive karma from proventesters was a net benefit. > > Thankfully this is the kind of thing that we can actually generate > numbers for. Luke pulled some statistics which are available at > http://lists.fedoraproject.org/pipermail/test/2011-October/104084.html . > The relevant section here is the set of packages that have (a) > sufficient positive karma to be pushed, but (b) negative proventester > karma - that is, the packages where negative proventester karma > prevented a push. > > Straight off, we can see that these amount to 1-2% of all critpath > updates. It's simply not common for proventester to make a difference to > the outcome. If we look at the individual packages, things get even more > interesting. Many of the updates receive a mixture of proventester > karma, so even with the negative the push would still go ahead. As far > as I can tell: > > https://admin.fedoraproject.org/updates/xorg-x11-drv-geode-2.11.9-1.fc14 > https://admin.fedoraproject.org/updates/system-setup-keyboard-0.8.6-2.fc14 > > are the only two updates where the proventester karma requirement would > have made a difference, out of 1942 critpath updates that made it to > stable. That doesn't seem like a great hit rate. > > So, assuming I'm not grossly misanalysing the data, it seems that we > could drop the proventester requirement from critical path updates with > a negligable change in the quality of the updates. Thoughts? So, we discussed this at the QA meeting a couple of weeks back. Initially the plan was for anyone who had concerns to reply to this mail, but that doesn't seem to have happened, so I will try to summarize the various responses from that meeting. You can check the full summary and log of the meeting: https://fedoraproject.org/wiki/QA/Meetings/20111107 if you want to make sure I'm not misrepresenting anything. adamw: 1-2% of all critpath updates (where proventester process is seen to have 'had an impact') is a small number, but it is not 0. It only takes *one* really bad update to cause big problems. adamw: it's possible we could use proven tester feedback more effectively: there have been cases where bad updates went out despite negative feedback, because we currently don't give negative feedback - even from proventesters - much significance at all. So there may have been more cases where proventester input *may have been significant* with a system where negative karma is more strongly considered. e.g. updates could be blocked from being auto-pushed if there is any negative feedback from a proven tester red_alert (sandro mathys): critpath packages should have detailed test plans, as currently proventesters often do not know exactly what they need to verify with some critpath packages: "the process as its done right now doesn't work (i.e. is no improvement to not having pts) but having pts with a better process would surely add to the overall quality" tflink: we have to keep in mind the reason for the proposal - maintainers frustrated that packages get stuck in updates-testing for weeks - is a valid reason and a significant problem. "if we object to getting rid of the proventester process, do we have any solutions to the root of their complaints?" brunowolff: how about keeping the PT karma requirement but adopting the plan to allow critpath updates to go through without 'required' karma after a two week wait adamw: "i assume the stats made the assumption that the proventester feedback on any update would still have been present but treated it no different from regular feedback. so, that's not necessarily a safe assumption: proventesters may feel a stronger 'responsibility to test', and if you cancel the process, they might stop doing so. but that's hard to gauge." red_alert: proposes a meeting during FUDCon NA to try and come up with a better pt process That's about all the concrete thoughts / suggestions I can filter out of the log. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora http://www.happyassassin.net -- devel mailing list devel@xxxxxxxxxxxxxxxxxxxxxxx https://admin.fedoraproject.org/mailman/listinfo/devel
