On Wed, 2010-06-09 at 01:46 +0200, Kevin Kofler wrote: > Luke Macken wrote: > > This report definitely conveys the shortcomings in our testing, however, > > it does show us improving with each release. By 'shortcomings in our testing', I mean, 'shortcomings in the process by which we currently use bodhi for "testing" updates'. > For Fedora 13, we implemented > > the No Frozen Rawhide process with improved Critical Path policies, which > > were definitely a success. With these enhanced procedures, along with the > > upcoming implementation of AutoQA and the new Package update acceptance > > criteria > > (https://fedoraproject.org/wiki/Package_update_acceptance_criteria), I > > think we'll see these numbers drastically improve in the future. > > Only because those numbers are taylored towards that very process (they > measure the exact same things that process is going to enforce) and do not > reflect the actual quality of the packages in any way. > > You can make really anything a "success" by measuring the very symptoms of > the process and calling them a metric of "quality". These metrics do not reflect the actual quality of our updates, the policy or process behind them, or even the quality of testing that they endured before being released. It is simply a measurement of how we have been interacting with this specific piece of infrastructure. By "success" I mean that I felt we were successful in drafting, implementing, deploying, and utilizing the mentioned policies as expected, and the results show increased community engagement. Attempting to quantify the quality of these community interactions is another story. As to whether or not the policies behind the processes have helped or hindered the release quality overall, that has yet to be determined. > The reasons for which Bodhi karma (especially in its current incarnation) is > a completely broken indicator of quality have been pointed out in several > past threads. I'm well aware, and I agree that the current bodhi karma implementation does not convey an effective measurement of update quality. The problem is not a lack of good ideas for improvement, but rather a lack of developers to help improve. luke -- devel mailing list devel@xxxxxxxxxxxxxxxxxxxxxxx https://admin.fedoraproject.org/mailman/listinfo/devel
