On 05/11/2010 07:30 PM, Jeff Spaleta wrote: > On Tue, May 11, 2010 at 3:10 PM, Przemek Klosowski > <przemek.klosowski@xxxxxxxx> wrote: >> This probably means at least a rudimentary application testing rig >> and a discipline that identifies and deals with distressed packages. > > Does the ongoing work with AutoQA provide the solution you are looking for? > > http://fedoraproject.org/wiki/AutoQA Yes, it looks like the right approach, and reasonable infrastructure, but it needs a lot of work on tests, and on the policy of integrating the tests with the packaging and release process. Currently the tests seem to only address the packaging issues; tallying the tests from https://fedorahosted.org/pipermail/autoqa-results/2010-May/thread.html gives this distribution: 23 rats_install 45 rats_sanity 98 conflicts: 171 repoclosure: 673 rpmguard: 799 rpmlint: We need a lot of application-specific tests, which at least presently would have to be hand-rolled. A reasonable future approach might be the regression testing, i.e. tests generated out of bug reports. In principle, the bugzilla informal query on how to replicate the bug could be broken out and formalized to a point where the data might be usable to a testing harness. The policy question is pretty complicated and multifaceted: * errors have to be classified as to severity/impact; * each failure requires a decision by the package manager on what is the appropriate action, and who should be responsible (upstream? manager?); * such decisions should probably be remembered from test run to test run to mask test noise, but probably should be reviewed periodically * what to do if the maintainer or upstream are not responsive It's a large and complicated task; even carefully maintained projects have test errors, and in fact such projects tend to have more extensive test suites so they may end up being noisier at least initially. -- devel mailing list devel@xxxxxxxxxxxxxxxxxxxxxxx https://admin.fedoraproject.org/mailman/listinfo/devel
