On Mon, 2010-03-01 at 11:57 -0500, Tom "spot" Callaway wrote: > On 03/01/2010 11:52 AM, Peter Jones wrote: > > If you think this isn't the right way > > to provide a safety net for package maintainers - what is? > > With the understanding that you're not specifically asking me that > question, I'd say that I'd prefer to first try to automate checks for > the most frequent update issues: > > * Causes broken deps This is in progress, and we have a working (albeit very alpha) test. See the depcheck ticket/milestone: https://fedorahosted.org/autoqa/ticket/111 https://fedorahosted.org/autoqa/milestone/autoqa%20depcheck > * Has ABI/API change (and is a Critical Path package) This should be handled by the current rpmguard test: https://fedorahosted.org/autoqa/wiki/rpmguard since changing the ABI/API should generally change the soname/version, thus changing the package Provides. Furthermore there's plans to actually diff the symbol tables themselves, for extra safety: https://fedorahosted.org/autoqa/ticket/75 > * Breaks clean upgrade path between releases That's an interesting test case, actually. I'm not sure we currently check packages against the corresponding versions *other* releases. I've added ticket 123 to track this: https://fedorahosted.org/autoqa/ticket/123 > * Fails to pass any package specific sanity tests (as written by either > the maintainer, QA, rel-eng, or qualified contributors) Handling package-specific tests is definitely a goal of AutoQA, and allowing maintainers to add tests to their packages is something I'm trying to keep in mind for the dist-git planning. But this is a big goal and it'll come after the previous three. > AutoQA has the potential to do this. I'd rather see energy and effort > spent on taking out these low hanging fruit. If, after that, we're still > having broken updates pushed directly to stable, then I'd be willing to > consider a policy with an enforced delay in "testing". I'm all for extra energy and effort toward getting AutoQA up and running, but automated testing does not obviate the need for manual testing. It just changes which tests need to be performed manually. Let's be clear about the current state of the test process, and its future: 1) We have some unspoken minimum standards of quality. 2) We are not meeting those standards. 3) Meeting those standards currently requires manual testing. 4) We can automate most tests required to meet those standards. 5) We would like to raise our standards of quality. Meeting those new standards would require new tests, which have not yet been automated. 6 GOTO 3. It should be obvious that the process by which we improve the quality of the distribution will always require some amount of manual testing. So I think it would be shortsighted for FESCo to refuse to even discuss a policy about what manual testing is currently required, since any plan for improving the quality of the distribution will always require some amount of manual testing. -w -- devel mailing list devel@xxxxxxxxxxxxxxxxxxxxxxx https://admin.fedoraproject.org/mailman/listinfo/devel
