On Wed, 2008-04-02 at 09:06 -0800, Jeff Spaleta wrote: > * Fedora QA is tasked with testing all the release spins. > Nominally being part of that team, I don't think QA has the > resources (ie) enough people involved considering that we have > a six month release cycle and lots of bugs to deal with > > How QA gets the testing done is QA's perview. But the tasking is most > certaintly correctly scoped. > > I will be blunt. I think we've had a significant problem with the QA > process for multiple releases now. We must find a way to organize > volunteer labor better with regard to QA..generally. And here we have a beautifully ironic illustration of the real problem with QA: It took me a day to respond to this email because I was too busy: - running the QA meeting, - triaging the F9Blocker/F9Tracker lists, - retesting bugs that should be fixed, - testing upgrades from F8, - testing fixed builds for problems in rawhide, and - filing bug reports for new problems found. Get it? I'm too busy *doing* QA to *improve* QA. I'm too understaffed to work on staffing up. Chicken, meet egg. Now. If you'd care to expand on *your* thoughts on the problems with the QA Process, I'm all ears. > If QA wants to officially get on the record as saying they aren't > going to take the responsibility of testing the composed images that > RelEng has selected and working on as part of a release cycle... then > by all means get on the record....as a group put your foot down in > protest...so I can come in with my riot gear and tear gas and bust > some skulls. Seriously? That's how we're going to start the discussion? You're gonna bust my head in unless I agree to take on *more* work when the entire problem with QA is that I'm *already* too overworked to do anything? Something is not quite right here. > Or QA as a group try to get involved with the formation of the Spin > SIG and make sure they are well prepared to do a significant amount of > the QA as part of Kickstart Pool management. Getting involved in the Spin SIG: yes. Kickstart Pool management: Also yes. Doing QA for every spin produced: Very no. Here's how it's gonna happen: QA will provide the same plans, guidelines, and tools that we use for Fedora now, and we'll provide advice on using and improving them. But *the spins are responsible for performing their own testing*. QA already provides these things for the standard releases: Release guidelines http://fedoraproject.org/wiki/QA/ReleaseCriteria A detailed installation test plan.. http://fedoraproject.org/wiki/QA/TestPlans/Fedora9Install ..composed of dozens of individual test cases.. http://fedoraproject.org/wiki/QA/TestCases ..along with templates for creating new test cases and test plans. http://fedoraproject.org/wiki/QA/TestCases/TestCaseTemplate http://fedoraproject.org/wiki/QA/TestPlans/TestPlanTemplate And a test summary page where we track our testing results. http://fedoraproject.org/wiki/QA/TestResults/TestSummaryTemplate For each release, we triage bugs according to the ReleaseCriteria, update the installation test cases to reflect new features, execute the installation test cases, and track and report results. The owners of each spin should be responsible for: - Maintaining spin-specific ReleaseCriteria or TestCases, if needed - Tracking spin-specific bugs, - Executing test plans for their spin, and - Tracking the results thereof. Now - I'm not saying they have to do all the same work. The spin owners can definitely *choose* to skip some testing, if they like. And I want to avoid duplicate work - it's perfectly fine to assume that if raid1-root works in the normal Fedora spin, it probably works in the Electronics Lab spin as well. And, really, all I'm asking is that each spin spend some time doing QA on their stuff. It helps the overall test effort by making more official "QA People", spreading the test load over all the spins, and will create new test cases. And hopefully this will drive an effort to improve the tools for managing and tracking test plans and execution. But this is the important thing: THE SPIN OWNERS ARE ULTIMATELY RESPONSIBLE FOR GETTING THEIR SPIN TESTED. You can find your own QA guys, you can do the testing yourself, whatever. But the current QA team is already far, far too busy to actually sit down and test all the bits we produce *now*. We cannot - and will not - test *new* spins that are dropped in our laps, so long as that would take time away from testing Fedora proper. This will probably remain the case until such time as we've managed to build the robust automated testing infrastructure that Fedora deserves. > The Spin SIG may very well find that its in their mutual best interest > in getting involved more directly in the QA process..even before a > spin has been selected for release. And it very well maybe in QA's > best interest to find a way to guide the Spin SIGs involvement in > post-release testing as well. But that's a conversation for the Spin > SIG and QA to have at some point between now and the start of the F10 > release cycle. Oh, definitely. I have a feeling the Spin SIG and QA and the Kickstart Pool will all be very chummy in the near future. Until then, though, we've got a release to work on. -w
Attachment:
signature.asc
Description: This is a digitally signed message part
_______________________________________________ fedora-advisory-board mailing list fedora-advisory-board@xxxxxxxxxx http://www.redhat.com/mailman/listinfo/fedora-advisory-board
