On Mon, 2008-06-16 at 21:10 -0400, John Poelstra wrote: > > I think this is a great idea. It sounds like you are focusing > specifically on test plans for these areas? Would these also result > in > test matrices which are filled out before we release and potentially > block the release if broken? You betcha! I'm not looking to go through a deep dive of every application. We'll start by getting a long list on a white board, then begin making some cuts for this first pass. > One thing that I think is still lacking in our overall release > criteria: > http://fedoraproject.org/wiki/QA/ReleaseCriteria is a more detail > surrounding the desktop applications... for example here > http://fedoraproject.org/wiki/QA/ReleaseCriteria#Desktop. I've gone to both extremes with respect to release criteria. What I've found works well is the "keep it simple" model. What we have now is fairly basic, but I might even suggest something more minimal. Perhaps we start by keeping it close to home and in terms of things we all understand ... bugs. Alpha - * All feature tracking bugs filed (and bound to feature wiki pages) * No OPEN bugs with priority >= urgent AND severity >= urgent Beta - * All feature tracking bugs in MODIFIED * No OPEN bugs with priority >= high AND severity >= high PreviewRelease - * No OPEN bugs with priority >= high AND severity >= high >From there, we define some rules on how to assess the impact of a bug and get it on the appropriate radar. My preference would be to defer the decision to individual test leads. There are two really solid definitions of bug severity out there * from Mark Cox ... http://www.redhatmagazine.com/2007/04/18/risk-report-two-years-of-red-hat-enterprise-linux-4 * from bugzilla ... https://bugzilla.redhat.com/page.cgi?id=fields.html#bug_severity Using these definitions, we can > Do we REALLY do complete "click-through" tests for every application? > I > don't think I've ever seen a test matrix created for this nor do I > think > we want to create one. I think it would be a better use of our time > to > create a list of the more often used applications (perhaps via > mugshot) > and add those to a matrix for specific testing. > > Another thing we might want to brainstorm about at FUDCon is creating > a > complete list of test matrices... including CD installs, live images, > etc. that we need for each release. I think we overlooked some of > these > during F9. I think the idea of creating test plans and test cases > beyond just installation testing are a great step forward. > -- ========================================== James Laska -- jlaska@xxxxxxxxxx Quality Engineering -- Red Hat, Inc. ========================================== -- fedora-test-list mailing list fedora-test-list@xxxxxxxxxx To unsubscribe: https://www.redhat.com/mailman/listinfo/fedora-test-list
