On 1/3/07, Will Woods <wwoods@xxxxxxxxxx> wrote:
On Wed, 2007-01-03 at 13:02 -0500, Greg Dekoenigsberg wrote: > On Wed, 3 Jan 2007, Bill Nottingham wrote: > > >>>> I'm wondering if we could provide solutions for both users: One update > >>>> channel that only gets security updates and important bugfixes while the > >>>> other is a bit more bold -- we for example could have firefox2 in the > >>>> bold channel for FC6 while shipping the latest firefox 1.5.x in the more > >>>> conservative channel. > >>> > >>> So, an idea like this: > >>> > >>> - starts to exponentially expand the QA problem > >> > >> Bill, I love you, but I'm going to have to call bullshit on this one. > >> The answer is not "avoid making QA harder", the answer is to SOLVE THE QA > >> PROBLEM. > > > > Sure, but which of these plans make more sense: > > > > 1) > > > > - Solve the QA problem for our repo configurations as they exist > > - Expand the QA solution to new, multiple, disparate and conflicting > > repositories > > > > 2) > > > > - Expand into new, multiple, disparate and conflicting repositories > > - Then try to solve the QA problem > > > > Honestly, before we can do multiple experimental repositories of this, > > that, and the other, we need to get our *OWN* house in order. > > I agree with this completely -- if, and only if, we actually make a > concerted effort at fixing the QA problem. > > Which means articulating the QA problem, actually. Okay. "The QA Problem" is a big, fat, multi-headed monster. I mentally group QA into four tasks, ordered by priority: Task #1: Testing updates to stable releases Task #2: Testing new releases before they go out (rawhide, TestX, etc.) Task #3: Bug triage (this is the stuff we miss in #1 and #2) Task #4: Writing tools and docs to make the previous tasks easier Examining the first task - we don't currently have the *dedicated* manpower to *guarantee* that every package will be tested and approved by the QA group before it leaves updates-testing and goes into updates. I don't think we actually lack for *available* manpower. AFAIK there's plenty of people willing and able to install packages from updates-testing. The problems here are, I believe, a good summary of "The QA Problem". Problem #1: Testing currently requires a lot of skill, which reduces usable manpower. We lack how-to-test documents, so each tester must know how to set up and test any given package/feature on his own. New features don't necessarily come with much documentation (e.g. iSCSI). - Possible solution(s): More docs would lower the barrier to entry. Setting up an official Fedora QA group will help keep track of team strength and help everyone w uork together.
This would be great. Knowing when you have tested enough, and covered the 'standard' way a package is meant to be used would help cut down on the trying to hit every corner case.
Problem #3: Motivation to report bugs / testresults is low. Even though it's easy to install packages from updates-testing, reporting problems with them is far harder than it needs to be. The same old bugs get reported over and over while some new problems don't get reported because the tester didn't want to spend 15 minutes looking for duplicate bugs, figuring out the appropriate component/version/etc.
Or when you do take the 15 minutes you don't find the duplicate because it was described differently than you would have done so (little solution to that). I sent an intern last month to report a bug he found.. frankly bugzilla scared him when trying to pin something down.. there are 14 states that an existing bug could be in (but only really 3 maybe are used.) Putting stuff for RHEL and Fedora probably causes confusion on the RH side also. The biggest grief for me is when I find a bug and then go to search and find that there are 30+ open NEW bugs on the package all the way back from 1999 or so. This is the immediate "It is obvious that the Developer refuses to use bugzilla or does not care about this package so why post another bug that wont be looked at." I would second the point system.. and I would like to add a wall of shame. If a developer doesnt move a bug from NEW to ASSIGNED (LOOKED AT?) within a week his mug is posted to Mugshot. At the end of the quarter, their fate is voted on by members of the community (will it be the dish rack? the cushions? the COMFY CHAIR? 2 hours of a board meeting?)
- Possible solutions: The updates system should have an easy way to report common problems with packages in updates-testing. A modified bug-buddy for Fedora would be very helpful here. These tools should also show the user commonly-reported bugs, and allow them to easily add a "me too" comment. A "Bugzilla RPG" or other ranking system (like GNOME's point system or Launchpad's Karma ranking) makes bug reporting and triage more interesting. So. Does that define The QA Problem? Or are there other issues I'm forgetting? -w _______________________________________________ fedora-advisory-board mailing list fedora-advisory-board@xxxxxxxxxx http://www.redhat.com/mailman/listinfo/fedora-advisory-board
-- Stephen J Smoogen. -- CSIRT/Linux System Administrator How far that little candle throws his beams! So shines a good deed in a naughty world. = Shakespeare. "The Merchant of Venice" _______________________________________________ fedora-advisory-board mailing list fedora-advisory-board@xxxxxxxxxx http://www.redhat.com/mailman/listinfo/fedora-advisory-board _______________________________________________ fedora-advisory-board-readonly mailing list fedora-advisory-board-readonly@xxxxxxxxxx http://www.redhat.com/mailman/listinfo/fedora-advisory-board-readonly
