Hear ye, hear ye! There will be a special meeting of the Fedora BugZappers in our usual meeting slot, Wednesday 2008-03-12 17:00UTC in #fedora-meeting on freenode. The purpose of this meeting is to solicit input on proposals for dealing with the current unmanageable backlog of bugs. In the long term, this backlog will cause Fedora irreparable harm, if it has not already. Our most valuable asset, the bug reporter, is feeling left out in the cold. Community triagers feel discouraged by what they see as a insurmountable task, thereby making the problem feed on itself. We have to act, and the time to do it is now. To that end, I am proud to present two proposals, One has to do with dealing with the backlog that we have now, and the other has to do with making sure we never get into this situation again -- ever. We believe that these proposals are the right thing to do, and now is the right time to do them, right before a release. I'd also like to give credit where it's due for these proposals. The primary author of both of them is John Poelstra, without whom many of the things that have been accomplished to date would not have been. In just a few short months, we've gone from having almost nothing formalized to having a formal bug workflow, and having formal plans of dealing with the backlog, both now and in the future. It's important to note that these are PROPOSALS at the current time, and have not been approved in any way. I am expecting, and welcome, impassioned debate on these proposals. In the course of these debates, however, lets be civil with one another - we're all responsible adults here. If you have comments or concerns about either of the proposals, there's a comments section at the bottom of both of them on the wiki, which I encourage you to use. Also, please come to the meeting and/or reply to this e-mail if you have comments. Please try to back up changes to the proposals with a specific example of where the proposed action is wrong/bad/whatever. Also note that the framework of the proposals is there. The exact text to be found in bugs is still yet to be written, however that will be written in the next week or so. The text of both of the proposals follows. I realize that both this introduction and the proposals themselves are extremely long, but please read them! The wiki versions of these proposals can be found at: http://fedoraproject.org/wiki/BugZappers/HouseKeeping http://fedoraproject.org/wiki/JohnPoelstra/BugzillaExtremeMakeOver --- Housekeeping - the go-forward plan --- ||<tableclass="message notice">This page is under construction. It will hold our standard operating procecedures (SOP) for what needs to be done at different times during the release cycle starting with Fedora 9 Development. March 2008|| = Bugzilla Maintenance SOP = ||<tableclass="message warning">This is a draft of the proposed approach. It will be circulated for review in Fedora, including FESCo, prior to execution (2008-03-10)|| [[TableOfContents]] == Background == The following page outlines the process for managing open Fedora bugs during the release process. We intend to proactively manage unresolved bugzillas in a systematic orderly way which will introduce predictability to the bugzilla maintenance cycle. We also do not want to disenfranchise one of our most valuable community resources: bug reporters. Historically we have a had a lot of stale open bugs. In order to remain focused as a project it is best to close out bugs we aren't going to be able to resolve so that we focus on the bugs we will. And if you think this is too aggressive you're welcome to keep your bugs alive by continually moving them to the latest version. == Bugzilla Product Versioning == * Fedora will track bugs solely based on the version number of the release or '''rawhide''': the release under development which has not been released. * Fedora will '''not''' create separate ''fedora versions'' in bugzilla for individual test releases, including, but not limited to: alpha, beta, preview release, etc. Fedora has tried this structure in the past, but found it difficult to maintain and its benefit imperceptible. * Changes to this policy require review and approval by FESCo. == Tracker Bugs == * Fedora creates a series of tracker bugs at the beginning of each new release cycle (1 day after GA of the previous release). The official tracker bugs are: 1. Alpha 1. Beta 1. Preview Release 1. GA * A list of tracker bugs is maintained at this page: [https://bugzilla.redhat.com/buglist.cgi?query_format=advanced&short_desc_type=allwordssubstr&short_desc=&product=Fedora&version=&component=&query_format=advanced&bug_status=NEW&bug_status=ASSIGNED&bug_status=NEEDINFO&bug_status=MODIFIED&bug_status=ON_DEV&bug_status=ON_QA&bug_status=VERIFIED&bug_status=FAILS_QA&bug_status=RELEASE_PENDING&bug_status=POST&bug_status=PASSES_QA&long_desc_type=substring&long_desc=&bug_file_loc_type=allwordssubstr&bug_file_loc=&status_whiteboard_type=allwordssubstr&status_whiteboard=&fixed_in_type=allwordssubstr&fixed_in=&qa_whiteboard_type=allwordssubstr&qa_whiteboard=&keywords_type=allwords&keywords=Tracking&emailassigned_to1=1&emailtype1=exact&email1=&emailassigned_to2=1&emailreporter2=1&emailqa_contact2=1&emailcc2=1&emailtype2=exact&email2=&bugidtype=include&bug_id=&votes=&changedin=&chfieldfrom=&chfieldto=Now&chfieldvalue=&cmdtype=doit&order=Reuse+same+sort+as+last+time&field0-0-0=noop&type0-0-0=noop&value0-0-0= here] ||<tableclass="message note">''Tracker'' bugs--commonly referred to as ''blocker'' bugs--are meta-bugillas used to monitor a group of bugs that must be resolved before a specific release milestone and are therefore considered to ''block'' the release.|| == Currently Supported Versions == * Fedora 8: EOL one month after GA of Fedora 10 (''roughly'' November 30, 2008) * Fedora 7: EOL on 2008-05-29 (based on current schedule as of 2008-03-10) * All other versions of Fedora are ''unmaintained'' and thus ''End Of Life'' (EOL). ||<tableclass="message note">An ''unmaintained release'' receives no maintenance or updated packages. Therefore bugs are not tracked against these releases. In the future we will seek an enhancement to bugzilla to disable the ability to file bugs against unmaintained versions as there is no value in doing so.|| == Release Processes == ||<tableclass="message note">The following sections talk about ''submitting tickets to Red Hat Engineering Operations'' and ''running scripts''. The overarching idea here is to have standardized processes and scripts to complete these tasks each release. The ''scripts'' have not been written and are referred to for sake of example. Red Hat Engineering Operations maintains Fedora's bugzilla. || === First Day of Development === * Create tracker bugs for test releases and GA of next release * Flush BugZappers/ActiveTriagers page and make way for the new triagers for the next release * There are no term limits, but we want to flush the page each release so it stays current without a lot of work. We don't think asking people to re-add their names once every six months is a big deal. * Send out announcement to fedora-test-list@xxxxxxxxxx asking triagers for next release to add their names === Two Weeks Before Release Date === * File a ticket with Red Hat Engineering Operations requesting advanced preparation for the following: 1. Creation of a new Fedora version the day before release day 1. Ready the ''rawhide-mover'' script for mass-change of all bugs currently open against the ''rawhide'' version to be mass moved to the new Fedora version * For example all ''rawhide'' bugs open up until the official release of ''Fedora 9'' will be changed to version ''9'' from ''rawhide''. 1. Update text that refers to the latest Fedora version on these pages (coordinate wording with marketing): a. https://bugzilla.redhat.com/ a. https://bugzilla.redhat.com/enter_bug.cgi 1. Create/update script ''eol-warning'' script for mass-change of all bugs for the oldest supported version which will become ''end of life'' (EOL) one month from GA date a. '''DECIDE''': Should these bugs be put in special state or tagged in some way (e.g. keyword)? a. include text explaining that: i. one month of support remains i. please attempt to reproduce on most recent version of Fedora released which is: '''FIXME''' i. if you discover this bug is present in the most recent version please change the version to that release i. if you are using this bug for other purposes and wish for it to remain open, change it to the latest Fedora version to avoid auto-closure i. if this bug remains open on: '''FIXME''' it will automatically be ''CLOSED:WONTFIX'' 1. Create/update script ''eol-closer'' script for mass-change of '''all''' OPEN bugs for the release which will become ''unmaintained'' (EOL). The script when run will: a. change status to CLOSED WONTFIX a. include text explaining that: i. this release is no longer supported--an updated package will not be created. As a project we are only capable of supporting two releases at a time and there for our efforts are focused on currently supported versions. i. we would still appreciate your help if you could attempt to reproduce this bug on the latest supported version which is: '''FIXME''' or latest test release for the version under development which is: '''FIXME''' . If issue still exists, please change the bugzilla version to reflect where you reproduced the issue and reopen the bug with a status of NEW. i. standard wording will be maintained on at the Bugzappers/BugzillaBoilerPlate ('''TODO: CREATE''') page === One Week Before Release Date === 1. Ready the ''eol-warning'' script for mass-change of all bugs for the oldest supported version which will become ''end of life'' (EOL) one month from GA date: a. select all bugs !CLOSED (any status except CLOSED) a. select all bugs open for the version becoming EOL a. include text explaining that: i. one month of support remains i. please attempt to reproduce on most recent version of Fedora which was just released and is: ____. If the issue still exists, please change the bugzilla version to reflect the current version and note the package NVR. i. if this bug remains open on: _____ (date of EOL) it will automatically be closed as WONTFIX i. standard wording will be maintained on at the Bugzappers/BugzillaBoilerPlate ('''TODO: CREATE''') page === One Day Before Release Date === 1. Confirm that release day is for sure with: '''FIXME''' 1. Enable new version in Bugzilla 1. Request execution of ''rawhide-mover-script'' === Day of Release === 1. Run ''eol-warning'' script 1. File a ticket with Red Hat Engineering Operations requesting advanced preparation of the ''eol-closer'' script for mass-change of all bugs for versions which are which are EOL * '''DECIDE''': Should these bugs be put in special state or tagged in some way (e.g. keyword)? * include text explaining that: a. one month of support remains a. please attempt to reproduce on most recent version of Fedora released which is: ____ a. if this bug remains open on: _____ (date of EOL) it will automatically be closed as WONTFIX 1. Run a quick query for '''all''' bugs that are !CLOSED (any state except CLOSED) for all release which have reached End of Life (EOL). a. Add boilerplate text explaining that it is Fedora's policy that all bugs all EOL releases remain in CLOSED state. Also encourage the reporter to attempt to reproduce the bug against the latest maintained version of Fedora a. Change status of bug to CLOSED. a. standard wording will be maintained on at the Bugzappers/BugzillaBoilerPlate ('''TODO: CREATE''') page === One Month after GA Release === 1. Confirm EOL date with: '''FIXME''' 1. Run ''eol-closer'' script 1. Update this page with correct maintained and EOL Fedora versions 1. Update Bugzappers/BugzillaBoilerPlate ('''TODO: CREATE''') page with correct maintained and EOL Fedora versions === Ongoing === 1. Monitor '''all''' bugs that are !CLOSED (any state except CLOSED) for all release which are no longer maintained have reached End of Life (EOL). 1. Monitor and triage all bugs for currently maintained and rawhide releases which have a bugzilla status of NEW or have been in status of NEEDINFO for greater than thirty (30) days. See [:BugZappers/BugStatusWorkFlow bug life cycle] for more information and policies. == Bugzilla Script Lexicon == 1. ''rawhide-mover'' mass moves all bugs currently open against the ''rawhide'' version to the latest released Fedora version 1. ''eol-warning'' posts a warning to all bugs for the oldest supported version (GA minus 2) which will become ''end of life'' (EOL) one month after the GA date 1. ''eol-closer'' closes '''all''' bugs for release becoming EOL == Comments and Concerns == 1. Comment #1 1. Comment #2 --- Extreme Makeover - i.e. lightening the current load --- = Bugzilla Extreme Make Over Fedora 9 = ||<tableclass="message warning">This is a draft of the proposed approach. It will be circulated for review prior to execution (2008-02-22)|| [[TableOfContents]] == Background == Fedora has a proliferation of open bugs, so much so, that we are getting to the point (or may well have reached it) that it is becoming unmanageable. It does not seem logical to believe that we can continue on as we are and not do long term damage to the Fedora project. The time to act is now! By not addressing this arms race of bugs we put a brighter Fedora future in jeopardy: 1. We alienate community members that report bugs, but do not see a timely response, if ever. 1. As fewer people report bugs, the quality of Fedora releases has a higher change of declining 1. We discourage community bug triagers from joining a situation where the results of their work cannot be seen because bug activity is lost in the ocean of other bugs. 1. We distract new bug triagers who want to focus time on the easiest bugs (versions which are EOL) which provide very little value to current work We need to start ASAP so there is enough time to let the proposed process below run its course prior to 2008-04-29 (GA for Fedora 9). == Goal == 1. Completely execute the following proposal in stages by GA of Fedora 9 1. Institute a [:BugZappers/HouseKeeping: go-foward plan] so we do not have to repeat this process again--ever == Open Bugs for EOL Versions == * Fedora Core for ''versions'' 1 through 6 === Overall strategy === * Change status of '''all''' open bugs to NEEDINFO and close '''all''' open bugs after waiting 30 days === Implementation === 1. START: now() 1. Script executed by RHT Engineering Operations which queries bugs according to following criteria and takes the following actions: a. Version of Fedora (1 to 6) a. Status !CLOSED (any bug not in a closed state) a. Ascertain if bug is a ''blocker'' or ''tracker'' bug 1. Set bugs meeting criteria to ''NEEDINFO'' 1. If the bug is being used as a ''tracker'' or ''blocker'' bug add special ''keyword'' of '''Tracking''' to identify this bug for proper handling in the future a. Auto-change ''version'' to '''rawhide''' to extend life of Tracker 1. Add a friendly comment addressing the following: a. apologize for untimely response to this issue a. explain present situation and go-forward plan a. request that, if at all possible, they reproduce the bug against Fedora 8 or a Fedora 9 test release (version ''rawhide'') a. if bug exists in current version, change ''version'' of bug to current version: '''8''' or '''rawhide''' a. if bug owner wishes (for whatever reason) for the bug to remain ''open'', the version must be ''changed'' to ''Fedora 8'' and status of ''ASSIGNED''. a. if this bug remains open against EOL version it will be auto-closed as WONTFIX 30 days from now i. regardless of current open state (NEW, ASSIGNED, NEEDINFO, MODIFIED, etc.) i. there will no additional warnings or waiting period 1. END: now() + 30 days == Open Bugs for ''rawhide'' == * Fedora bugs for ''version'' '''rawhide''' * open rawhide bugs are harder to address because it is not readily apparent which ''released'' Fedora version they are most closely tied to--the following sub-sections address this problem * A process has been proposed to avoid this problem with rawhide bugs in the future at [:BugZappers/HouseKeeping: Good Bugzilla Hygiene] === Overall strategy === * separate ''very stale rawhide'' bugs from ''relevant rawhide'' bugs using the previous release schedule as a guide: http://fedoraproject.org/wiki/Releases/HistoricalSchedules * stratify bugs and process in buckets resulting in bugs that will be automatically closed OR manually triaged as part of the regular processes * sync up rawhide bugs with the currently supported release going forward === Implementation === * Script written and executed by RHT Engineering Operations which selects bugs according to specified criteria and takes the specified actions. ==== Stale Rawhide ==== 1. START: now() 1. Select all bugs meeting the following criteria: a. Version = ''rawhide'' a. All states !CLOSED (Any status other than CLOSED) a. No comments or internal activity of any kind since GA of Fedora 7 (May 31, 2007): nine months of inactivity * Most likely these are bugs reported against releases that are EOL * admittedly this is a little aggressive, but risk is low; but we need to address as many of these as possible--a bug can always be reopened. 1. If the bug is being used as a ''tracker'' or ''blocker'' bug add special ''keyword'' of '''Tracking''' to identify this bug for proper handling in the future 1. Act on each bug meeting the specified criteria: a. Change bug status for all bugs meeting criteria to NEEDINFO a. Add ''whiteboard'' tag of ''bzcl34nup'' * this helps to clearly tag bugs targeted for this exercise--the date of last activity will no longer be an accurate field to trigger off of after posting comments requesting action. a. Add a friendly comment addressing the following: i. apologize for untimely response to this issue i. explain present situation and go-forward plan i. request that, if at all possible, they reproduce the bug against Fedora 8 or a Fedora 9 test release (''rawhide'') i. if the bug exists in current version, change ''version'' of bug to current version (or leave as ''rawhide'') and change status to '''NEW''' * NEW rawhide bugs will be picked up in subsequent step and manually triaged i. if no action is taken this bug will be auto-closed as WONTFIX 30 days from now without any additional warnings or waiting period i. if bug is being used as a tracking bug add keyword of '''Tracking''' and change status to ASSIGNED 1. now() + 30 days a. Select all bugs meeting the following criteria: i. Version = ''rawhide'' i. ''whiteboard'' tag of ''bzcl34nup'' i. Status = ''NEEDINFO'' i. Not a tracker bug (does NOT have ''Tracking'' keyword set) a. auto-close bugs as WONTFIX a. Add a friendly comment explaining the following: i. following up to request posted 30 days ago i. if bug exists in a currently supported version feel free to reopen bug against that release ==== Possibly Relevant Rawhide ==== The easiest way to get these bugs into the pipeline for eventual closeout or resolution is be to tie them to the Fedora 8 release 1. START and END: now() 1. Selection criteria a. ''rawhide'' version a. Opened AFTER May 31, 2007, but BEFORE November 1, 2007 a. Any status other than CLOSED a. Not a tracker bug (does NOT have ''Tracking'' keyword set) 1. Action: a. Change version to Fedora 8 a. add a friendly note i. highlight version change i. explain reason for change i. include link to this proposal ==== Current Rawhide ==== * Opened against ''rawhide'' version on or after November 1, 2007 * Do '''nothing''' to these bugs--they will be addressed as part of [:BugZappers/HouseKeeping: good bugzilla hygiene] * Once all of the above processes have run--take regular triage actions on these bugs * Triagers review all ''rawhide'' bugs in NEW or NEEDINFO == Concerns & Questions == * YourWikiName 1. Comment 1 1. Comment 2 -- Jon Stanley Fedora Bug Wrangler jstanley@xxxxxxxxxxxxxxxxx -- fedora-devel-list mailing list fedora-devel-list@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/fedora-devel-list
