On Mon, 2010-11-29 at 09:56 -0700, Kevin Fenzi wrote: > Greetings. > > On the devel list recently there was a long thread discussing how we > could improve the current https://fedoraproject.org/wiki/Updates_Policy > > I gathered up ALL the ideas people put forth that were concrete in a > list for fesco to look into. I'd love feedback from QA/Testers as well. > > If you have additional (concrete) ideas to add, please feel free. > > Note that these are not my ideas. ;) Thanks for sending these along Kevin. > The list I have so far: > > General: > > * Just drop all the requirements/go back to before we had any updates > criteria. Hmm, certainly an idea. I feel like this is definitely a step backward, not forward. Has the initial motivation for an updates policy gone away or changed? Have we encountered problems that didn't yet exist, or weren't as painful, when the policy was first enabled? Are there other problems we need to focus on resolving (I suspect this is the case)? > * back off current setup until autoqa is ready, see what we want to do > after that lands. Much the same as above. In addition, automation doesn't magically solve problems, it only automates a repeatable process that already exists. If we haven't documented the steps manually, I wouldn't know where to get started. I'd like to see us work out the manual steps required to test stuff ... then we can incorporate automation where sensible. > * Change FN-1 to just security and major bugfix. Nothing else allowed. I don't have objections, and with repos.fedoraproject.org we have an avenue for maintainers to provide more disruptive packages to existing releases. From a QA standpoint, this would certainly lower the volume of updates that need QA attention. But this definitely seems like a general Fedora strategy/audience topic, than a QA specific one. > * allow packages with a %check section to go direct to stable. Interesting, I like the spirit of this idea, but would like to see if we can incorporate this with autoqa karma plans. So, perhaps packages with %check get automated karma? Just the same as with packages that pass automated tests ... they'll eventually get positive karma of some form. > * setup a remote test env that people could use to test things. I could use more details on this point. Is this talking about setting up QA systems hosted in Fedora infrastructure that any tester could login and use to test updates? > * require testing only for packages where people have signed up to be > testers. Hmm, I like this idea in part as it allows an opt-in for testing for maintainers. I'd stand behind this for packages outside critpath, but for critpath, we need to test them. We've moved forward with the policy without a implementation from QA for hosting/contributing test documentation. I suspect that may be the cause of the policy push back, that there isn't enough test feedback. I believe that's in part because we haven't provided clear test instructions for people to follow. It's a *significant* time investment to test an update that has no documented test instructions and that you aren't familiar with. QA plans to address this starting in Fedora 15 by laying some wiki ground work for documenting test procedures for specific updates. > * Ask maintainers to provide test cases / test cases in wiki for each package This is always the case. Proper testing comes from all levels, including development. However, as noted above, Fedora QA needs to provide some area/place for test documentation to develop/mature. Currently, we can discuss test guidance and instructions on the test@ mailing list. This practice works well for tests related to release verification. This is a great public forum to knock out the details. Moving forward, we need to capture those instructions on the wiki ... and ideally integrate it with the bodhi update request (e.g. "Click here for test instructions"). > * have a way to get interested testers notified on bodhi updates for packages > they care about. Interesting, almost like a watch-list for bodhi updates. Seems like a feature worth exploring in more detail. I use a similar feature with koji to watch for new builds. > * reduced karma requirement on other releases when one has gone stable > * aggregated karma across the releases for the same package version. I don't have data to indicate how many updates have been released, and then reverted/obsoleted on only a subset of releases. > * PK updates-testing integration of some kind. Open to any ideas here ... are you thinking about some PK updates-testing feedback workflow. Like integrating fedora-easy-karma? Something else? > * allow anon karma to count. Or maybe it counts, but counts less (.5 karma or something). > * setup fedora-qa package or group to more easily bring up more testers. Certainly not opposed to it, we've been focused on building a larger community of testers (proventesters). Over time, I could see that group breaking out into more specific interest groups. I'm inclined to listen to that request more if it consistently came from proventesters noting it as a obstacle to participation. Meaning, I'm not aware that the lack of component-specific test groups is preventing tester participation. That said, I like the idea of micro-communities developing to focus on specific use cases or components. > * Testing is only required for certain packages. Those packages are the > packages where problems have occurred before so fesco or other maintainers > affected by the changes deem it necessary to supplement the maintainer's > testing with outside help. > > - Option: supplement this list with critpath packages where the > maintainers desire extra testing. This means that we would no longer be > dragging in dependencies immediately... only if updates by the > dependency's maintaner to that package are breaking things. > > * updates that only modify the spec could have a lower requirement. > (ie, to fix a packaging issue, no changes in the upstream software). All %obsoletes, %requires, %provides, %files and %patch statements are only recorded in the .spec file. Just because they are in the .spec file doesn't mean they are any less disruptive. > Security updates: > > * allow security updates to go direct to stable Risky > * ask QA to commit to testing security updates We can't commit to testing without guidance or instructions. Let's commit to documenting repeatable procedures that testers can follow and expand upon. For some security updates, they may have already been functionally tested upstream. I think it's reasonable to provide proxy karma linking to upstream functional tests. Though, I don't think upstream functional tests alone can bless a security update. > * allow timeout for security updates before going to stable. > > Critpath updates: > > * allow critpath timeout for going to stable. I think this will come back to hurt us. These things landed in critpath for a reason. I'd like to work on increasing tester engagement, before we loosen the process around critpath packages. > Non critpath/security: > > * reduce timeout for non critpath from 7 to 3 days. > > * change default autokarma to 2 or 1. No immediate thoughts on these points. Thanks, James
Attachment:
signature.asc
Description: This is a digitally signed message part
-- test mailing list test@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
