On Wed, 2009-12-02 at 15:47 -0500, Peter Jones wrote: > On 12/02/2009 03:16 PM, James Laska wrote: > > If I'm summarizing the pain correctly ... the push back seems to come > > from providing testing before development is ready to accept bug > > reports? > > > > Does this only happen during early development? > > No. In fact, a part of this *is* due the difficulty in testing loader > changes. Pretty often the (current) workflow goes like: > > 1) joe writes a loader change that's complex > 2) joe compile tests it, tries to unit test code snippets isolated into > their own tiny (transient) test harnesses where possible (which it > often is not). > 3) joe fixes whatever problems are found > 4) joe submits the change for review on a-d-l > 5) a-d-l review finds some problems, misses others > 6) if a-d-l review finds problems, joe goes back to step 2 > 7) else, joe commits the change > 8) 0 or more days later a build gets done by joe or somebody else > 9) rawhide composes with the new build > 10) rawhide is broken because of one character in a string someplace > being wrong! > 11) bob comes in in the morning and does a test install with rawhide > 12) bob fixes the error, goes through steps 1-7 again (or maybe > just commits a trivial change, but generally not these days) > 13) pile-o-bug-reports arrives, almost all of them after the bug is fixed > 14) mike spends 8 hours triaging bug reports that are completely > irrelevant > > Note how there's no set schedule on step 8. > > This happens at all times during development. We can alleviate some of > the problem by writing/improving tools, for example to build/update > anaconda's initrd.img, but I don't honestly believe that will cut down > more than, say, half of this problem. It also doesn't really help > with of the /combined/ tree (that is, with changes other developers are > working on), with arches that aren't what we're using on the desktop, > or with e.g. CD media. For those, there's not much you can do without > just building it in koji and punging up a tree. And we don't really > feel it's worth pushing to testers, no matter what quality of tester > or test methodology, until we've done some simple trials first. > > The new way, including some feedback from this thread, is more like this: > > 1) steps 1-8 from above happen > 9) somebody on the anaconda team composes a tree[1] > 10) we (the anaconda team) smoke test it and go through steps 11 and 12 > above > 13) one of us asks for a tree to be composed for testing > 14) we (I guess rel-eng does this bit) put some images somewhere for > testing [details needed here] > 15) if they're not catastrophic we replace the LNG images with those > 16) we (the anaconda team) go back to step 1. This is great, thanks for taking the time to outline the detailed steps. The above procedure sounds like a solid plan. In addition to the above process, during the initial F-13 rollout of NFR, might I request a few additions? First, that we continue having rawhide install images available at all times. However, they aren't images that are pungi'd every night ... they are simply be the images built and tested from the last time through step#15 above. So for now in the F-13 cycle, they would be the LNG (last known good) of F-12. This seems to have the best of both worlds by allowing us to work the kinks out of this new process, while keeping up expectations/messaging around rawhide image availability. Second, that we choose a few dates where we will attempt to walk through the 16 steps prior to the freeze. My worry with not planning ahead would be we wait until our pre-alpha verification step ... and it's a train wreck. > [1]: I'm most likely going to automate this so we automatically get > x86_64 and i386 trees composed on cutlet (our local mirror) any > time it sees a new anaconda package in rawhide. Let's talk @ FUDCon. Will Woods and David Cantrell have some interesting ideas on how AutoQA might be used to respond to certain events and provide test feedback for you. Presently, it will detect new rawhide install images and run through a basic install. The image detection mechanism and the desired tests can be adjusted as needed. > > Meaning, after Alpha/Beta is having nightly built rawhide install > > images still a problem? When in the schedule [1] would be an > > acceptable time to start providing test feedback? > > I think maybe after beta could work if we feel it's really necessary; > it's not wholly unreasonable to expect us (the anaconda team) to be a > bit more rigorous in composing our own trees and testing before (and > after) step 7. But you've got to realize that that's basically just > reverting to the original process above, which still has step 8 with no > set schedule. So it's not like that's really different - you're just > seeing the same installer over and over. > > > Another point, I think we can't limit the discussion to just > > anaconda-devel. There are a *lot* of other critical components pulled > > into the install images that are developed outside that group. > > We didn't ever limit this discussion to anaconda-devel. Actually, > I don't think we've discussed this there at all. Sorry, didn't meant to imply that discussion was only taking place on anaconda-devel-list. Just meant there may be other development teams who rely on the rawhide install images for test feedback. Thanks, James
Attachment:
signature.asc
Description: This is a digitally signed message part
-- fedora-test-list mailing list fedora-test-list@xxxxxxxxxx To unsubscribe: https://www.redhat.com/mailman/listinfo/fedora-test-list
