James Laska said the following on 12/02/2009 02:01 PM Pacific Time:
> On Wed, 2009-12-02 at 15:47 -0500, Peter Jones wrote:
>> On 12/02/2009 03:16 PM, James Laska wrote:
>>> If I'm summarizing the pain correctly ... the push back seems to come
>>> from providing testing before development is ready to accept bug
>>> reports?
>>>
>>> Does this only happen during early development?
>> No. In fact, a part of this *is* due the difficulty in testing loader
>> changes. Pretty often the (current) workflow goes like:
>>
>> 1) joe writes a loader change that's complex
>> 2) joe compile tests it, tries to unit test code snippets isolated into
>> their own tiny (transient) test harnesses where possible (which it
>> often is not).
>> 3) joe fixes whatever problems are found
>> 4) joe submits the change for review on a-d-l
>> 5) a-d-l review finds some problems, misses others
>> 6) if a-d-l review finds problems, joe goes back to step 2
>> 7) else, joe commits the change
>> 8) 0 or more days later a build gets done by joe or somebody else
>> 9) rawhide composes with the new build
>> 10) rawhide is broken because of one character in a string someplace
>> being wrong!
>> 11) bob comes in in the morning and does a test install with rawhide
>> 12) bob fixes the error, goes through steps 1-7 again (or maybe
>> just commits a trivial change, but generally not these days)
>> 13) pile-o-bug-reports arrives, almost all of them after the bug is
fixed
>> 14) mike spends 8 hours triaging bug reports that are completely
>> irrelevant
>>
>> Note how there's no set schedule on step 8.
>>
>> This happens at all times during development. We can alleviate some of
>> the problem by writing/improving tools, for example to build/update
>> anaconda's initrd.img, but I don't honestly believe that will cut down
>> more than, say, half of this problem. It also doesn't really help
>> with of the /combined/ tree (that is, with changes other developers are
>> working on), with arches that aren't what we're using on the desktop,
>> or with e.g. CD media. For those, there's not much you can do without
>> just building it in koji and punging up a tree. And we don't really
>> feel it's worth pushing to testers, no matter what quality of tester
>> or test methodology, until we've done some simple trials first.
>>
>> The new way, including some feedback from this thread, is more like
this:
>>
>> 1) steps 1-8 from above happen
>> 9) somebody on the anaconda team composes a tree[1]
>> 10) we (the anaconda team) smoke test it and go through steps 11 and 12
>> above
>> 13) one of us asks for a tree to be composed for testing
>> 14) we (I guess rel-eng does this bit) put some images somewhere for
>> testing [details needed here]
>> 15) if they're not catastrophic we replace the LNG images with those
>> 16) we (the anaconda team) go back to step 1.
>
> This is great, thanks for taking the time to outline the detailed steps.
> The above procedure sounds like a solid plan. In addition to the above
> process, during the initial F-13 rollout of NFR, might I request a few
> additions?
>
> First, that we continue having rawhide install images available at all
> times. However, they aren't images that are pungi'd every night ...
> they are simply be the images built and tested from the last time
> through step#15 above. So for now in the F-13 cycle, they would be the
> LNG (last known good) of F-12. This seems to have the best of both
> worlds by allowing us to work the kinks out of this new process, while
> keeping up expectations/messaging around rawhide image availability.
I really like this. Then we don't need a separate canonical location
for 'LNG' because we only update the install images in 'rawhide' if they
have gone through some level of sanity testing?'
Would there be any downside to continuing to do this even after NFR is
an established part of our process?
>
> Second, that we choose a few dates where we will attempt to walk through
> the 16 steps prior to the freeze. My worry with not planning ahead
> would be we wait until our pre-alpha verification step ... and it's a
> train wreck.
If you want to propose these dates (see other schedule) email I can put
them in now so we have a working draft to discuss at FUDCon.
John
--
fedora-test-list mailing list
fedora-test-list@xxxxxxxxxx
To unsubscribe:
https://www.redhat.com/mailman/listinfo/fedora-test-list
