On Mon, 2008-06-16 at 10:16 -0400, John Poelstra wrote: > 1) An "officially" available version for reproducing bugs or test > results only exists for one day--contrast that with our build > environment that was created specifically with the intention of always > being able to recreate binaries in their original environment. The difference between rawhide on any two days is, on average, somewhere between 0.1% and 0.5%. If this *blazing* development pace is too much for you, you're welcomed and encouraged to keep multiple copies of the rawhide repos. It'd definitely be nice if we had some good scripts for maintaining multiple local copies of rawhide, with hardlinks to save space. > 2) For most people, the economical way to get rawhide is a mirror--there > is no simple definitive way to determine that you have "the originally > composed version" of rawhide for that day. There are a few methods that > can give you "reasonable certainty", but nothing like a single check sum > or a way to confirm that the tree you've downloaded is exactly the same > as what was composed. This is just plain wrong. The timestamp in .treeinfo is unique per-tree. Compare the master mirror: http://download.fedora.redhat.com/pub/fedora/linux/development/i386/os/.treeinfo With your local mirror, say: http://limestone.uoregon.edu/ftp/fedora/development/i386/os/.treeinfo If the timestamps match, you have the most current rawhide. You can also use 'snake' to do this: [wwoods@metroid ~]$ snake-tree info MIRROR_URL Name : Fedora development i386 (20080617) Arch : i386 Id : 1213693723.22 Version : development Family : Fedora Variant : Time : 2008-06-17 05:08 EDT URI's : Images : i386 (kernel, family, timestamp, variant, boot.iso, initrd, version, arch), xen (kernel, family, timestamp, variant, initrd, version, arch) > 3) There is no "last known good version" when things completely get > messed up and you need to reinstall from scratch except the Alpha, Beta, > and Preview Release. This is definitely a problem, and a tricky one at that. It deserves further discussion in its own mail thread and/or at FUDCon. > 4) You never know when it will install or not. It isn't smoke > tested--we leave that for *everyone* to do themselves based on their own > install attempt. Even if Fedora hosted an automated smoke test to > determine if it installs you still have the problem of mirrors being out > of sync and not knowing if what you have is the exact same group of > packages that passed the smoke test. This is the problem that the rawhide dashboard page is supposed to solve. We're automating SNAKE to do smoke tests on rawhide every night, and post results on the dashboard. So you *will* know whether it installs or not. We've got a prototype rawhide dashboard page that shows whether boot images are present, at least: http://wwoods.fedorapeople.org/rawhide.html As for the mirrors being out of sync - see above. > 5) It is the community's only access point for obtaining the (what is > close to) the Release Candidate for testing. I know several people will > disagree with me immediately here--we've had the argument several times > on IRC. I still think the underlying assumptions that it is "close > enough" and "most likely the same thing" are not good enough. I think you're wrong. I also think you underestimate the time it requires to put out an RC versus the turnaround time on testing it. Given the amount of time it adds to the schedule and lengthens the development freeze, plus the amount of work it takes from every part of the release team, what would we actually *gain* from doing all that work? > 6) We often place a higher value on daily rawhide than the Alpha and > Beta releases by proclaiming that they "don't really matter that much > because they are 'simply snapshots' of rawhide." The community at large > seems to focus more on the Alpha, Beta, and Preview releases as > evidenced by spikes in traffic on fedora-test-list after these releases. I think you're mixing up cause and effect here. People don't focus on the milestones because they just inherently *love* milestones. They focus on the milestones because we go to the effort of making them easy to consume, and we publicize them and test them to ensure they'll be installable and put up ISOs on torrents and the mirrors. So *obviously* more people use them. > 7) We consider rawhide our primary testing target yet there is no > practical way to create a test matrix around it because it changes every > day. Instead we create test matrices for the Alpha, Beta, and Preview > releases which..... see the previous point. How do we know when we have > completed a full test run? How can you thoroughly test a moving target? I completely disagree with the assertion that "there is no practical way to create a test matrix for Rawhide". First of all, the only reason we don't create test matrices for every day's rawhide is that it's time-consuming to create the matrix itself. The old wiki's pretty slow, remember? Better test run tracking (with Testopia, for instance) will make it trivial to create a matrix for every day's rawhide. Further, automating our test cases will let us fill in most of that matrix automatically. The test plan for rawhide is, obviously going to be somewhat less exhaustive than the test plans for milestone releases or the final release. So we can complete a "full test run" by looking at the (mostly-full, auto-created) matrix for that day's rawhide and.. filling in the blanks. > that doesn't mean one of Fedora's goals has to be > emphasizing a testing process that is flawed and could be better if we > all put our collective brains together to come up with something better > :-) We innovate in so many other areas... why not innovate here? > I would like to advocate that we reconsider the value we place on > rawhide and the emphasis we place around the Alpha, Beta, and Preview > Release. Innovate.. by emphasizing the traditional milestones? I think you'll find that "innovation" and "tradition" are actually *opposites*. Rawhide might be a strange testing target from the point of view of someone coming from RHEL or proprietary systems or other traditional milestone-focused development, but I don't think Rawhide is that strange in the Open Source world. You can apply nearly all of your arguments to, say, the way the kernel is developed - it changes too quickly! Not enough freeze time! We need more milestones! You can't possibly test it! You know what would *really* be innovative? Engineering our test efforts to match the pace and reality of typical Open Source development, rather than working the other way around. > I think a good place to start would be document in our test > plan where using rawhide for test results makes sense and where it does > not. I believe rawhide does have its place, but I think we are trying > to use it to cover too many bases and could do more effective testing > with a more refined approach which in the end makes Fedora better! Okay. So start documenting where you think it does and doesn't make sense and we'll discuss *that*. But so far all you've done is rehashed a bunch of complaints and offered no solutions. > What do other people think? Is there something here worth throwing > around here on this list with a following up discussing at FUDCon later > in the week? Sure, if we go to the effort of actually *defining* problems and discussing *solutions*, it's totally worth it. -w
Attachment:
smime.p7s
Description: S/MIME cryptographic signature
-- fedora-test-list mailing list fedora-test-list@xxxxxxxxxx To unsubscribe: https://www.redhat.com/mailman/listinfo/fedora-test-list
