Will Woods said the following on 06/17/2008 11:12 AM Pacific Time:
On Mon, 2008-06-16 at 10:16 -0400, John Poelstra wrote:
1) An "officially" available version for reproducing bugs or test
results only exists for one day--contrast that with our build
environment that was created specifically with the intention of always
being able to recreate binaries in their original environment.
The difference between rawhide on any two days is, on average, somewhere
between 0.1% and 0.5%. If this *blazing* development pace is too much
for you, you're welcomed and encouraged to keep multiple copies of the
rawhide repos.
Rapid amount of change is not the issue. I'm speaking more to the
situation of encountering a problem, reporting a bug, being asked for
more information on that bug, but not being able to troubleshoot further
because the exact same tree used to install is gone... there is no history.
It'd definitely be nice if we had some good scripts for maintaining
multiple local copies of rawhide, with hardlinks to save space.
2) For most people, the economical way to get rawhide is a mirror--there
is no simple definitive way to determine that you have "the originally
composed version" of rawhide for that day. There are a few methods that
can give you "reasonable certainty", but nothing like a single check sum
or a way to confirm that the tree you've downloaded is exactly the same
as what was composed.
This is just plain wrong. The timestamp in .treeinfo is unique
per-tree.
Just because .treeinfo is present doesn't mean all the other packages
associated with it from that day are present on the mirror.
<<CUT>>
4) You never know when it will install or not. It isn't smoke
tested--we leave that for *everyone* to do themselves based on their own
install attempt. Even if Fedora hosted an automated smoke test to
determine if it installs you still have the problem of mirrors being out
of sync and not knowing if what you have is the exact same group of
packages that passed the smoke test.
This is the problem that the rawhide dashboard page is supposed to
solve. We're automating SNAKE to do smoke tests on rawhide every night,
and post results on the dashboard. So you *will* know whether it
installs or not.
Excellent. I didn't realize that was in process. Could the dashboard
also contain a historical record of which days install and which days
don't? I think this would be useful information to have.
We've got a prototype rawhide dashboard page that shows whether boot
images are present, at least:
http://wwoods.fedorapeople.org/rawhide.html
snake-treeinfo can tell this too, right?
As for the mirrors being out of sync - see above.
5) It is the community's only access point for obtaining the (what is
close to) the Release Candidate for testing. I know several people will
disagree with me immediately here--we've had the argument several times
on IRC. I still think the underlying assumptions that it is "close
enough" and "most likely the same thing" are not good enough.
I think you're wrong. I also think you underestimate the time it
requires to put out an RC versus the turnaround time on testing it.
Given the amount of time it adds to the schedule and lengthens the
development freeze, plus the amount of work it takes from every part of
the release team, what would we actually *gain* from doing all that
work?
I think we can create a more polished release with less critical items
on the "known issues" wiki page for that release. I realize the
"criteria" and "how much time is too much" is somewhat subjective and
something we do not agree on.
6) We often place a higher value on daily rawhide than the Alpha and
Beta releases by proclaiming that they "don't really matter that much
because they are 'simply snapshots' of rawhide." The community at large
seems to focus more on the Alpha, Beta, and Preview releases as
evidenced by spikes in traffic on fedora-test-list after these releases.
I think you're mixing up cause and effect here.
>
People don't focus on the milestones because they just inherently *love*
milestones. They focus on the milestones because we go to the effort of
making them easy to consume, and we publicize them and test them to
ensure they'll be installable and put up ISOs on torrents and the
mirrors. So *obviously* more people use them.
I was attempting to respond to past situations when the alpha and beta
haven't been in that great of shape and it has been said that it doesn't
matter that much because "beta == snapshot of rawhide".
7) We consider rawhide our primary testing target yet there is no
practical way to create a test matrix around it because it changes every
day. Instead we create test matrices for the Alpha, Beta, and Preview
releases which..... see the previous point. How do we know when we have
completed a full test run? How can you thoroughly test a moving target?
I completely disagree with the assertion that "there is no practical way
to create a test matrix for Rawhide".
First of all, the only reason we don't create test matrices for every
day's rawhide is that it's time-consuming to create the matrix itself.
The old wiki's pretty slow, remember?
Better test run tracking (with Testopia, for instance) will make it
trivial to create a matrix for every day's rawhide. Further, automating
our test cases will let us fill in most of that matrix automatically.
The test plan for rawhide is, obviously going to be somewhat less
exhaustive than the test plans for milestone releases or the final
release. So we can complete a "full test run" by looking at the
(mostly-full, auto-created) matrix for that day's rawhide and.. filling
in the blanks.
that doesn't mean one of Fedora's goals has to be
emphasizing a testing process that is flawed and could be better if we
all put our collective brains together to come up with something better
:-) We innovate in so many other areas... why not innovate here?
I would like to advocate that we reconsider the value we place on
rawhide and the emphasis we place around the Alpha, Beta, and Preview
Release.
Innovate.. by emphasizing the traditional milestones? I think you'll
find that "innovation" and "tradition" are actually *opposites*.
Innovate by taking a different approach we haven't considered before. I
didn't suggest that "traditional milestones" are the answer.
Rawhide might be a strange testing target from the point of view of
someone coming from RHEL or proprietary systems or other traditional
milestone-focused development, but I don't think Rawhide is that strange
in the Open Source world.
I'll give that some more thought.
You can apply nearly all of your arguments to, say, the way the kernel
is developed - it changes too quickly! Not enough freeze time! We need
more milestones! You can't possibly test it!
>
You know what would *really* be innovative? Engineering our test efforts
to match the pace and reality of typical Open Source development, rather
than working the other way around.
I think a good place to start would be document in our test
plan where using rawhide for test results makes sense and where it does
not. I believe rawhide does have its place, but I think we are trying
to use it to cover too many bases and could do more effective testing
with a more refined approach which in the end makes Fedora better!
Okay. So start documenting where you think it does and doesn't make
sense and we'll discuss *that*. But so far all you've done is rehashed a
bunch of complaints and offered no solutions.
I guess that is the tricky part. I'd like to think that being a
contributor to Fedora does not mean you always have to have the solution
to raise what you perceive as something that is wrong. Hopefully my
contributions in other areas Fedora make up for it here.
What do other people think? Is there something here worth throwing
around here on this list with a following up discussing at FUDCon later
in the week?
Sure, if we go to the effort of actually *defining* problems and
discussing *solutions*, it's totally worth it.
I agree.
Or if the problems are misunderstood then we need to do a better
explaining why our current approach really does make sense--I am not the
only one who has echoed these same questions. Maybe someone else can
explain the issues I'm trying to get at in a better way?
John
--
fedora-test-list mailing list
fedora-test-list@xxxxxxxxxx
To unsubscribe:
https://www.redhat.com/mailman/listinfo/fedora-test-list
