Rawhide serves several purposes in Fedora and yet I don't think we
clearly delineate what they are. It is often said in Fedoraland that
the best way to test the Feodora is to "run rawhide." There is a
difference between installing and running rawhide to experience the
absolute latest and greatest Fedora has to offer and intentionally
testing it. I just think we need to be clearer about our approach and
use of rawhide for testing (specific QA) because sometimes it doesn't
make sense.
Take a step back and consider what a strange testing target rawhide is:
1) An "officially" available version for reproducing bugs or test
results only exists for one day--contrast that with our build
environment that was created specifically with the intention of always
being able to recreate binaries in their original environment.
2) For most people, the economical way to get rawhide is a mirror--there
is no simple definitive way to determine that you have "the originally
composed version" of rawhide for that day. There are a few methods that
can give you "reasonable certainty", but nothing like a single check sum
or a way to confirm that the tree you've downloaded is exactly the same
as what was composed.
3) There is no "last known good version" when things completely get
messed up and you need to reinstall from scratch except the Alpha, Beta,
and Preview Release.
4) You never know when it will install or not. It isn't smoke
tested--we leave that for *everyone* to do themselves based on their own
install attempt. Even if Fedora hosted an automated smoke test to
determine if it installs you still have the problem of mirrors being out
of sync and not knowing if what you have is the exact same group of
packages that passed the smoke test.
5) It is the community's only access point for obtaining the (what is
close to) the Release Candidate for testing. I know several people will
disagree with me immediately here--we've had the argument several times
on IRC. I still think the underlying assumptions that it is "close
enough" and "most likely the same thing" are not good enough. I think
we have been lucky so far and there was a situation with F9 where the RC
contained a different package version than what was in rawhide.
6) We often place a higher value on daily rawhide than the Alpha and
Beta releases by proclaiming that they "don't really matter that much
because they are 'simply snapshots' of rawhide." The community at large
seems to focus more on the Alpha, Beta, and Preview releases as
evidenced by spikes in traffic on fedora-test-list after these releases.
7) We consider rawhide our primary testing target yet there is no
practical way to create a test matrix around it because it changes every
day. Instead we create test matrices for the Alpha, Beta, and Preview
releases which..... see the previous point. How do we know when we have
completed a full test run? How can you thoroughly test a moving target?
I know I will hear the stock replies of "but it is rawhide" or "this is
Fedora, not an enterprise distro like RHEL" and while I agree that
"rawhide is rawhide" (a circular, but well understood argument for long
term rawhide veterans) and that Fedora is not striving to be an
"enterprise distro" that doesn't mean one of Fedora's goals has to be
emphasizing a testing process that is flawed and could be better if we
all put our collective brains together to come up with something better
:-) We innovate in so many other areas... why not innovate here?
I would like to advocate that we reconsider the value we place on
rawhide and the emphasis we place around the Alpha, Beta, and Preview
Release. I think a good place to start would be document in our test
plan where using rawhide for test results makes sense and where it does
not. I believe rawhide does have its place, but I think we are trying
to use it to cover too many bases and could do more effective testing
with a more refined approach which in the end makes Fedora better!
What do other people think? Is there something here worth throwing
around here on this list with a following up discussing at FUDCon later
in the week?
--
fedora-test-list mailing list
fedora-test-list@xxxxxxxxxx
To unsubscribe:
https://www.redhat.com/mailman/listinfo/fedora-test-list
