#227: Document SOP for acceptance test event
--------------------+-------------------------------------------------------
Reporter: rhe | Owner: wutao85
Type: task | Status: new
Priority: major | Milestone: Fedora 16
Component: Wiki | Version:
Resolution: | Keywords:
--------------------+-------------------------------------------------------
Comment (by jlaska):
Replying to [comment:10 adamwill]:
> I'm not sure just linking to the acceptance test plan is sufficient
instruction on how to do the testing. It's a planning document, and it has
"TODO: Automate these test cases" written in it. :) Just reading that
page, I have no idea of the actual steps one has to perform to complete
the testing: run this program, click this button, etc. You could do the
testing *manually* by following each test case linked there, but that's
not really the point.
Just a note ... this isn't fully automated. I'd classify the automation
as a mature proof-of-concept. In addition, I've rarely been satisfied
with the automated results alone. The automation helps me figure out how
far up I need to roll up my sleeves for testing. In my experience, this
milestone calls for more hands on attention until the automation has
proven itself.
What I've discussed with twu is that this milestone needs to be revisited
and possibly extended to cover more test blocking issues ... not just it's
original mission of 'rawhide acceptance'. The concept of rawhide has
changed since this test plan was born. While the need still exists, this
plan doesn't go far enough to find the bugs that will block testing of the
'test compose'. What I'd like to see out of this test run is
identification of bugs for items identified in the current acceptance
plan, but also bugs that block significant portions of the release
validation test matrices. The one week between TC1 and RC1 is *not*
sufficient time to identify, resolve, verify and refix those bugs. I'm
convinced this has contributed to slips in all of the previous ALpha slips
for ~4 releases (if memory serves).
What I'd recommend is to focus on defining the goal and documenting the
process for this effort ... then we can follow-up with focused patching to
the rats_* test automation.
> What we really need are instructions by which someone who'd never done
the acceptance testing before could successfully do the acceptance testing
- automated, not manually. Thanks!
Pet peeve of mine ... I recommend ensuring the tests themselves are
documented first. From there, we can fill-in the gaps with any automation
that exists. The problem I have with the automation is that it's
insufficient for what this activity calls for. It's a good start, it
needs more work, and in it's current state, it can be a distraction from
the goal of finding bugs that will prevent further release validation
testing. Let's definitely improve the automation, which twu is looking
into, but track that separately.
> (If it requires access to some Red Hat-only system, or something, then
a) that sucks, but b) we should still write it down.)
It doesn't.
--
Ticket URL: <https://fedorahosted.org/fedora-qa/ticket/227#comment:11>
Fedora QA <http://fedorahosted.org/fedora-qa>
Fedora Quality Assurance
--
test mailing list
test@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe:
https://admin.fedoraproject.org/mailman/listinfo/test
