#175: Improve transfer of previous test results in the installation matrix
--------------------------+-------------------------------------------------
Reporter: kparal | Owner: rhe
Type: enhancement | Status: new
Priority: major | Milestone: Fedora 15
Component: Wiki | Version:
Resolution: | Keywords:
--------------------------+-------------------------------------------------
Comment (by rhe):
Replying to [ticket:175 kparal]:
> Currently in our installation matrix there are some test results which
are from "anonymous" source (no name filled in). Example:
>
> https://fedoraproject.org/wiki/Test_Results:Fedora_15_Alpha_RC2_Install
>
> I was little confused by that, but rhe explained me that those are the
test results that were transfered from the previous test result (i.e. from
F15 Alpha RC1 in this case). We already agreed if would be better if we
added <ref> with explanation to each such result, so other testers
understand the meaning of it (I suppose many of them might be confused
same as I was). Still, I think there are some improvements that could be
made. I'd like to discuss them in this ticket.
Thanks for raising this topic, it was originally discussed at:
- https://fedorahosted.org/fedora-qa/ticket/84
I supposed testers would look up the unknown results icons at
[https://fedoraproject.org/wiki/QA:Fedora_15_Install_Results_Template#Key
key section] where I explained. But it seems this part could be ignored, I
agree to have a more visible solution.
> My proposal:
> 1. Transferring old results to the new matrices is a great idea, I
strongly support that.
> 2. In order to avoid confusion about "anonymous" results, we should
clearly separate new results from transfered results. Using a <ref>
comment is possible, but I'd like to do something more visible. We could
do something like {{result|fail|rc1|12345}} and then link rc1 to correct
wiki page (if {{result}} template is not suitable for it, we could create
{{oldresult}} template or similar). But even though it could be scripted,
it seems like too much work. Easier solution is to do
{{result|fail|previous run|12345}}, and if "previous run" is linked to
[[User:previous run]], we could use that wiki page for short explanation.
{{result|fail|previous rc1 run|12345}} is good for me. Why do we need a
link here? The result has already been transferred to the page, I don't
think every previous result should have a link to previous page.
> 3. We should transfer only fails and warnings, not passes. The
rationale is that the pass result makes you (me, many people) think: "Hey,
there's a pass, let's work on something else". But pass from previous run
doesn't mean pass in current run (as I experienced today, when I changed
two previous passes to fails). Therefore warnings and fails are good to
know (we can check whether they are fixed or not, and we won't forget
about them), but passes are counter-productive - let's leave empty fields
instead.
The transfer of results depend on the change of anaconda/certain
components, not the results themselves in my opinion. It's not necessary
to retest many cases just because they passed in previous run. For
example, if the change of a new build won't affect the partitioning and
recovery parts, which were all passed in previous runs, do we need to test
them again and again in every new run? Certainly I can't guarantee the
passed case still pass in new run, that's why these results should be
distinguished with new tested ones.
> 4. All transfered results must have bugzilla number assigned. Otherwise
there's no information value. If I don't know what was broken, I can't
check whether it's been fixed or not.
Ah yeah, when one issue affected all cases in the same area, I used to
only assign a bug id to one case, use {{result|fail}} to the others. It's
not due to the transfer, I will assign the bug number to all cases to
reduce the confusion.
> 5. We should document all of this in the "Test Results Format" table
and also advise people how to interact with these transfered results. My
idea: If there was a transfered warn/fail, and you checked that the
problem is fixed, remove it from the table (and put your result in there
instead). If you checked that the problem is still present, again remove
it from the table and put your result in there instead (referencing the
same bug number). If you didn't check the problem, put your result in the
table and leave the transfered result intact.
I can see what you mean, but I'm afraid that rule will make posting
results more complicated, and also different environments could have
different results. Sometimes I can't reproduce a issue even though there's
not updates fixing it. If I know it's not fixed, I will keep previous
result, if I don't know, I will just remove it. So my general idea is: if
you test the case carefully and step by step according to the case, you
can replace the previous result with yours. I assume most testers will pay
attention on the previous results and carefully remove them. Besides, we
have history rollback and bugzilla for tracking, do we really need such
strict rule?
--
Ticket URL: <https://fedorahosted.org/fedora-qa/ticket/175#comment:2>
Fedora QA <http://fedorahosted.org/fedora-qa>
Fedora Quality Assurance
--
test mailing list
test@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe:
https://admin.fedoraproject.org/mailman/listinfo/test
