#175: Improve transfer of previous test results in the installation matrix
--------------------------+-------------------------------------------------
Reporter: kparal | Owner: rhe
Type: enhancement | Status: new
Priority: major | Milestone: Fedora 15
Component: Wiki | Version:
Resolution: | Keywords:
--------------------------+-------------------------------------------------
Comment (by rhe):
Replying to [comment:3 kparal]:
> Replying to [comment:2 rhe]:
> >
> > Thanks for raising this topic, it was originally discussed at:
> > - https://fedorahosted.org/fedora-qa/ticket/84
>
> Ah, that ticket slipped my attention :-)
>
> > {{result|fail|previous rc1 run|12345}} is good for me. Why do we need
a link here? The result has already been transferred to the page, I don't
think every previous result should have a link to previous page.
>
> I just supposed that [[User:previous run]] could explain a little how to
result transfer works. So noone thinks "previous run" is a real user name
:-) But you're right, that's not necessary. And "previous rc1 run" is even
more explanatory. No need for links.
Okay, then I'll begin using the format as {{result|status|previous <build>
run}} from the next text event.
> > Ah yeah, when one issue affected all cases in the same area, I used to
only assign a bug id to one case, use {{result|fail}} to the others. It's
not due to the transfer, I will assign the bug number to all cases to
reduce the confusion.
>
> Thanks, that will help. Maybe we can share the reference number somehow
(so the bug is not duplicated many times)?
Yes, how do you think the reference No.29 in
[https://fedoraproject.org/wiki/Test_Results:Current_Installation_Test#References
current install test]?
> > The transfer of results depend on the change of anaconda/certain
components, not the results themselves in my opinion. It's not necessary
to retest many cases just because they passed in previous run. For
example, if the change of a new build won't affect the partitioning and
recovery parts, which were all passed in previous runs, do we need to test
them again and again in every new run? Certainly I can't guarantee the
passed case still pass in new run, that's why these results should be
distinguished with new tested ones.
>
> We can never be sure where we can find some regressions. Even comments
change can break up stuff :-) OTOH I understand saving our resources. I
have split opinions on this. But if you selectively transfer only
"probably safe" passes, I am absolutely fine with that.
Right, transfer task need be very careful. So when all tests in a test
area seem safe passes, I would still keep at least one test with no
previous result so that every test area is tested in each run.
> > > 5. We should document all of this in the "Test Results Format"
table and also advise people how to interact with these transfered
results.
> > I can see what you mean, but I'm afraid that rule will make posting
results more complicated, and also different environments could have
different results. Sometimes I can't reproduce a issue even though there's
not updates fixing it. If I know it's not fixed, I will keep previous
result, if I don't know, I will just remove it. So my general idea is: if
you test the case carefully and step by step according to the case, you
can replace the previous result with yours. I assume most testers will pay
attention on the previous results and carefully remove them. Besides, we
have history rollback and bugzilla for tracking, do we really need such
strict rule?
>
> I see. So what about this - no rules at all. Let the tester just add
their results, don't recommend them any replacing of the old ones. At the
end of the day we will look over it and decide where we believe the old
one should be kept (or transferred to the next run) and where it should be
erased. I think it's better to count on our judgement than on arbitrary
tester's judgement. But I have no hard opinions in here.
Good idea. Agree with it. :)
I've modified the key section to:
*
https://fedoraproject.org/wiki/QA:Fedora_15_Install_Results_Template#Key
Do we need to add more instructions? Feel free to adjust it.
--
Ticket URL: <https://fedorahosted.org/fedora-qa/ticket/175#comment:4>
Fedora QA <http://fedorahosted.org/fedora-qa>
Fedora Quality Assurance
--
test mailing list
test@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe:
https://admin.fedoraproject.org/mailman/listinfo/test
