Hello, this is a final call for comments to our Package Update Acceptance Test Plan [1]. This test plan is tightly related to Package Update Acceptance Criteria [2] approved by FESCo. It basically says how we, the QA team, will decide whether a package update should be accepted or rejected. It also tries to map the requirements of the policy into an actionable test plan. If you have any comments about the test plan, please post them now, so we can include them in the document. Although AdamW said he was pretty pleased with the state of the test plan (yay!), I still think there are a few things that should be clarified. I attach a list of them below, sorted by the individual test plan's paragraphs: = Test Priority = 1. Do we want to have the strict requirement that introspection and advisory tests may be started only after all mandatory tests have finished? In other words, do we want to have the testing sequence like this: -> mandatory tests -> (wait for completion) -> introspection + advisory tests or this: -> mandatory + introspection + advisory tests (no particular order) Reason for (the first sequence): It prioritizes the most important tests, they will be finished sooner. It can save resources in case some mandatory test fails and we won't run any subsequent tests (see question 2). Reason against: More difficult to implement. Less effective from overall performance view. = Test Pass/Fail Criteria = 2. Do we want to continue executing tests when some mandatory test fails? Reason for: We will have more results, maybe the maintainer will have a look at all the results and may fix more issues at once (not only the failed mandatory test). Reason against: It wastes resources - the update will not be accepted anyway. When a mandatory test fails (like installability or repo sanity), many follow-up tests may fail because of that, so they may not produce interesting output anyway. 3. We should complement requirement "all tests have finished" with a definition what happens if some test crashes. It is obvious that we can't accept package for which some substantial (read mandatory or introspection) test has crashed. So we can add a requirement like this: "all mandatory and introspection tests have finished cleanly (no test crashes)" The question remains - what about advisory tests? Do we require that they also don't crash? Or even if some of them crashes it won't be an obstacle for accepting the update? Reason for: Advisory tests are not important for accepting the update, a test crash should not cause rejection. Reason against: Some information won't be available. It could happen that those information would cause the maintainer to withdraw/renew the updated package. = Introspection tests = 4. Rpmlint - "no errors present" vs "no new errors present": It is obvious we have to provide an option for package maintainers to whitelist some rpmlint messages. The actual implementation of the whitelist is still to be discovered, but that doesn't matter, it will be there. In this respect is seems to me that "no new errors" requirement has no benefit over "no errors" requirement, because the latter one does everything the prior one and even more. When it is possible to whitelist errors then there's no reason for us to allow any errors in the output. Implementation note: "no new errors" is more an rpmguard's task than rpmlint's. We can implement that as an rpmguard check and put it to introspection tests, that's possible. But it's not needed if we agree on "no errors" requirement. = Responsibilities and permissions = 5. "Introspection tests failures may be waived by:" 5a. Should Release Engineering be there? 5b. Should we add the new proventesters group? Thanks for ideas. Kamil Paral [1] https://fedoraproject.org/wiki/User:Kparal/Proposal:Package_update_acceptance_test_plan [2] https://fedoraproject.org/wiki/Package_update_acceptance_criteria -- test mailing list test@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
