----- "Jesse Keating" <jkeating@xxxxxxxxxx> wrote: > On Wed, 2010-04-21 at 10:13 -0400, Kamil Paral wrote: > > Hello, > > this is a final call for comments to our Package Update > > Acceptance Test Plan [1]. This test plan is tightly related > > to Package Update Acceptance Criteria [2] approved by FESCo. > > It basically says how we, the QA team, will decide whether > > a package update should be accepted or rejected. It also > > tries to map the requirements of the policy into an actionable > > test plan. > > > > If you have any comments about the test plan, please post > > them now, so we can include them in the document. > > > > > > I'm a bit concerned about the test environment part. We have already > determined in the work that WWoods is doing for depcheck, that we do > need to consider all the pending updates when testing a particular > pending update. It would make sense to me to follow the same logic > here. The Test Environment section tries to say that we are mainly interested in testing changes that would happen for users with fully updated system, because that is the most common case and we can't simply test all the possibilities. Good example is Package Sanity Test, where we want to test upgrading from the most recent package in 'updates' to the package coming to 'updates-testing'. We are not really interested in upgrading from one package in 'updates-testing' to another package in 'updates-testing', because that's not what's gonna happen for most of our users. Of course some tests have a little different scope. Repo sanity (the depcheck) tests repository contents rather than packages on a system. That means that the third bullet point not really applies to it. So it really depends on a particular test case, but I think generally the idea of Test Environment should be valid as it is. > > As far as Responsibilities and Permissions, as others have said we > could > collapse the QA and Releng into the proventesters group. Also we > should > add the bodhi ticket author, as the person who owns the package, the > person who did the build, and the person who submitted the update can > all be different people. Thanks, added. > > As far as Schedule we should make it clear at what phase the test > runs. > I think we want it to run post-update-submission, and must be > completed > before the request to push to either -testing or stable is > "accepted". > We talked with Luke a bit about this and how to handle it within > koji/bodhi and have a game plan. Well I'm not really sure where's the best place to define this. I'm not even sure the Schedule section should be part of the test plan. Because things you talk about are also described here: https://fedoraproject.org/wiki/User:Kparal/Proposal:Package_update_policy (which is not approved yet, but this or something similar will be in the future). It seems to me like a duplication of places where the definition is. Ideas? -- test mailing list test@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
