On Tue, 2010-12-21 at 18:12 -0500, James Laska wrote: > > the first isn't particularly specific to this, but it was a prerequisite > > that I discovered was missing: it's a guide to test case creation in > > general, explaining the actual practical process of how you create a > > test case, and the best principles to consider in doing it. > > Nice job here, this is something that's difficult to explain if you've > done it a lot, but I think you've captured the key points. If possible, > it might be helpful to highlight a few existing examples that stand out > for the different characteristics you mention (comprehensive, but able > to stand the test of time). Thanks. I'll see if I can find some and add them. > Another thought, any reason that we wouldn't want to keep all wiki tests > in the QA: namespace (and with the prefix QA:Testcase_)? The door is > left open for other names, I wonder if we want to cut that off ahead of > time to keep our sanity by having all tests in the same namespace? I was a bit unsure on that one. I think I thought of some possible scenario where you might want to write a test case in a different name space, but I'm not entirely sure I remember what it was. I can just change it to say test cases should always go in the qa namespace, I guess. > The page also talks about using [[Category:Test_Cases]]. I worry if we > are too lax in categorizing new tests we'll end up with a large amount > of random tests in the main [[Category:Test_Cases]] making it a > maintenance nightmare to cleanup that category. Should we instead > direct users to your other page > (https://fedoraproject.org/wiki/User:Adamwill/Draft_QA_SOP_package_test_plan_creation) for guidance on categorizing test cases? This was something I wanted to call out for discussion and forgot - so far we've put all test cases directly into the Test_Cases category, but like you, I'm worried that really won't scale. I did wonder whether others would agree we should stop doing that and instead have them usually go into a more specific category which in turn would be a sub-category of Test_Cases, and only have test cases be members of Test_Cases directly if it really made no sense to have them in a more specific category. > > The second is what's really specific to this subject. It describes how > > to create a set of test cases for a particular package, and a proposed > > standardized categorization scheme which will allow us to denote test > > cases as being associated with specific packages, and also denote them > > as concerning critical path functionality. > > I think I mentioned this previously, in the section 'Preparation', I > appreciate the distinction of 'core' and 'extended'. But I it resonates > with me better under the context of test "priority". I don't see why we > can't keep using the terms 'core' and 'extended', but just want to > clarify their purpose. They're intended to add some sense of execution > priority to a list of test cases, right? Where critpath comes first, > then core, then extended, then other? Also, you describe > categorizing/grouping test cases in more detail below, maybe just link > to that instead? well, the idea is that the two are complementary: if you're going to separate the test cases into 'core' and 'extended' groups, then why not identify which functionality is 'core' and which is 'extended' at the time you're identifying functionality to write test cases for? I'm not quite sure what your proposal is here - could you draft it up in terms of an actual change to the page so I can see it more clearly? thanks! > In the section, 'Simple (required)', would it help to add a link to > http://fedoraproject.org/wiki/BugZappers/CorrectComponent#Which_component_is_it.3F (or similar page). Something to help testers find the right src.rpm name of the component under test? Side note, this might also be a maintenance task we can define where I, or anyone interested, could manually scrub (or script) finding Categories:Test_Cases searching incorrectly named category pages. > > Also in 'Simple (required)', we don't tell the author to add their > 'Category:Package_${sourcename}_test_cases' to 'Category:Test_Cases'. I > think we want all newly created package categories anchored under > 'Category:Test_Cases'. yup, indeed, oversight - will add it. thanks! > General comment. I know we've got an eye towards integrating this work > with bodhi and/or f-e-k. Until that work is complete, I wonder if those > notes will introduce confusion/speculation. Should we leave out the > bits about possible future tool integration until such support is > active? possibly. I was meaning those bits to be read simply as a potential illustration of programmatic use of the categories to illustrate why consistent categorization is important, but if you think it's confusing, we could take it out. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org http://www.happyassassin.net -- devel mailing list devel@xxxxxxxxxxxxxxxxxxxxxxx https://admin.fedoraproject.org/mailman/listinfo/devel
