#154: Tracker: critical path test case creation
--------------------------+-------------------------------------------------
Reporter: adamwill | Owner: adamwill
Type: enhancement | Status: new
Priority: major | Milestone: Fedora 15
Component: Wiki | Version:
Resolution: | Keywords:
--------------------------+-------------------------------------------------
Comment (by jlaska):
Replying to [comment:16 adamwill]:
> The category name for specific source packages should be:
>
> Package_(srcrpmname)_Test_Cases
That works fine. I think the "Package_" prefix is not needed, but it
certainly doesn't hurt anything to have it (just longer wiki page names
"Category:Package_mobile-broadband-provider-info_Test_Cases" vs "Category
:Mobile-broadband-provider-info_Test_Cases", but that's certainly not
horrible).
> same reasoning as my initial proposal for the test case name: we need to
specify that (srcrpmname), in this context, is specifically a source RPM
name. I can imagine, for instance, we might want to create a generic
category for Python test packages, called Category:Python_Test_Cases, and
put test cases into that category which are not actually test cases for
the Python .src.rpm. So we'd also have a
Category:Package_python_Test_Cases . That could probably be a member of
Category:Python_Test_Cases , but not vice versa.
>
> I think the page name of each test case can be a bit more flexible and
we don't really need to worry about defining it, the categories are the
crucial thing.
Definitely!
> Do we also want to suggest a category for each .src.rpm along the lines
of:
>
> Category:Package_python_core_Test_Cases
>
> ? Rationale: I'm thinking about how we want the bodhi or fedora-easy-
karma listing for each package to look. I'm thinking of something like:
>
> -------------------------------------------------------
>
> Update: python-2.7.9-blahblah
>
> This update does blah blah foo foo whee whee poop.
>
> The following test cases can be used to check the functionality of this
package.
>
> '''Critical path tests'''
>
> These tests verify critical path functionality (link to critical path
process page) and should take highest priority:
>
> * links to all tests in Category:Package_python_Test_Cases *and*
Category:Critical_path_Test_Cases
>
> '''Core functionality tests'''
>
> These tests verify core functionality of the package and should take
highest priority after critical path tests:
>
> * links to all tests in Category:Package_python_core_Test_Cases (but not
Category:Critical_path_Test_Cases)
>
> '''Remaining tests'''
>
> These tests verify non-critical path, non-core functionality:
>
> * links to all other tests (in Category:Package_python_Test_Cases but
not Category:Package_python_core_Test_Cases or
Category:Critical_path_Test_Cases)
>
> ------------------------------------------------------
> ------------------------------------------------------
>
> The thinking being that some packages will grow fairly large sets of
tests (abrt already has, for instance) and it would help to isolate core
functionality tests for rapid sanity checking.
The way I read your organization above is that the front-ends will present
the tests by priority (critical path, core, *). I really like this
visualization, but wonder if it is outside the scope for the first pass.
Meaning, I think I'd want to see more focus on just getting critpath tests
defined for as many critpath packages, rather than spending time on non-
critpath or non-core stuff. With the setup you've designed, it doesn't
exclude people from creating non-critpath tests.
An aside, I think the previously discussed Category: organization would
lend better to allowing maintainers to prioritize tests themselves. For
example ...
bodhi and f-e-k will look for wiki pages under
Category:Package_%{sourcerpm}_Test_cases. Any tests in that category
would be listed, any tests in sub-categories of that page will be
presented as you demonstrate above. So to achieve your desired output,
you might organize tests like ...
{{{
Category:Package_python_Test_Cases
|
+--> Category:Critpath_python_Test_Cases
| |
| +-> QA:Testcase_python_do_something_1 [1]
| +-> QA:Testcase_python_do_something_2 [1]
| +-> QA:Testcase_python_do_something_3 [1]
|
+--> Category:Core_python_Test_Cases
| |
| +-> QA:Testcase_python_do_something_4
| +-> QA:Testcase_python_do_something_5
|
+--> Category:Low_python_Test_Cases
| |
| +-> QA:Testcase_python_do_something_6
| +-> QA:Testcase_python_do_something_7
|
|--> QA:Testcase_python_do_something_8
|--> QA:Testcase_python_do_something_9
[1] Could also exist in Category:Critpath_test_cases, but not required
}}}
This would allow maintainers to group the tests as they see fit, but holds
that everything must be anchored out of
Category:Package_python_Test_Cases. We would still focus on prioritizing
development of critpath tests, but the support exists for
maintainers/packagers to organize as they feel is appropriate. Also, the
mediawiki API provides the needed methods to find and present the
information listed above.
Another something to keep in mind, any heavy/complex wiki organization we
do may require extra work when migrating data to another system (which is
a possible mid-term future scenario).
> random side observation: our naming of test case pages and categories
currently stinks.
Sure, page naming wasn't as important when we weren't attempting to
integrate it with other tools so I gather some wiki renaming may be
required. Or did it stink in another regard?
> next big concern: where do we effectively document this? All we
currently have on test cases is:
>
> https://fedoraproject.org/wiki/Category:Test_Cases
> I can add some overview and stuff to that page but it's not a terribly
visible page, really. I guess the best approach is to edit that page and
add some links to it from strategic other places, especially developer
instruction pages. Thoughts?
Agreed. Do we need an SOP/HOWTO for each of the use cases where
proventesters, or maintainers, will interact with this setup? Such as ...
1. How to create a test case
2. How to organize test cases
3. How to post results for test cases
4. ... ?
--
Ticket URL: <https://fedorahosted.org/fedora-qa/ticket/154#comment:18>
Fedora QA <http://fedorahosted.org/fedora-qa>
Fedora Quality Assurance
--
test mailing list
test@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe:
https://admin.fedoraproject.org/mailman/listinfo/test
