Hi!
In order to keep test execution efficient when number of test
cases grows, it is crucial to maintain corresponding metadata,
which define some aspects of how the test coverage is executed.
For example limiting environment combinations where the test is
relevant or selecting a subset of important test cases for quick
verification of essential features when testing a security update.
Within the BaseOS QE team we were thinking (for a long time) about
an efficient metadata solution which would cover our use cases and
would be open source. Recently we've been involved in the Upstream
First initiative which increased the need for an open metadata
solution which would enable us to more easily share test code
between Red Hat Enterprise Linux and Fedora.
We've put together a draft solution which covers some of the most
important stories we've gathered so far. It does not cover all use
cases and it is not complete. In this early stage we would like to
invite others who might have similar use cases to gather your
feedback, share your experience or even join the project:
https://fedoraproject.org/wiki/Flexible_Metadata_Format
The page lists some of our core user stories as well as a couple of
real-life examples to demonstrate proposed features of the format.
Can you see similar user stories in your team? Is this something
that could be useful for you as well? Do you know of a different
solution for these use cases? Any other relevant ideas?
To illustrate where we could be heading: In the ideal future there
could be just a single test case for a particular feature stored
in public with a single set of metadata attached close to the test
code and together used for testing in both upstream and downstream
without need to duplicate the test code (maintain both copies).
This proposal does not suggest in any way to replace tests.yml [1]
files defined by the Standard Test Interface. The new format could
serve as an extension for selecting the right tests to be executed
(e.g. filtering tests by tag instead of listing them manually).
Looking forward to your feedback!
psss...
[1] https://fedoraproject.org/wiki/CI/Tests
In order to keep test execution efficient when number of test
cases grows, it is crucial to maintain corresponding metadata,
which define some aspects of how the test coverage is executed.
For example limiting environment combinations where the test is
relevant or selecting a subset of important test cases for quick
verification of essential features when testing a security update.
Within the BaseOS QE team we were thinking (for a long time) about
an efficient metadata solution which would cover our use cases and
would be open source. Recently we've been involved in the Upstream
First initiative which increased the need for an open metadata
solution which would enable us to more easily share test code
between Red Hat Enterprise Linux and Fedora.
We've put together a draft solution which covers some of the most
important stories we've gathered so far. It does not cover all use
cases and it is not complete. In this early stage we would like to
invite others who might have similar use cases to gather your
feedback, share your experience or even join the project:
https://fedoraproject.org/wiki/Flexible_Metadata_Format
The page lists some of our core user stories as well as a couple of
real-life examples to demonstrate proposed features of the format.
Can you see similar user stories in your team? Is this something
that could be useful for you as well? Do you know of a different
solution for these use cases? Any other relevant ideas?
To illustrate where we could be heading: In the ideal future there
could be just a single test case for a particular feature stored
in public with a single set of metadata attached close to the test
code and together used for testing in both upstream and downstream
without need to duplicate the test code (maintain both copies).
This proposal does not suggest in any way to replace tests.yml [1]
files defined by the Standard Test Interface. The new format could
serve as an extension for selecting the right tests to be executed
(e.g. filtering tests by tag instead of listing them manually).
Looking forward to your feedback!
psss...
[1] https://fedoraproject.org/wiki/CI/Tests
_______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx
