На 18.12.2013 20:09, Tim Flink написа:
On a side note, it might be interesting to find out what percentage of
packages are running things in %check. I don't know what we would do
with that metric, but I think it would be interesting :)
I have something in the works which can be modified to extract this info. It
should be fairly easy to make a good guesstimate of the presence/non presence of
test suites as well. I think I'll start from here.
Does Fedora have its own CI infrastructure coupled with Koji ?
That depends on what you consider CI to be, I guess. I consider both
autoqa and taskotron to be forms of CI but not the exact same thing as
travis or jenkins.
Both autoqa and taskotron are designed to run various things based on a
package's state in either koji or bodhi.
Thanks, I have to look into these. I gave Travis as example because it is well
known and because somebody else decided to make use of it in python-bugzilla.
I'm still unclear on what you're looking to do. Are you talking about
looking for test suites in package upstreams and running those tests,
regardless of whether they're run in %check during build? Are you
talking about creating and maintaining an out-of-band set of unit tests
for packages without an upstream unit test suite? Are you talking about
creating a set of package-specific functional/integration test suites
that are run when packages are built?
Pretty much all of these in the long term but I wanted to get a general feeling
so I can concentrate the effort on something specific.
* For packages which have upstream test suites (e.g. parted) - contribute more
tests where necessary; Ideally covering bugs reported against Fedora or RHEL for
which there isn't a test case.
* For packages without upstream test suite my preference is to create one and
contribute it back upstream. If upstream doesn't accept it and the package is
critical enough maybe maintain that in the Fedora branch.
* Functional/integration test suites (e.g. using Beaker tasks) are less priority
goal for the moment but also need to be considered. For some packages these are
more suitable (e.g. anaconda), for others maybe not (e.g. pykickstart seems to
have fairly good amount of unit tests).
Whatever the test suite(s) come out to be we need an environment(or
environments) where to execute them and a method of starting the suite. I can
see how some test cases are part of %check and can halt the build process and
others are started async after the build and are left for Fedora QA and devel to
investigate.
In the case of Travis, test runs are triggered by git commits and Travis is not
connected to any build infrastructure from what I know. It provides the results
to whoever is interested in them.
--
test mailing list
test@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe:
https://admin.fedoraproject.org/mailman/listinfo/test
