> > > My worries are basically that this mechanism is hiding the
> > > tests from the package maintainers. It splits the concerns
> > > between people maintaining the artifacts and people
> > > maintaining the tests, which is exactly what the original
> > > plan to bring CI was *not* about.
> >
> > I do not see a split here but rather potential for enhanced or
> > extended collaboration. Repositories in the tests namespace
> > can be set up with more open access than rpms git. In this way
> > devel can invite qe for direct collaboration on tests.
>
> I don't see how that's not possible with tests being in
> dist-git.
What I've heard from devels is that they often don't feel like
giving direct commit access to the rpms dist git to QE as spec
files and packaging in general is a big responsiblity. But they
are quite OK with directly collaborating with QE on test coverage.
> If the tests are defined in dist-git and adding new tests means
> committing these changes to dist-git's yaml file, how does that
> solve the concern about "noise" in the git commits?
Not necessarily. You could use advanced metadata to select desired
tests only once. The very first proof of concept of using Flexible
Metadata Format for testing selinux quite nicely illustrates this:
https://src.fedoraproject.org/tests/selinux/pull-request/1
https://github.com/psss/fmf
Instead of listing individual tests you could say: Run all tests
for this selinux component + all Tier1 selinux integration tests:
fmf --key test --filter 'component:libselinux | tags:Tier1'
For tests covering multiple components the benefit of the tests
namespace approach is unquestionable. However, I see benefit of
the approach for single components as well (and developers seem
to often feel the same): That is not having to cherry-pick/merge
test coverage between supported branches. So even for these cases
there is the benefit of sharing the test code.
I believe that extending the integration testing coverage is one
the most essential goals of Fedora CI. This is the crucial piece
of the puzzle how to make an Always Ready Operating System: It is
to ensure that packages play nicely together. So I guess there
will be quite many tests which will benefit from this approach.
Do we really want/need to assess every new request for a tests
repo if it is good-enough for the test namespace? Will there be
a committee to decide whether a particular set of tests has a
potential to be used by other components in the future or not? :)
To sum up what I've heard so far from the developer side:
* I would like to enable tests for my component (yes, I want)
* I will take care of them (really, I see the benefit in CI)
* I want to easily collaborate on tests with qe (direct commits)
* I don't want to give qe commit access to the rpms dist git
* I want to efficiently maintain tests long-term
* I want to have just a single place for my tests (no duplication)
* I want to keep rpms dist git clean (just metadata & patches)
* I want to easily enable testing for all supported branches
* I don't want to backport new tests to old branches (enable once)
* I want to run all available relevant tests (not to list them)
* I want to execute set of tests based on a tag (e.g. Tier1)
* I want an easy way to clone tests (fedpkg clone tests/pkg)
I believe the tests namespace resolves them all.
psss...
> > > tests from the package maintainers. It splits the concerns
> > > between people maintaining the artifacts and people
> > > maintaining the tests, which is exactly what the original
> > > plan to bring CI was *not* about.
> >
> > I do not see a split here but rather potential for enhanced or
> > extended collaboration. Repositories in the tests namespace
> > can be set up with more open access than rpms git. In this way
> > devel can invite qe for direct collaboration on tests.
>
> I don't see how that's not possible with tests being in
> dist-git.
What I've heard from devels is that they often don't feel like
giving direct commit access to the rpms dist git to QE as spec
files and packaging in general is a big responsiblity. But they
are quite OK with directly collaborating with QE on test coverage.
> If the tests are defined in dist-git and adding new tests means
> committing these changes to dist-git's yaml file, how does that
> solve the concern about "noise" in the git commits?
Not necessarily. You could use advanced metadata to select desired
tests only once. The very first proof of concept of using Flexible
Metadata Format for testing selinux quite nicely illustrates this:
https://src.fedoraproject.org/tests/selinux/pull-request/1
https://github.com/psss/fmf
Instead of listing individual tests you could say: Run all tests
for this selinux component + all Tier1 selinux integration tests:
fmf --key test --filter 'component:libselinux | tags:Tier1'
For tests covering multiple components the benefit of the tests
namespace approach is unquestionable. However, I see benefit of
the approach for single components as well (and developers seem
to often feel the same): That is not having to cherry-pick/merge
test coverage between supported branches. So even for these cases
there is the benefit of sharing the test code.
I believe that extending the integration testing coverage is one
the most essential goals of Fedora CI. This is the crucial piece
of the puzzle how to make an Always Ready Operating System: It is
to ensure that packages play nicely together. So I guess there
will be quite many tests which will benefit from this approach.
Do we really want/need to assess every new request for a tests
repo if it is good-enough for the test namespace? Will there be
a committee to decide whether a particular set of tests has a
potential to be used by other components in the future or not? :)
To sum up what I've heard so far from the developer side:
* I would like to enable tests for my component (yes, I want)
* I will take care of them (really, I see the benefit in CI)
* I want to easily collaborate on tests with qe (direct commits)
* I don't want to give qe commit access to the rpms dist git
* I want to efficiently maintain tests long-term
* I want to have just a single place for my tests (no duplication)
* I want to keep rpms dist git clean (just metadata & patches)
* I want to easily enable testing for all supported branches
* I don't want to backport new tests to old branches (enable once)
* I want to run all available relevant tests (not to list them)
* I want to execute set of tests based on a tag (e.g. Tier1)
* I want an easy way to clone tests (fedpkg clone tests/pkg)
I believe the tests namespace resolves them all.
psss...
_______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx
