On Thu, Mar 01, 2018 at 02:44:37PM +0100, Petr Šplíchal wrote: > > > > My worries are basically that this mechanism is hiding the > > > > tests from the package maintainers. It splits the concerns > > > > between people maintaining the artifacts and people > > > > maintaining the tests, which is exactly what the original > > > > plan to bring CI was *not* about. > > > > > > I do not see a split here but rather potential for enhanced or > > > extended collaboration. Repositories in the tests namespace > > > can be set up with more open access than rpms git. In this way > > > devel can invite qe for direct collaboration on tests. > > > > I don't see how that's not possible with tests being in > > dist-git. > > What I've heard from devels is that they often don't feel like > giving direct commit access to the rpms dist git to QE as spec > files and packaging in general is a big responsiblity. But they > are quite OK with directly collaborating with QE on test coverage. > > > If the tests are defined in dist-git and adding new tests means > > committing these changes to dist-git's yaml file, how does that > > solve the concern about "noise" in the git commits? > > Not necessarily. You could use advanced metadata to select desired > tests only once. The very first proof of concept of using Flexible > Metadata Format for testing selinux quite nicely illustrates this: > > https://src.fedoraproject.org/tests/selinux/pull-request/1 > https://github.com/psss/fmf > > Instead of listing individual tests you could say: Run all tests > for this selinux component + all Tier1 selinux integration tests: > > fmf --key test --filter 'component:libselinux | tags:Tier1' > > For tests covering multiple components the benefit of the tests > namespace approach is unquestionable. However, I see benefit of > the approach for single components as well (and developers seem > to often feel the same): That is not having to cherry-pick/merge > test coverage between supported branches. So even for these cases > there is the benefit of sharing the test code. > > I believe that extending the integration testing coverage is one > the most essential goals of Fedora CI. This is the crucial piece > of the puzzle how to make an Always Ready Operating System: It is > to ensure that packages play nicely together. So I guess there > will be quite many tests which will benefit from this approach. > > Do we really want/need to assess every new request for a tests > repo if it is good-enough for the test namespace? Will there be > a committee to decide whether a particular set of tests has a > potential to be used by other components in the future or not? :) > > To sum up what I've heard so far from the developer side: > > * I would like to enable tests for my component (yes, I want) > * I will take care of them (really, I see the benefit in CI) > * I want to easily collaborate on tests with qe (direct commits) > * I don't want to give qe commit access to the rpms dist git This is quite likely the crux of the problem. Personally, I'm perfectly happy to give write access to any repo to people who have shown that they know what they are doing. We have pull requests in dist-git pagure now, and I think this is a better approach: 1. ask QE contributors to submit PRs 2. if enough cooperation happens and trust is established, give privileges to write to the repo directly, possibly with an agreement that this specific person should only touch tests, and not the packaging. I think it's perfectly fine to never get to point 2.: many many upstream projects require a review from a second person, or sometimes even two reviews before a PR is merged, which means that one _never_ merges their own PR, and another contributor is always involved. We usually don't do this in dist-git, but I'm quite sure that requiring reviews for dist-git changes would raise quality and catch many silly mistakes. Either way, it's nowadays possible to cooperate using a single repo without fully trusting the other person, frictionlessly. > * I want to efficiently maintain tests long-term > * I want to have just a single place for my tests (no duplication) > * I don't want to backport new tests to old branches (enable once) > * I want to easily enable testing for all supported branches Those four items depend strongly on the package. My thinking is biased by some specific usecases (mainly systemd), but I'm sure many other packages are like that: a lot of tests would be for new functionality, and then the tests _are_ tied to the branch being tested. While I agree that keeping tests separate avoids a bit of effort required to update multiple branches, this effort isn't very big. If the tests indeed apply cleanly to all branches, then it's just a matter of doing 'git merge' or 'git cherry-pick' once per branch. > * I want to keep rpms dist git clean (just metadata & patches) Meh. > * I want to run all available relevant tests (not to list them) > * I want to execute set of tests based on a tag (e.g. Tier1) Those two sound like stuff that should be solved in the tooling, whatever is used to run tests. > * I want an easy way to clone tests (fedpkg clone tests/pkg) Tests alone make no sense, you need something to test, and cloning one repo is easier than cloning two repos, so there's no advantage to the split here. > I believe the tests namespace resolves them all. None of those arguments convince me that separate repos for tests are a good default. Sure, there are specific packages where this will make sense, or specific packagers with idiosyncratic workflows, but I'd rather "start small", with the simple configuration, and only move specific packages to the more complicated setup once that proves to be required. Zbyszek _______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx
