Hi! We've had some more discussions regarding this topic within the CI team. Here's a brief summary of the recommendation we agreed upon: Tests may be written in different ways, but are enabled in a standard way directly in the package git repository as defined by the Standard Test Interface. [1] Test code can be stored directly in the package git repository (recommended as default) or fetched from another location hosted in the Fedora infrastructure such as the Tests Namespace. [2] The decision where the test code is stored is up to the package maintainer. [1] https://fedoraproject.org/wiki/CI/Standard_Test_Interface [2] https://fedoraproject.org/wiki/CI/Share_Test_Code psss... On 1 March 2018 at 18:06, Honza Horak <hhorak@xxxxxxxxxx> wrote: > On 03/01/2018 04:00 PM, Petr Šplíchal wrote: >>>> >>>> To sum up what I've heard so far from the developer side: >>>> >>>> * I would like to enable tests for my component (yes, I want) >>>> * I will take care of them (really, I see the benefit in CI) >>>> * I want to easily collaborate on tests with qe (direct commits) >>>> * I don't want to give qe commit access to the rpms dist git >>> >>> This is quite likely the crux of the problem. >>> >>> Personally, I'm perfectly happy to give write access to any repo >>> to people who have shown that they know what they are doing. >>> We have pull requests in dist-git pagure now, and I think this is >>> a better approach: >>> 1. ask QE contributors to submit PRs >>> 2. if enough cooperation happens and trust is established, give >>> privileges to write to the repo directly, possibly with an agreement >>> that this specific person should only touch tests, and not the >>> packaging. >>> >>> I think it's perfectly fine to never get to point 2.: many many >>> upstream projects require a review from a second person, or sometimes >>> even two reviews before a PR is merged, which means that one _never_ >>> merges their own PR, and another contributor is always involved. We >>> usually don't do this in dist-git, but I'm quite sure that requiring >>> reviews for dist-git changes would raise quality and catch many silly >>> mistakes. Either way, it's nowadays possible to cooperate using a >>> single repo without fully trusting the other person, frictionlessly. >> >> >> Good point. However, if there is a qe/devel team which prefers to >> collaborate on tests in a separate repo because this is the most >> efficient way they found so far, why should we stop them and try >> to enforce a less efficient workflow? (Which they can workaround >> by a different git repo.) >> >>>> * I want to efficiently maintain tests long-term >>>> * I want to have just a single place for my tests (no duplication) >>>> * I don't want to backport new tests to old branches (enable once) >>>> * I want to easily enable testing for all supported branches >>> >>> Those four items depend strongly on the package. My thinking is >>> biased by some specific usecases (mainly systemd), but I'm sure many >>> other packages are like that: a lot of tests would be for new >>> functionality, and then the tests _are_ tied to the branch being >>> tested. >>> >>> While I agree that keeping tests separate avoids a bit of effort >>> required to update multiple branches, this effort isn't very big. If >>> the tests indeed apply cleanly to all branches, then it's just a >>> matter of doing 'git merge' or 'git cherry-pick' once per branch. >>> >>>> * I want to keep rpms dist git clean (just metadata & patches) >>> >>> Meh. >>> >>>> * I want to run all available relevant tests (not to list them) >>>> * I want to execute set of tests based on a tag (e.g. Tier1) >>> >>> Those two sound like stuff that should be solved in the tooling, >>> whatever is used to run tests. >>> >>>> * I want an easy way to clone tests (fedpkg clone tests/pkg) >>> >>> Tests alone make no sense, you need something to test, and >>> cloning one repo is easier than cloning two repos, so there's no >>> advantage to the split here. >> >> >> But "fedpkg clone tests" is easier than cloning from a "random" >> git repository where I was forced to save my tests because I was >> not allowed to save them in Fedora tests namespace. >> >>>> I believe the tests namespace resolves them all. >>> >>> None of those arguments convince me that separate repos for tests are >>> a good default. Sure, there are specific packages where this will make >>> sense, or specific packagers with idiosyncratic workflows, but I'd >>> rather "start small", with the simple configuration, and only move >>> specific packages to the more complicated setup once that proves to >>> be required. >> >> >> Why default? Test namespace should be an option. Not the default. >> Storing tests directly in dist git is and will be possible. >> Anybody who finds this as a better way can do so. But why >> enforcing this approach to all? > > > +1 Exactly! I don't really see any reason why to force people to follow one > way when there are obvious disadvantages. All concerns against tests/ > namespace in this thread are based on expectations that it won't work, but > how can anybody know at this point? The only proof we have is that the > tests/ namespace works internally. > > I believe that what we really need here is flexibility -- let people find > the way that works best for them. > > Honza > > _______________________________________________ > devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx > To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx _______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx
