On 14 February 2018 at 18:18, Adam Williamson <adamwill@xxxxxxxxxxxxxxxxx> wrote: > On Wed, 2018-02-14 at 17:28 +0100, Petr Šplíchal wrote: >> Hi! >> >> During the last days there have been concerns raised regarding >> what is an appropriate content for the tests namespace. [1] My >> original idea was to enable sharing tests even across branches >> of the same component, not only for tests to be used by >> completely different packages. The initial examples might have >> been a bit misleading in this respect. One of the main points >> still holds: >> >> > * Tests might follow a different branching pattern than the >> > dist-git repo, leading to code duplication >> >> From the feedback from developers I feel they always keep on >> mind and care a lot about the maintenance costs. So it >> perfectly makes sense to me if they want to keep and maintain >> tests in a separate repo instead of merging/cherry-picking >> between dist-git branches. >> >> When, for a particular package, it is the most efficient way to >> maintain tests in a separate repo why should we discourage from >> this approach? There are packages where it makes more sense to >> store test code directly in dist-git. And it is still an >> option. But why should we enforce this for all? >> >> Please share your thoughts and real-life examples. For those >> who are not familiar with the topic there is a new wiki page >> with a summary of the Share Test Code approach [2]. > > I don't have any problem with the concept *in itself*; in fact I > know of another reason to have something like it. That is > 'generic' tests, tests we want to run on all packages, or at > least a very large set of packages - like the tests currently > running in Taskotron, which are generally run on all packages > (rpmgrill, rpmdeplint...) or a large subset (the Python tests). Yes, sharing tests useful for multiple packages makes sense in general. Test code for generic tests can be stored basically anywhere but not in specific components. However I see some additional benefits of having a dedicated tests namespace: * Source of inspiration and real-life examples (at one place) * Potential for workflow automation (tests CI, fedpkg --tests) * Easy cherry-picking/backporting upstream/downstream tests The difference I see when comparing these shared tests to generic tests mentioned above like rpmdeplint is that you don't want to reference these from the individual package repos (e.g. using the Standard Test Interface), but just run them on all packages. If the expectation is to run tests only on a subset of rpms then a question comes: How to store the list of relevant components or criteria to decide which packages are to be tested or not. Here the Flexible Metadata Format might come handy in the future: https://github.com/psss/fmf > What I did think was maybe a concern is that we should figure > out in advance the squishy human consequences of different > technical approaches. > > Basically this boiled down to "who is responsible for these > 'shared' tests, and who gets to decide which 'shared' tests > can/should block packages?" > > Looking at the proposal, I think it actually has at least > workable initial explicit/implicit answers to this, if I'm > reading it correctly. Anyone can create a shared test > repository (and is therefore "responsible" for maintaining it), > but the definition of which tests are run on which packages > remains in the package repositories. Thus the package > maintainer(s) retain the ultimate choice over which tests are > run against their packages (and thus can pick which shared tests > are run, and disable shared tests if they are no longer properly > testing their package). Yes. This is exactly true. The maintainer of the package repo has always the last word on what is to be executed for qualification. It can be done by referencing only suitable tests from the shared repo or selecting an appropriate branch or even a specific commit to ensure that future test development does not break stable gating (and only manually update reference after review of shared test updates) or even completely disable tests by removing the reference. > The question 'who decides which tests block which packages' is > left a bit up in the air, but in fact no more so than it already > was for package-specific tests... Right. Do we want to change this? Specify this more strictly? Like exactly defining requirements which an Always Ready Operating System has to meet? And then mapping these requirements to the test coverage? (Here again FMF mentioned above might come handy.) psss... > -- > Adam Williamson > Fedora QA Community Monkey > IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net > http://www.happyassassin.net _______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx
