On Fri, Feb 23, 2018 at 10:19:35AM +0100, Petr Šplíchal wrote: > On 22 February 2018 at 15:50, Pierre-Yves Chibon <pingou@xxxxxxxxxxxx> wrote: > > On Wed, Feb 14, 2018 at 05:28:20PM +0100, Petr Šplíchal wrote: > >> Hi! > >> > >> During the last days there have been concerns raised regarding > >> what is an appropriate content for the tests namespace. [1] My > >> original idea was to enable sharing tests even across branches of > >> the same component, not only for tests to be used by completely > >> different packages. The initial examples might have been a bit > >> misleading in this respect. One of the main points still holds: > >> > >> > * Tests might follow a different branching pattern than the > >> > dist-git repo, leading to code duplication > >> > >> From the feedback from developers I feel they always keep on mind > >> and care a lot about the maintenance costs. So it perfectly makes > >> sense to me if they want to keep and maintain tests in a separate > >> repo instead of merging/cherry-picking between dist-git branches. > >> > >> When, for a particular package, it is the most efficient way to > >> maintain tests in a separate repo why should we discourage from > >> this approach? There are packages where it makes more sense to > >> store test code directly in dist-git. And it is still an option. > >> But why should we enforce this for all? > > Thanks for sharing your thoughts, Pierre. > > > My worries are basically that this mechanism is hiding the tests > > from the package maintainers. It splits the concerns between > > people maintaining the artifacts and people maintaining the > > tests, which is exactly what the original plan to bring CI was > > *not* about. > > I do not see a split here but rather potential for enhanced or > extended collaboration. Repositories in the tests namespace can be > set up with more open access than rpms git. In this way devel can > invite qe for direct collaboration on tests. I don't see how that's not possible with tests being in dist-git. > That's one of the > reasons why we should try to fix the non-packager access as soon > as possible. > > https://pagure.io/fedora-infrastructure/issue/6361 Agreed and being worked on. > > The idea has always been to bring the tests where the code > > lives, so that both can be worked on as one, to make tests be a > > concern of the package maintainer very much like updating the > > artifact (rpms, image..) is, while getting support from QE for > > them (the tests). > > I see the benefits of having code with tests at the very same > place, I find this very useful especially for unit tests in the > project upstream. However, the situation here is a bit different > in two respects: > > 1) The rpms dist git does not include package code itself but > rather build metadata only (mainly spec files describing how to > build the package) so it makes very good sense to host at this > place testing metadata only (how to test the package). This is > quite nicely consistent approach. I still believe it creates a separation of concerns that is in opposition with the original idea. > 2) As far as I can see, within the Always Ready Operating System > effort we are not looking to cover every and each feature in all > packages but rather ensuring the OS works as a whole. Thus basic > functionality and integration tests which guard the stability of > the packages are more appropriate for testing. These do not need > to be kept so close to the package code. > > I believe we don't want to duplicate upstream unit testing here. > We should probably clarify more which test types are appropriate > or expected in the Fedora CI. As part of the docs consolidation I > added a brief section about test types to the main CI portal: > > https://fedoraproject.org/wiki/CI#Test_Types > > Does it decribe well what we are looking for? Shall we extend it? > > > In addition, this is what I fear most: > > The tests for the package are stored elsewhere, we're not sure > > where (the tests namespace, a random git repo on the > > internet...), the packager update package and the tests fail. > > I see no hiding of tests from maintainers: Tests are clearly > referenced and discoverable Referenced maybe, discoverable, much less than if they are right there. > from the rpms dist git as defined by > the Standard Test Interface and devels have access, see: > > https://fedoraproject.org/wiki/CI/Share_Test_Code#Examples If the tests are defined in dist-git and adding new tests means committing these changes to dist-git's yaml file, how does that solve the concern about "noise" in the git commits? It seems to me that whether the commit touches just the yaml file or the yaml and tweak the tests themselves, the outcome is the same. > Devel decides which tests are good for testing, it does not matter > where the test code lives. However there needs to be trust to the > referenced repo. But this is the same as with any other open > source project. > > > What do you think will happen? > > a) the packager will look for $random_place_of_the_internet > > where tests are and spend time trying to fix them. > > b) the packager will turn off the tests because they get in the way > > > > If the packager wants to turn off the tests, they will have to > > go through dist-git to do it, they may very well end up turning > > the tests off anyway, but if the tests are right there, they may > > as well have a quick look at them to see if they can fix them > > quickly before deciding. > > It's not random_place_of_the_internet, it's well selected and > trusted source where devel can (usually) directly contribute. I > meet devels around who see benefit in having tests enabled. So > I don't share your concern about turning the test off so easily. > > > In addition, if the packager turn the tests off and the people > > maintaining the tests do not realize that, they will end up > > spending time maintaining $somewhere_else tests that aren't > > being used. > > > > Having them interact directly with the dist-git repo will > > increase the chances that they see/realize when something is > > turned off. > > Working on something that is not being used supposes there is no > communication/collaboration between devel and qe. And I doubt that > enforcing inefficient workflow with test code duplication and less I don't see how this duplication works, if you need to touch the yaml file anyway, there will be just as much "duplication". > ACL flexibility will help to increase it. How are ACLs less flexible than with any of the open source project out there? You start by submitting pull-request and at one point the people trust you enough to grant you direct commit access. > I believe we should set > up the workflow as smooth as possible to encourage better > collaboration. > > > If that means we have less tests in dist-git because the > > maintainers do not want them, I'd say so be it. > > > > In the long term this is their loss, they are the ones who will > > get the bug reports and will have to deal with them, they are > > the ones who will have to dive into old code rather than going > > back into something that is still fresh in their mind. > > > > As long as, it is clear that they have been approached and that > > it is their choice to not merge pull-requests adding tests, I > > think the people offering to help should not be the ones blamed. > > The question is not about devels who do not want tests enabled in > dist-git but about those who do not want to inefficiently maintain > the test code there. So diving into old code does not apply here. I don't see what's inefficient here sorry. Pierre _______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx
