On Sun, May 24, 2020 at 12:18 PM William Roberts <bill.c.roberts@xxxxxxxxx> wrote: > > On Fri, May 22, 2020 at 2:40 AM Ondrej Mosnacek <omosnace@xxxxxxxxxx> wrote: > > > > On Thu, May 21, 2020 at 4:12 PM William Roberts > > <bill.c.roberts@xxxxxxxxx> wrote: > > > On Thu, May 21, 2020 at 7:58 AM Ondrej Mosnacek <omosnace@xxxxxxxxxx> wrote: > > > > > > > > On Thu, May 21, 2020 at 2:52 PM Stephen Smalley > > > > <stephen.smalley.work@xxxxxxxxx> wrote: > > [...] > > > > > Last I looked, his script builds and installs the userspace code on > > > > > top of the Fedora libraries and programs (make LIBDIR=... install...) > > > > > and then runs the testsuite. That was my suggestion. > > > > > > > > Ah, yes, I can see that line now. Sorry, somehow I missed it before. > > > > > > > > > While it is the > > > > > kernel testsuite, it exercises a lot of SELinux userspace > > > > > functionality that isn't tested by the userspace tests. > > > > > > > > OK, I suppose it's better than nothing... > > > > > > > > > > Stephen pointed out the additional ways userspace gets tested, and > > > perhaps my title and description > > > of the patch could be better. But the main point is to increase the > > > test coverage > > > and perform the testing steps we expect are done before a release in > > > the CI. We should have > > > the testing coverage and the confidence to release userspace from master at any > > > point. We also have forward facing proof that tests are being executed > > > and we can make sure > > > nothing regresses. > > > > > > My ultimate goal here, is to help make sure that if Petr gets hit by a > > > bus, releases will > > > move forward without worry and without any change in quality among the various > > > maintainers. > > > > > > Additionally, we pick up some cross project testing and can find other > > > surprises. > > > > Ah, so you want to move an integration test, which we would normally > > run only before release, down to per-commit testing, which is fine > > because it is quite fast... OK, it started to make sense to me now :) > > Exactly, plus we pick up a little more test coverage on the userspace bits > by swapping out what's installed in the VM with the current build and running > the tests. The speed is less important, it's just fast enough where our CI isn't > going to take years to complete. CI doesn't need to be super snappy per se, > but it also cannot take a fortnight. > > > > > If I find time I'll have a closer look at the scripts. I already see > > some tiny possible improvements... I have Nicolas's last comments > addressed and staged, so ill wait a few days and see what you come > back with and re-send a V3. This is looking good to me. Only questions I have are: 1) Have you confirmed that a testsuite failure within the VM gets correctly propagated up and treated as a failure by travis-ci itself? 2) Have you seen any problems with instability in running of the tests due to the additional complexity and time? I've certainly already seen instances where travis-ci of selinux or selinux-testsuite fails randomly due to timeouts or something when downloading external components.
