Re: Libvirt upstream CI efforts

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Thu, Feb 21, 2019 at 03:39:15PM +0100, Erik Skultety wrote:
> number of test cases (and also many unnecessary legacy test cases). An
> alternative set of functional test cases is available as part of the
> libvirt-tck framework [2]. The obvious question now is how can we build upon
> any of this and introduce proper functional testing of upstream libvirt to our
> jenkins environment at ci.centos.org, so I formulated the following discussion
> points as I think these are crucial to sort out before we move on to the test
> suite itself:

Having thought about this some more I think it would be helpful to outline
the various areas of testing libvirt is missing / could benefit from, as
I think it is broader than just running an integration test suite on the
ci.centos.org

Listing in order of the phase of development, not priority....


 - Testing by developers before code submissions

   Developers today (usually) run unit tests + syntax check before
   submission, though even that is forgotten at times.

   Ideally some level of functional testing would be commonly performed
   too.

   Amount of time devs are likely to want spend on testing will depend
   on the scope of the work being done. ie they're not going to test
   on all distros, with all integration tests for simple patches. Also
   no desire to make devs run QEMU tests on patches which are changing
   Xen code and vica-verca.

   Essentially goal is to give developers confidence that they have
   not done something terrible before submission. Not expecting devs
   to catch all bugs themselves at this stage.


 - Testing of patches posted to mailing list pre merge

   patchew.org currently monitors patch postings to libvir-list
   and imports each posting into a new github branch. In theory
   it runs syntax-check against them & reports failures but this
   has not been reliable.

   Highly desirable to have all patches go through build + unit
   tests at this point, across multiple distros. It is common for
   devs to break mingw and/or *BSD and/or macOS builds, since vast
   majority of the dev focus is Linux.  There is generally long
   enough between patch posting & review approval that build+unit
   tests should be doable with sufficient patchew worker resource.


   Extra brownie points if the build + tests ran across each
   individual patch to prove that  git bisect-ability isn't
   broken. This would require significantly more worker
   resources though. This is the only place bi-sect could
   be tested as anything beyond is too late.


   Running functional tests at this point would be beneficial,
   on the general principle that the sooner we find a problem,
   the cheaper it is to fix & less impact it has on people.
   Massively dependant on worker resource.


 - Testing of latest git master post merge

   This is where almost all of our current effort has gone.
   
   ci.centos.org does build & unit testing fully chained
   together all libvirt components on Linux + BSD using VMs

   Travis CI does testing of individual of libvirt components
   on Linux + macOS, using contaniers for Linux.

   Both of these are x86 only thus far. Through use of Debian
   cross compilers we can get non-x86 coverage for builds,
   but not much else without finding some real hardware.

   Desirable to have functional testing here to detect problems
   before they get into any formal release. Dependent on resource
   to run on ci.centos.org or Travis, or another system we might
   get access to. Still likely to be x86 only.


 - Testing during RPM builds

   When building new packages for distros, 'make check' is usually
   run. This has caught problems appearing in distros which have
   sometimes been missed by ci.centos.org.

   Desirable to have functional testing here in order to prevent
   breakage making its way into distros, by aborting the build.

   Runs on all Fedora architectures which is a big plus, since
   all earlier upstream testing resources are x86 only.

   The environment is quite restrictive as it is inside Koji/Brew
   but libguestfs test suite has shown its possible to do very
   thorough testing of Libvirt & QEMU in this context that frequently
   identifies bugs in libvirt & QEMU & kernel & other Fedora/RHEL
   components.

   Fedora has an automated system that frequently rebuilds the
   RPMs to check FTBFS (fail to build from source) status of
   packages to detect regresisons over time.


 - Testing of composed distros

   Real integration testing belongs here, as its validating the
   exact software build & deployment setup that users will ultimately
   run with.

   The test environment is more flexible than during RPM build,
   but by the time it runs the update is already in the distro
   repos, potentially breaking downstream users (libguestfs).

   Note 100% sure, but I think the Fedora CI is x86 only.



 - Testing by developers investigating CI failures

   For any of the above steps which are run by any kind of automated
   system there needs to be a way for developers to reproduce the
   same test environment in an easy manner.

   For ci.centos.org we can re-generate the VMs.
   For Travis we can pull the docker images from quay.io
   For koji/brew we can run mock locally

   None of these are a perfect match though, as they can't reproduce
   the exact hardware setup, or the load characteristics, just the
   software install setup.



It is clear we have lots of places where we should/could be doing
functional testing, and none of them is going to cover all the
bases.

The environments in which we need to be able to do testing are
also quite varied in scope. Some places (ci.centos) have freedom
to bring a full VMs customized as we desire, others (Travis, Gitlab)
are supporting docker containers with arbitrary Linux, others (brew,
koji) we just have to accept whatever environment we're executing
in.

In terms of developers, we can't rely on ability to run VMs, because
they may already be running inside a VM, and lacking nested-virt.

Essentially I think this means we need to make it practical to run
the functional tests as is, in whatever the current execution
environment is. If that works, then pretty much by implication,
it ought to be possible to optionally launch it inside a VM, or
inside a container to reproduce the precise environment of a
particular test system.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

--
libvir-list mailing list
libvir-list@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/libvir-list



[Index of Archives]     [Virt Tools]     [Libvirt Users]     [Lib OS Info]     [Fedora Users]     [Fedora Desktop]     [Fedora SELinux]     [Big List of Linux Books]     [Yosemite News]     [KDE Users]     [Fedora Tools]

  Powered by Linux