Erik,
Thanks for this detailed response. It answers many questions we had
about CI for libvirt.
On 6/2/2021 3:32 AM, Erik Skultety wrote:
On Thu, May 27, 2021 at 11:17:04AM -0500, Praveen K Paladugu wrote:
Hi,
While developing cloud-hypervisor driver for libvirt, we re-fitted
cloud-hypervisor project's CI to libvirt. This CI was built on Rust and
currently supports VM boot up tests.
https://github.com/cloud-hypervisor/libvirt/tree/ch/ch_integration_tests
Hi,
so excited to hear about ^this effort - I haven't gone over the code base yet,
but nevertheless having something set up already at your side is awesome.
We are working on extending this CI to incorporate more functional tests:
Networking, Thread Pinning etc. We are curious to know if libvirt project
has any plans to setup a CI to run functional tests.
I noticed https://gitlab.com/libvirt/libvirt-ci effort which focuses on
running builds against various platforms and formats. Could you please
clarify libvirt project's plan for setting up a CI to run functional tests.
Yes, we do have plans. As you've already correctly noted, currently we only
have automated builds running in GitLab the configuration of which comes and is
maintained by the libvirt-ci project. As for the functional tests though, we're
currently fighting on 3 fronts:
- infrastructure
- automation machinery
- testing framework
Without going in too much detail we're working on improving libvirt-ci project
in a way that would allow to make libvirt CI configurable for any interested
party, IOW we'd set a standard on what the machines should look like and how
libvirt expects to interact with them \wrt executing tests on them, but what
tests you run on them is up to you as long as you report them back to the
upstream libvirt pipeline (most likely gitlab).
To go to a little more detail:
Infrastructure
--------------
- we're planning on running the workloads in nested KVM VMs, because it's much
easier to set up in an automated fashion, is suitable to be run on local
developer's machines and can be thrown away afterwards
- machines will be connected to gitlab with the gitlab-runner agent
(provided we don't decide to move to a different platform which is not likely
even with the restriction on using public runners with the free tier)
Automation
----------
- this will be handled by libvirt-ci (lcitool) which already has support for
both container and VM workloads, we just have to shape it so that it can do
what we want in terms of functional tests in VM workloads
Test framework
--------------
- there already are 2 frameworks: Avocado-VT and libvirt TCK none of which
ultimately will be used, because they're not particulary libvirt developer
friendly for various reasons (not the point of this email)
- we're experimenting with using the plain Avocado framework integration just
like QEMU has already done upstream (so we'd like to stay in aligned with QEMU)
- the main arguments for selecting the Avocado framework are (there a few more):
-> the native language of Avocado is Python which we already decided in
upstream libvirt to be the dominant, modern and preferred scripting
language of many users/contributors
-> Avocado is truly language agnostic as far as tests go, so if you're
bringing an existing test suite in a different language like e.g. Bash,
that is fine Avocado can detect and execute external tests as well
-> Avocado supports many of the new test result formats out there along
with the simple and older TAP format, so it can satisfy various needs
-> Avocado supports parallel test executions
- as for what the initial deployment will utilize, it's impossible to think
that we're going to port all of Avocado-VTs 17k test case (hell no!), so we'd
start with the older test set coming from libvirt TCK which has proven to
What do you mean by "older test set" here? Could you please point me to
some links/docs related to this test set?
If this test set has sufficient coverage for our usecase, we will
consider pivoting to avocado suite for future test cases.
catch serious bugs, but new tests would be written solely in Python and
very very slowly we'd migrate the TCK test cases from Perl to Python and host
all of it as part of the main libvirt repo
I hope that ^this gist would be enough of an answer to you :). Stay tuned for
next plans on the CI front.
Regards,
Erik
Lastly, where can we follow the adoption of Avocado Framework? Will this
start in libvirt-ci repository?
--
Regards,
Praveen K Paladugu
