On Fri, Jun 04, 2021 at 09:52:52AM -0500, Praveen K Paladugu wrote: > Erik, > > Thanks for this detailed response. It answers many questions we had about CI > for libvirt. > > > On 6/2/2021 3:32 AM, Erik Skultety wrote: > > On Thu, May 27, 2021 at 11:17:04AM -0500, Praveen K Paladugu wrote: > > > Hi, > > > > > > While developing cloud-hypervisor driver for libvirt, we re-fitted > > > cloud-hypervisor project's CI to libvirt. This CI was built on Rust and > > > currently supports VM boot up tests. > > > > > > https://github.com/cloud-hypervisor/libvirt/tree/ch/ch_integration_tests > > > > Hi, > > so excited to hear about ^this effort - I haven't gone over the code base yet, > > but nevertheless having something set up already at your side is awesome. > > > > > > > > We are working on extending this CI to incorporate more functional tests: > > > Networking, Thread Pinning etc. We are curious to know if libvirt project > > > has any plans to setup a CI to run functional tests. > > > > > > I noticed https://gitlab.com/libvirt/libvirt-ci effort which focuses on > > > running builds against various platforms and formats. Could you please > > > clarify libvirt project's plan for setting up a CI to run functional tests. > > > > > > > Yes, we do have plans. As you've already correctly noted, currently we only > > have automated builds running in GitLab the configuration of which comes and is > > maintained by the libvirt-ci project. As for the functional tests though, we're > > currently fighting on 3 fronts: > > - infrastructure > > - automation machinery > > - testing framework > > > > Without going in too much detail we're working on improving libvirt-ci project > > in a way that would allow to make libvirt CI configurable for any interested > > party, IOW we'd set a standard on what the machines should look like and how > > libvirt expects to interact with them \wrt executing tests on them, but what > > tests you run on them is up to you as long as you report them back to the > > upstream libvirt pipeline (most likely gitlab). > > > > To go to a little more detail: > > > > Infrastructure > > -------------- > > - we're planning on running the workloads in nested KVM VMs, because it's much > > easier to set up in an automated fashion, is suitable to be run on local > > developer's machines and can be thrown away afterwards > > > > - machines will be connected to gitlab with the gitlab-runner agent > > (provided we don't decide to move to a different platform which is not likely > > even with the restriction on using public runners with the free tier) > > > > Automation > > ---------- > > - this will be handled by libvirt-ci (lcitool) which already has support for > > both container and VM workloads, we just have to shape it so that it can do > > what we want in terms of functional tests in VM workloads > > > > Test framework > > -------------- > > - there already are 2 frameworks: Avocado-VT and libvirt TCK none of which > > ultimately will be used, because they're not particulary libvirt developer > > friendly for various reasons (not the point of this email) > > > > - we're experimenting with using the plain Avocado framework integration just > > like QEMU has already done upstream (so we'd like to stay in aligned with QEMU) > > > > - the main arguments for selecting the Avocado framework are (there a few more): > > -> the native language of Avocado is Python which we already decided in > > upstream libvirt to be the dominant, modern and preferred scripting > > language of many users/contributors > > > > -> Avocado is truly language agnostic as far as tests go, so if you're > > bringing an existing test suite in a different language like e.g. Bash, > > that is fine Avocado can detect and execute external tests as well > > > > -> Avocado supports many of the new test result formats out there along > > with the simple and older TAP format, so it can satisfy various needs > > > > -> Avocado supports parallel test executions > > > > - as for what the initial deployment will utilize, it's impossible to think > > that we're going to port all of Avocado-VTs 17k test case (hell no!), so we'd > > start with the older test set coming from libvirt TCK which has proven to > > What do you mean by "older test set" here? Could you please point me to some > links/docs related to this test set? Hi Praveen, so, documentation in libvirt-derived projects is kinda "problematic" (read "basically non-existent") and so the best I can do is to share the link to the repo: https://gitlab.com/libvirt/libvirt-tck/ (this is the old perl-based framework, but FWIW it supports arbitrary languages) As for Avocado-VT: https://github.com/avocado-framework/avocado-vt (framework) https://github.com/autotest/tp-libvirt (17k test cases for avocado-vt) > > If this test set has sufficient coverage for our usecase, we will consider > pivoting to avocado suite for future test cases. > > > catch serious bugs, but new tests would be written solely in Python and > > very very slowly we'd migrate the TCK test cases from Perl to Python and host > > all of it as part of the main libvirt repo > > > > I hope that ^this gist would be enough of an answer to you :). Stay tuned for > > next plans on the CI front. > > > > Regards, > > Erik > > > > Lastly, where can we follow the adoption of Avocado Framework? Will this > start in libvirt-ci repository? This will be proposed as an RFC to libvir-list. We intend to make it part of the main libvirt repo - there are a couple of reasons to do it, but the main one is to remain aligned with QEMU with our efforts. Regards, Erik
