On Thu, Jul 01, 2021 at 07:04:32PM +0100, Daniel P. Berrangé wrote:
> On Wed, Jun 30, 2021 at 01:36:30PM -0300, Beraldo Leal wrote:
> > lavocado aims to be an alternative test framework for the libvirt
> > project using Python, python-libvirt and Avocado. This can be used to
> > write unit, functional and integration tests and it is inspired by the
> > libvirt-tck framework.
> >
> > This series introduces the basic framework along with some basic test
> > examples that I got from libvirt-tck. I would appreciate your comments
> > on this RFC, to see if this fits this project's needs. Also, names and
> > locations are just a proposal and can be changed.
>
...
> Libvirt has consumers writing applications in a variety of
> languages, and periodically reporting bugs. My general wish
> for a test harness would be for something that can invoke
> and consume arbitrary test cases. Essentially if someone
> can provide us a piece of code that demonstrates a problem
> in libvirt, I would like to be able to use that directly as
> a functional test, regardless of language it was written
> in.
Well, in theory yes - it is all nice to let anyone contribute tests in their
language of preference. Realistically, look at the nwfilter stuff in TCK, it's
a pile of unintelligible Bash which cannot be debugged. So, I understand why
you'd want to imagine or write tests in Go, but someone will have to maintain
them otherwise we'd end up with a 3rd framework nobody will want to use.
Luckily, Python as a language doesn't suffer from this and we've already made
that preference in the libvirt repo already. So, I'd suggest to use our
expertise in helping contributors who wish to contribute a test with porting
their test case to a Python one - yes, it does consumer human resources and
prolongs the time for a contribution to be accepted, but that pays off later
down the road when the test cases need to be adapted to new
conditions/environments.
>
> In theory the libvirt TCK allowed for that, and indeed the
> few tests written in shell essentially came to exist because
> someone at IBM had written some shell code to test firewalls
> and contributed that as a functional test. They just hacked
> their script to emit TAP data.
Others have already responded that Avocado can already do the same.
What I'd like to say, again in context of arbitrary languages used in tests,
the problem with the whole bash machinery IBM contributed is not only that it
cannot be debugged, but also that you cannot replay a single failed test, so if
an error happened in e.g. test 1148, you have literally no idea how to work
only with that test case. Knowing literally zero about Avocado's nrunner and
external (non-native) test suite support I cannot tell whether Avocado would
have done a better job than TCK (I doubt it), but what I know for sure is that
if you write native tests, you get much better support and control over the
test suite.
>
>
>
> > For now, this framework assumes that you are going to run the tests in a
> > fresh clean environment, i.e. a VM. If you decide to use your local
> > system, beware that execution of the tests may affect your system.
>
> It is always good to be wary of functional test frameworks trashing
> your system, but at the same time I think makes things more flexible
> if they are able to be reasonably safely run on /any/ host.
>
> For the TCK we tried to be quite conservative in this regard, because
> it was actually an explicit goal that you can run it on any Fedora
> host to validate it correctly functioning for KVM. To achieve that
> we tried to use some standard naming conventions for any resources
> that tests created, and if we saw pre-existing resources named
> differently we didn't touch them. ie all VMs were named 'tck-XXX',
> and were free to be deleted, but other named VMs were ignored.
>
> For anything special, such as testing PCI device assignment, the test
> configuration file had to explicitly contain details of host devices
> that were safe to use. This was also done for host block devices or
> NICs, etc. Thus a default invokation only ran a subset of tests
> which were safe. The more dangerous tests required you to modify the
> config file to grant access.
This is what I've always disliked about Avocado-VT - it never cared about your
environment and always thrashed your system. Therefore we're approaching this
very carefully and even try to think off how could libvirt-ci be integrated
into the core Avocado which would help in creating clean environments.
IIRC it was still necessary in TCK to run some of the network-related
tests with root permissions, but polkit may help (I haven't tried) - I sure
don't want to run the test suite with root permissions to overcome some of the
issues, because that doesn't simulate the real world usage. Therefore, I'd like
libvirt Avocado as a framework to be set up by default in a way where polkit
rules are in effect.
>
> I think it would be good if the Avocado supporting test APIs had a
> similar conceptual approach, especially wrt to a config file
> granting access to host resources such as NICs/Block devs/PCI devs,
> where you need prior explicit admin permission to avoid danger.
>
> > One of the future goals of this framework is to utilize nested
> > virtualization technologies and hence make sure an L1 guest is
> > provisioned automatically for the tests to be executed in this
> > environment and not tamper with your main system.
>
> I think the test framework should not concern itself with this
> kind of thing directly. We already have libvirt-ci, which has
> tools for provisioning VMs with pre-defined package sets. The
> test harness should just expect that this has been done, and
> that it is already running in such an environment if the user
> wanted it to be.
I suggested Beraldo to put this one in so as to make it very clear what our
intentions are wrt/ coming up with a new framework and our expectations of it
as an end result (like I said, we already have 2 frameworks, we don't want this
one to end up the same)
>
> In the TCK config file we provided setting for a URI to connect
> to other hosts, to enable multi-hosts tests like live migration
> to be done.
We would not exclude live migration from the test suite, it's simply a beast on
its own (mainly because of automation, more specifically preparing a know
destination to migrate to). But again, ^this is not something the framework
itself should be concerned about, just a background story :).
>
> > I'm adding more information with some details inside the README file.
>
> Overall, I'm more enthusiastic about writing tests in Python
> than Perl, for the long term, but would also potentially like
> to write tests in Go too.
>
> I'm wondering if we can't bridge the divide between what we
> have already in libvirt-tck, and what you're bringing to the
> table with avocado here. While we've not done much development
> with the TCK recently, there are some very valuable tests
> there, especially related to firewall support and I don't
> fancy rewriting them.
Nobody does fancy rewriting those, but there's no way going around that, we'll
have to do it at some point because like I said, they're very error prone
since they match against the verbatim output of firewall CLI tools which leads
to frequent breakages and you figuring out what went wrong is pure torment.
>
> Thus my suggestion is that we:
>
> - Put this avocado code into the libvirt-tck repository,
> with focus on the supporting infra for making it easy to
> write Python tests
Clearly I missed the section where you mentioned that we don't need to host
the framework in the main repo...Anyway, I disagree here, again, we have a
precedent of 2 failed frameworks where nobody really knows where to contribute
a test nor wants to deal with any additional repos. A few remarks
- it is much easier if the
test suite lives alongside our unit test suite which is an "environment" every
developer is familiar with.
- integration into gitlab CI would be much easier with a single repo than with
a multi-repo setup
-> yes, we already do that with binding builds, but I'm talking about
setups where we'd have to configure gitlab to clone 3 repos and build
them in chain, setup the framework config and execute the tests;
- keep in mind that we're trying to keep our efforts in sync with QEMU wrt/
(should apply vice-versa as well), so if avocado-qemu was accepted in QEMU
upstream as the current framework which also happens to be hosted under the
main repo, I don't see a problem why we could not do the same in libvirt
- the main selling point IIRC of having TCK hosted in a standalone repo was
that you could test ANY version of libvirt
-> we didn't have CI back then, but we do now...so if we start running the
functional tests early, every upstream release would technically be
tested, so there would not be a need to test older versions of libvirt
with the same test suite, because it would have already been done before
releasing that specific version at some point in time
>
> - Declare that all tests need a way to emit TAP format,
> no matter what language they're written in. This could
> either be the test directly emitting TAP, or it could
> be via use of a plugin. For example 'tappy' can make
> existing Python tests emit TAP, with no modifications
> to the tests themselves.
>
> https://tappy.readthedocs.io/en/latest/consumers.html
>
> IOW, you can still invoke the python tests using the
> standard Python test runner, and still invoke the perl
> tests using the stnadard Perl test runner if desired.
>
> - Switch the TCK configuration file to use JSON/YAML
> instead of its current custom format. This is to enable
> the Python tests to share a config file with the Perl
> and Shell tests. Thus they can all have the same way
> to figure out what block devs, NICs, PCI devs, etc
> they are permitted to use, and what dirs it is safe
> to create VM images in, etc.
I'm worried that any more time we spend on trying to re-shape TCK into
something modern is wasted compared to adopting Avocado. I've been fixing the
tests in TCK for a while just to make sure TCK doesn't fail, because the
original idea was for Avocado to utilize its nrunner to run TCK in its current
shape and start testing early just to give us time to transition from TCK to
Avocado by porting a good fistful of tests.
Erik
