On Fri, Jul 2, 2021 at 7:55 AM Daniel P. Berrangé <berrange@xxxxxxxxxx> wrote: > > On Fri, Jul 02, 2021 at 07:23:57AM -0400, Cleber Rosa wrote: > > Hi Daniel, > > > > On Thu, Jul 1, 2021 at 2:05 PM Daniel P. Berrangé <berrange@xxxxxxxxxx> wrote: > > > > > > On Wed, Jun 30, 2021 at 01:36:30PM -0300, Beraldo Leal wrote: > > > > lavocado aims to be an alternative test framework for the libvirt > > > > project using Python, python-libvirt and Avocado. This can be used to > > > > write unit, functional and integration tests and it is inspired by the > > > > libvirt-tck framework. > > > > > > > > This series introduces the basic framework along with some basic test > > > > examples that I got from libvirt-tck. I would appreciate your comments > > > > on this RFC, to see if this fits this project's needs. Also, names and > > > > locations are just a proposal and can be changed. > > > > > > Some high level thoughts > > > > > > - More extensive functional integration testing coverage is good > > > > > > - We need to actually run the functional tests regularly reporting > > > via GitLab pipelines in some way > > > > > > - Using Python is way more contributor friendly than Perl > > > > > > - This does not need to live in libvirt.git as we don't follow > > > a monolithic repo approach in libvirt, and it already depends > > > on functionality provided by other repos. > > > > > > > > > When it comes to testing, I feel like there are several distinct > > > pieces to think about > > > > > > - The execution & reporting harness > > > - Supporting infrastructure to aid writing tests > > > - The set of tests themselves > > > > > > If I look at the TCK > > > > > > - The harness is essentially the standard Perl harness > > > with a thin CLI wrapper, thus essentially works with > > > any test emitting TAP format > > > - The support infra is all custom APIs using libvirt-perl > > > - The tests are mostly written in Perl, but some are > > > written in shell (yuk). They all output TAP format. > > > > > > One key thing here is that the test harness is fairly loosely > > > coupled to the support infra & tests. > > > > > > The TAP data format bridged the two, letting us write tests > > > in essentially any language. Of course writing tests in > > > non-Perl was/is tedious today, since there's no support > > > infra for that which exists today. > > > > > > The TAP data format bridge also means we can easily throw > > > away the current TCK harness and replace it with anything > > > else that can consume tests emitting TAP data. > > > > > > > > > If I look at Avocado, I think (correct me if i'm wrong) > > > > > > 1. The harness is essentially the standard Python harness > > > with a thin CLI wrapper. Thus needs all tests to implement > > > the Python test APIs > > > > Not really. Even though Avocado is mostly written in Python, there > > have been provisions for accommodating foreign types of tests (in > > different forms) since its inception. The most basic way, is, of > > course, simply treating a test as an executable. But this is far from > > the only way. For instance, these are other possibilities: > > > > a) if an executable generates TAP, it can consume the test's TAP > > output and determine the test results > > Can you show how to actually make this work, since from the manpage > I can only see how to make it emit TAP, not consume it. I see there > is a 'tap' plugin but that doesn't seem to provide a runner. > There are a couple of ways. The simplest is hinting to Avocado that a file is of kind "tap" with a hintfile. Suppose you have a "test-suite" directory, and in it, you have "test.sh": #!/bin/sh -e echo "1..2" echo "ok 2 /bin/true" echo "ok 3 /bin/uname" And ".avocado.hint": [kinds] tap = *.sh [tap] uri = $testpath If you "cd test-suite", and do "avocado list --resolver *sh" you get: Type Test Tag(s) tap test.sh Resolver Reference Info TEST TYPES SUMMARY ================== tap: 1 And then you can run it with: $ avocado run --test-runner=nrunner ./test.sh JOB ID : 2166ad93ffc25da4d5fc8e7db073f4555d55e81a JOB LOG : /home/cleber/avocado/job-results/job-2021-07-02T08.39-2166ad9/job.log (1/1) ./test.sh: STARTED (1/1) ./test.sh: PASS (0.01 s) RESULTS : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 JOB HTML : /home/cleber/avocado/job-results/job-2021-07-02T08.39-2166ad9/results.html JOB TIME : 1.31 s This is a relevant documentation pointer: https://avocado-framework.readthedocs.io/en/latest/guides/user/chapters/introduction.html#the-hint-files And I'll make sure the man page is updated, thanks for noticing it. Thanks, - Cleber. > Regards, > Daniel > -- > |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| > |: https://libvirt.org -o- https://fstop138.berrange.com :| > |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| >
