On Thu, Jul 22, 2021 at 09:44:28AM +0100, Daniel P. Berrangé wrote: > On Wed, Jul 21, 2021 at 04:22:19PM -0300, Beraldo Leal wrote: > > On Wed, Jul 21, 2021 at 06:50:03PM +0100, Daniel P. Berrangé wrote: > > > On Thu, Jul 01, 2021 at 06:09:47PM -0300, Beraldo Leal wrote: > > > > On Thu, Jul 01, 2021 at 07:04:32PM +0100, Daniel P. Berrangé wrote: > > > > > On Wed, Jun 30, 2021 at 01:36:30PM -0300, Beraldo Leal wrote: > > > > > > I'm adding more information with some details inside the README file. > > > > > > > > > > Overall, I'm more enthusiastic about writing tests in Python > > > > > than Perl, for the long term, but would also potentially like > > > > > to write tests in Go too. > > > > > > > > > > I'm wondering if we can't bridge the divide between what we > > > > > have already in libvirt-tck, and what you're bringing to the > > > > > table with avocado here. While we've not done much development > > > > > with the TCK recently, there are some very valuable tests > > > > > there, especially related to firewall support and I don't > > > > > fancy rewriting them. > > > > > > > > > > Thus my suggestion is that we: > > > > > > > > > > - Put this avocado code into the libvirt-tck repository, > > > > > with focus on the supporting infra for making it easy to > > > > > write Python tests > > > > > > > > > > - Declare that all tests need a way to emit TAP format, > > > > > no matter what language they're written in. This could > > > > > either be the test directly emitting TAP, or it could > > > > > be via use of a plugin. For example 'tappy' can make > > > > > existing Python tests emit TAP, with no modifications > > > > > to the tests themselves. > > > > > > > > > > https://tappy.readthedocs.io/en/latest/consumers.html > > > > > > > > > > IOW, you can still invoke the python tests using the > > > > > standard Python test runner, and still invoke the perl > > > > > tests using the stnadard Perl test runner if desired. > > > > > > > > This is supported already: > > > > > > > > $ avocado run --tap - --test-runner='nrunner' tests/domain/transient.py > > > > 1..3 > > > > ok 1 tests/domain/transient.py:TransientDomain.test_autostart > > > > ok 2 tests/domain/transient.py:TransientDomain.test_lifecycle > > > > ok 3 tests/domain/transient.py:TransientDomain.test_convert_transient_to_persistent > > > > > > This is nice, showing fine grained TAP output lines for each > > > individual test within the test program > > > > > > > > > I tried using the hints file that Cleber pointed to make avocado > > > *consume* TAP format for the Perl/Shell scripts: > > > > > > $ cd libvirt-tck > > > $ cat .avocado.hint > > > [kinds] > > > tap = scripts/*/*.t > > > > > > [tap] > > > uri = $testpath > > > > > > > > > And I can indeed invoke the scripts: > > > > > > $ avocado run ./scripts/domain/05*.t > > > JOB ID : b5d596d909dc8024d986957c909fc8fb6b31e2dd > > > JOB LOG : /home/berrange/avocado/job-results/job-2021-07-21T18.45-b5d596d/job.log > > > (1/2) ./scripts/domain/050-transient-lifecycle.t: PASS (0.70 s) > > > (2/2) ./scripts/domain/051-transient-autostart.t: PASS (0.76 s) > > > RESULTS : PASS 2 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 > > > JOB HTML : /home/berrange/avocado/job-results/job-2021-07-21T18.45-b5d596d/results.html > > > JOB TIME : 1.90 s > > > > > > which is good. > > > > > > And also I can ask it to produce tap output too: > > > > > > $ avocado run --tap - ./scripts/domain/05*.t > > > 1..2 > > > ok 1 ./scripts/domain/050-transient-lifecycle.t > > > ok 2 ./scripts/domain/051-transient-autostart.t > > > > > > > > > But this output isn't entirely what I was after. This is just summarizing > > > the results of each test program. > > > > > > I can't find a way to make it show the fine grained tap output for the > > > individual tests, like it does for the python program > > > > Actually, the first Python TAP output example is showing a coarse TAP > > result. I have combined three transient tests into one single file but > > splitting them into three different methods, so we could execute each one > > individually. i.e: tests/domain/transient.py:TransientDomain.test_autostart. > > > > So, I did some reorg when migrating to Python test. > > > > In order to archive the same with Perl, we could do the same there, > > because the way individual tests are written there, doesn't allow for > > individual execution. > > > > Yes, we could do some tricks, to parse and combine outputs and list as > > it was a more fine graned, but afaict, we could not individually execute > > those. This is part of Avocado test definition where in order to be > > called a test, we need to be able to execute those individually as well. > > Ok, I'm not so fussed about whether avocado can ultimately preserve the > fine grained TAP output. Mostly I'm looking to understand how you should > debug failures when they go wrong, becuase the default output I see from > avocado is very terse giving no indiciation of the the failure - merely > that there has been a failure. > > In an interactive environment you can just re-rnu the individual failed > test directly. In an automated CI environment you need the test harness > to display enough info to debug the failure directly. Ideally it should > dump the full stdout+stderr either to the console or a log file that > can be publish as an artifact from CI. Meson tends todo the latter > creating a log file will full stdout/err from all tests. Sure, I completely understand your concerns. In this case, the output here is "very terse" because you are running the "--tap" version. When using the "default/human" output, Avocado outputs some pointers to where you could get more details. We understand that debugging is a critical part of the process, so Avocado stores the job information inside "~/avocado/job-results/[job-id|latest]/" folder by default (regardless of the output format used). Inside this folder, you can find a "debug.log" file and a "test-results" sub-folder (with per-test stdout/stderr files). The entire folder could be uploaded as an artifact for any CI. It is also generated by default a results.xml (xUnit format) for better debugging on CI that supports this format (i.e: Gitlab). If you would like to see an example, QEMU project's test list[1] on Gitlab is based on this XML file. I understand that there is some room for improvements here and we are glad to address them. In meanwhile I was working on a proposal for the libvirt-tck based on the progress of this thread. Hopefully, this will help with the Avocado bootstrap phase. [1] - https://gitlab.com/qemu-project/qemu/-/pipelines/340613922/test_report [2] - https://gitlab.com/beraldoleal/libvirt-tck/-/tree/avocado-libvirt/ Regards, -- Beraldo
