On Thu, Sep 05, 2019 at 06:15:04PM +0100, Daniel P. Berrangé wrote: > On Thu, Sep 05, 2019 at 12:30:27PM -0400, Laine Stump wrote: > > (BTW, what does the removal of perl from libvirt say about continued > > use of perl for libvirt-tck? There are a lot of useful tests in there > > that find real bugs, but they tend to languish (both in use and in > > enhancements) because nobody wants to do anything with perl (or with > > the big shell scripts that do the nwfilter testing). It would be nice > > if the tests were all in a language that was more accessible, but > > they're kind of married to the perl TAP module (and besides, who > > wants to spend time rewriting a bunch of test scripts when they > > already work?). This is mostly a moot point, because I think hardly > > anyone runs the libvirt-tck tests anymore, which is too bad because > > it has historically caught some regressions that no other testing > > framework did.) > > I believe they are run by the Red Hat virt QE team, as I ave > got bug reports against the Perl libvirt bindings, where the > reproducer is a TCK script. > > The TCK really covered three distinct things > > - framework for controlling test execution > - library modules/helper APIs to facilitate test creation > - the large set of indivdual tests > > There is no code binding between the tests and the framework. They > communicate exclusively through the TAP protocol. In that way we can > throw out the current framework and drop in a different one quite > easily - the new framework merely needs to be able to consume the > TAP data format. For any framework which does not natively support > the TAP format, it is quite easy to write an adapter / shim that > would convert to the desired data format. You might loose some > finese / fine grained details in the logs/progress reporting but > functionally the tests should work fine. > > As a point of reference the GLib testing harness has recently > switched from using their own custom output format to using the > TAP output format. So if we adopt GLib in the libvirt.git and > use it in our unit tests those would end up using TAP. GLib > provides a shim for automake that allows it to consume the > results from GLib based units tests. > > QEMU's makefiles recently switched to using TAP for its unit > and (some of) its functional tests too. > > IOW, I'm perfectly happy to throw away the existing TCK framework > impl if someone proposes a better alternative. I came across this the other day, which looks like it could be a viable replacement for the TCK control framework: https://github.com/python-tap/tappy https://tappy.readthedocs.io/en/latest/ > > > As for the individual TCK tests themselves, there's two distinct > questions > > - Should they be re-written in a different language > - Should they continue to use TAP output format > > If we are not going to rewrite the tests, then I think they should > continue to use TAP output format. It is the easiest thing to use > when writing tests in Perl. > > If we are going to rewrite the tests, then the tests should ideally > use the output format that best suits the framework invoking them. > IOW, if the framework still uses TAP, we should continue to use > TAP. If not, then don't. There is an adapter for python that can > convert its native unittest output format into TAP format. > > > I'm pretty ambivalent on rewriting the existing tests. The language > of existing tests only matters if you expect people to be modifying > them. Assuming a decent support library it should be no harder to > define a new test, than to extend an existing test. > > IOW, if we want to write TCK tests in Python, we don't really need > to rewrite existing ones. We instead need to write a Python variant > of the helper APIs, so that you can trivially write new Python tests > which operate in the same manner as the Perl tests. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| -- libvir-list mailing list libvir-list@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/libvir-list
