Hello, The purpose of this long email is to justify the need for a /usr/tests directory and to ask for approval for its creation, with the final goal of packaging ATF, its tests and the tests provided by arbitrary applications (e.g. the just-approved lutok package[1]). This email is long because I feel the need to provide tons of background to explain why this directory is required. I hope the text is clear and that it makes enough sense to convince you of its approval. Here we go! I am the author of the Automated Testing Framework[2] (ATF for short). ATF is the testing framework currently used in the NetBSD operating system, although the ATF project itself is OS-agnostic. In other words, ATF builds and runs fine on multiple Linux distributions, BSD systems and OS X. (ATF is being replaced by a new project called Kyua[3]; this will come into play later in the text.) I am now interested in creating an RPM package of ATF (and Kyua) for Fedora and there is a "little detail" in ATF that will surely be controversial. This is why I am reaching the FPC before sending any spec file for review, as this particular point will need discussion and approval. ATF is made up of two parts: 1) a collection of libraries (C, C++ and shell) that enable developers to write test programs for their applications and libraries; and 2) a collection of tools (atf-run, atf-report) that allow the developers and *end users* to execute these tests. To re-emphasize the point above, note that I said "end users". One of the major features that differentiate ATF from other testing frameworks is that *the tests written using ATF are supposed to be installed in the system* as part of the regular application or library. ATF then provides standalone tools to execute these tests and to generate reports of their results, all without the need of the original program sources nor any development tools (read: no gcc, and specially no make). These tests can be run by any user as they don't require write access to the directory in which the tests live. To put this in the context of a Linux distribution, let's consider a fictitious example: imagine that glib and gtk came with ATF-based tests. These tests would get installed into testsdir/glib/... and testsdir/gtk/... The user would then do "cd testsdir/gtk/ && atf-run" to execute the tests on his machine, without the need for any -devel packages. The key point of this idea is that this allows end users to gain confidence in their systems, because they can validate that a particular piece of software runs fine on their OS / hardware combination. What is more, suppose that glib got upgraded to a new version but not gtk. If there were any incompatibilities between the two, the installed gtk tests would highlight it because they'd start failing. The packager may never have noticed that fact himself if he always kept his build environment up-to-date (as is commonly the case). (Also, consider creating a cron job to run all these tests nightly!) Well, imagine no more. We already do something like this in NetBSD. NetBSD defines the testsdir above as "/usr/tests". This tree[4] contains a subdirectory for each package that provides ATF-based tests, and there is a top-level Atffile file that allows atf-run to recurse into all the package subdirectories. (I.e. executing atf-run within /usr/tests runs *all* the tests available in the system for all packages.) You can see in [5] a more detailed description as to how this comes into practice. There are two important things to consider about /usr/tests: 1) The first is that there is a single directory acting as the "root" for the tests of arbitrary packages because this is what allows all these tests to be run at once. In other words, instead of having /usr/pkg1/tests /usr/pkg2/tests, we have /usr/tests/pkg1 and /usr/tests/pkg2. 2) The second is that ATF is not encoded anywhere in the path. I.e. the directories are /usr/tests/pkg*, not /usr/atf/pkg* nor /usr/tests/atf/pkg*. The reason for this is that the package-specific tests needn't be implemented using the ATF libraries. In fact, Kyua, the replacement for ATF, is able to execute these same unmodified tests and can also execute tests written other frameworks. In a future world, packages would install arbitrary tests into /usr/tests written using their preferred libraries (e.g. pyunit, autotest, etc.) and a single tool (Kyua) would run them all. Encoding the tool name in the directory introduces a dependency on the tool that does not necessarily exist. So, to sum it up: the whole point of this email is to discuss what the "testsdir" above should be in Fedora. I think it makes a lot of sense to keep the default of /usr/tests. However, of course, this does not complain with any the LSB policies. The problem is that no other already-existing directory seems to suit the goals of /usr/tests. For example: /usr/share/tests is unsuitable because the tests are obviously executables and thus not shareable. /usr/libexec/tests also appears unsuitable because these programs are not helper binaries internally used by other programs: they are supposed to be exposed to the end user, who may choose to run them by hand. /usr/lib/tests is even worse because these are not libraries and also has the same problem of libexec in the sense that the files are hidden from end users. End users need to be encouraged to "cd /usr/tests" and to run stuff from there, just as they do with /usr/bin or /usr/sbon. Therefore, I believe that introducing a new directory is sensible enough, particularly because it is potentially optional (e.g. could be made part of a -filesystem package). Actually, the way I envision the packaging to work is the following. Let's consider lutok again, as its distribution file contains ATF-based tests. lutok would come in these packages: lutok, lutok-devel, lutok-tests. The latter package would place files in /usr/tests/lutok only and it would depend on libatf-c++ (because lutok's tests are implemented in C++) and maybe some "tests-filesystem" package. lutok-tests needn't depend on the atf tools. Then, the user could just install the 'atf' package providing atf-run and atf-report to execute these tests from within /usr/tests or /usr/tests/lutok. Similarly, other tools could place their installable tests in /usr/tests/<pkg>. So, what packages would make use of /usr/tests if it were introduced now? As far as I can tell, these would be atf, lutok and kyua (including all of its subpackages for the libraries, etc.). Bind 9 recently adopted ATF too, although I'm not sure if they install the tests. This should be all for now. Is it OK to introduce this optional /usr/tests directory? Please let me know if you have any questions or concerns. Thank you, Julio Merino 1: https://bugzilla.redhat.com/show_bug.cgi?id=785619 2: http://www.NetBSD.org/~jmmv/atf/ 3: http://code.google.com/p/kyua/ 4: http://cvsweb.netbsd.org/bsdweb.cgi/src/tests/?only_with_tag=MAIN 5: http://blog.netbsd.org/tnf/entry/testing_netbsd_easy_does_it -- packaging mailing list packaging@xxxxxxxxxxxxxxxxxxxxxxx https://admin.fedoraproject.org/mailman/listinfo/packaging
