On 2/6/12 7:27 AM, Tom Callaway wrote:
On 02/06/2012 04:59 AM, Julio Merino wrote:
Hello,
[...]
Hello Tom,
Thanks for the detailed reply. A longer reply comes below ;)
So, the first thought I had was "these binaries won't be in the $PATH".
You say above that "End users need to be encouraged to 'cd /usr/tests'
and to run stuff from there, just as they do with /usr/bin or
/usr/sbin." ... but end users do not cd into /usr/bin to run a binary
from there, they simply call the binary name and expect it to work.
I'm assuming that the fact that these tests are not in the $PATH is a
feature, not a bug. :)
Correct. They are not supposed to be in the PATH. (They could be
though, but their utility would be reduced and they'd just screw up
autocompletion. Also, this might not be totally doable given that
testsdir is a tree with multiple nested subdirectories; see below for a
detailed example showcasing corner cases.)
It'd be argued that the tools to run the tests (atf-run in this case)
should automatically look for stuff in /usr/tests unless told otherwise,
thus hiding the details of where the tests are stored. I'll grant you
that this is a nice idea to pursue, but that's not how things work at
the moment and I'm not sure about what the ramifications of such change
would be.
(The idea of having to cd into the tests directory before running the
tests is that you can have tests in multiple "roots". One root is the
testsdir, where the tests for all installed packages live. But other
valid roots are the top directory of your source trees, so that you can
execute the tests while doing development before doing an actual
installation!)
If this is the case, then it may be possible after all to use
/usr/libexec/tests for this. After all, /usr/libexec is a bit of a grey
area, the FHS doesn't mention it at anywhere (despite repeated pleading
to them to do so), it comes from the GNU Coding Standards and Fedora has
adopted it as an exception. The Fedora Packaging Guidelines say:
Libexecdir (aka, /usr/libexec on Fedora systems) should only be used
as the directory for executable programs that are designed primarily
to be run by other programs rather than by users.
That said, I could definitely see the argument being made that
/usr/libexec/tests is a perfectly acceptable use case for binaries that
are "designed primarily to be run by testing frameworks, but also
available outside of $PATH for users to manually execute".
I see. /usr/libexec/tests doesn't sound too bad considering this
description! However, the data files issue you raised could get in the
way. Some more details below.
However, whether it is /usr/tests or /usr/libexec/tests, there is going
to be a need to tightly constrain the acceptable files that could go
into /usr/tests and document exactly how they are allowed to go into it.
For example, my experience with test suites is almost certainly far less
than yours, but there seems to also be a fair amount of non-executable
test data content that comes along with the executable tests. E.g.
test-xpdf reads in 00-Broken.pdf with XFAIL as the result. Would
non-executable test data live in /usr/[libexec/]tests/$program/ ? Or
/usr/share/tests/$program/ ?
As it turns out, I forgot to mention this exact point (data files) in my
original email. The way things work at the moment within ATF is to have
data files and helper binaries in the same directory as the test program
that needs them. The test programs can then dynamically query what
their "source directory" is so that they can locate these files
regardless of where the test program lives.
The rationale for this was two-folded. First, it allows the test
programs to run both from the source directory and the installed tree
because they don't hardcode absolute paths to their helpers. And
second, the whole tests tree lives in a self-contained directory that
can either be pruned from the system (which is convenient due to the way
NetBSD works) or shared on the network. Looking at existing practice, a
test tree is composed mostly of binaries or scripts; the percentage of
data files is very low. (There was also an idea of being able to
provide such programs as "bundles" that could be copied to slave
machines for remote testing, but has never been implemented yet.)
In other words, the layout for your example would be:
testsdir/
testsdir/xpdf/
testsdir/xpdf/parser_test - Binary; atf-run executes it from within
a temporary directory to "isolate" it from the read-only testsdir
and other parts of the system. parser_test can query its "srcdir",
which in this case is "testsdir/xpdf/", and thus locate where the
00-broken.pdf file is.
testsdir/xpdf/00-broken.pdf - The auxiliary data file for the xpdf
tests.
This may rule out the suitability of /usr/libexec/tests as the location
for these files, given that your description seems to imply that libexec
is for binaries only. But if it isn't, I'd be OK with using this directory.
It is also possible that introducing a new home for binaries would cause
multi-lib heartburn in RPM, but I suspect that is a simple enough
problem to fix, since we haven't really hit it with our use of
/usr/libexec to date.
What do you mean by "multi-lib heartburn in RPM"? (Sorry, not familiar
from the term and a couple of Google searches did't provide much
insight.) FWIW, I have a preliminary atf.spec file here that provides
subpackages for libraries, tests and binaries and things seem to be fine
to my untrained eye. I can share this file if it helps at all.
You probably should open a ticket with the FHS working group:
https://bugs.linuxfoundation.org/ (product: FHS) and explain all of this
to them so that it has a chance of being officially added in the next
major FHS revision, whenever that may be.
OK, I can do that. Wouldn't it be easier to illustrate the point /
convince them if there was existing practice somewhere? (Not sure if
BSD flavors count in this case!)
Above and beyond your proposal, the only other way I can think of
handling this would be an elaborate namespacing of the test binaries in
/usr/bin, e.g. /usr/bin/test-$pkg-$testname , and while that would be a
solution that would permit a strict compliance with the FHS as it exists
today, I'm not very excited about it and am not seriously recommending
it as an alternative.
I don't think this would be an appropriate layout, and it doesn't play
well with the structure of atf (let aside the clutter that would appear
in /usr/bin). Let me provide some additional details of how the tests
tree is structured.
Each directory in testsdir has an Atffile that indicates which test
programs exist in that directory, which subdirectories to recurse into,
and additional properties that indicate meta-data for the test programs.
So, for example, the testsdir would look like this:
testsdir/ - The top-level directory. This directory would be owned
by a filesystem package and be a dependency of anything that places
files in testsdir/.
testsdir/Atffile - The main control file. This has the knowledge to
recurse into the direct subdirectories of testsdir/ looking for
other Atffiles to execute. E.g. this would find testsdir/a/Atffile
and testsdir/b/Atffile to source tests, but it would not find
testsdir/c/Notatf nor testsdir/d/e/Atffile. This file would be
installed by the "atf" tools package, as it does not make any
sense to have it without the corresponding tools.
testsdir/lutok/ - Owned by a lutok-tests package, including all the
subfiles and subdirectories.
testsdir/lutok/Atffile - The control file for the tests provided by
lutok. Contains the list of test programs to run and any optional
meta-data for them. It also includes other subdirectories to
"recurse" into.
testsdir/lutok/operations_test - A test program for the "operations"
module of the source tree. (The name of the binary is arbitrary.)
testsdir/lutok/state_test - Another test program.
testsdir/lutok/detail/ - A subdirectory of lutok's tests containing
tests for internal modules only. (This would mimic the structure
of the source package.)
testsdir/lutok/detail/Atffile - Same as above, but for the tests in
this subdirectory.
testsdir/lutok/detail/internal_test - Yet another test program.
testsdir/lutok/detail/state_test - Another test program. Note that
this can have the same name as the program in the parent
directory! There is nothing to prevent this, and thus flattening
the test programs into /usr/bin as you propose would be very
complicated.
testsdir/lutok/foo/ - Another subdirectory for additional tests, etc.
testsdir/foobar/ - Another package, with the same structure as above.
testsdir/baz/ - Another package, but this time without any Atffile in
it. This would not be executed by atf-run because the top-level
Atffile would discard this subdirectory. (Kyua, the replacement
of atf, could run these tests though provided that a Kyuafile was
created within the subdirectory. This is out of scope now though.)
A little detail to note above is that in a project with a good amount of
test coverage, there will potentially be a test program per each source
file. This means that the testsdir contains lots of binaries, and you
don't want that many, potentially with conflicting names, anywhere in
your path! To illustrate the point, check out this output page which
contains the results of the execution of the NetBSD test suite:
http://www.whooppee.com/~paul/amd64-results/last_atf.html
Anyway. This layout may remind you a lot of make(1) and a traditional
source tree populated with Makefiles. And actually, this is the main
idea of how the tests tree is organized and is an artifact of ATF's
origins (NetBSD's source tree). However, having played with other large
source code bases, I can say that this organization is not unique to NetBSD.
We'd discuss extensively whether this layout is suboptimal for Fedora's
guidelines or not... But please note that all I'm trying to achieve
here is to figure out how to package an existing piece of software, not
how to re-architect it!
If you made it this far again, thank you.
--
Julio Merino
--
packaging mailing list
packaging@xxxxxxxxxxxxxxxxxxxxxxx
https://admin.fedoraproject.org/mailman/listinfo/packaging
