On 2/7/12 6:26 AM, Michael Schwendt wrote:
On Sun, 05 Feb 2012 22:59:10 -0500, JM (Julio) wrote:
The purpose of this long email is to justify the need for a /usr/tests
directory
To put this in the context of a Linux distribution, let's consider a
fictitious example: imagine that glib and gtk came with ATF-based tests.
These tests would get installed into testsdir/glib/... and
testsdir/gtk/... The user would then do "cd testsdir/gtk/&& atf-run"
to execute the tests on his machine,
2) The second is that ATF is not encoded anywhere in the path. I.e. the
directories are /usr/tests/pkg*, not /usr/atf/pkg* nor
/usr/tests/atf/pkg*. The reason for this is that the package-specific
tests needn't be implemented using the ATF libraries.
Still, the tests must be compatible with the testing framework's set of
tools. That's enough of a reason to add some sort of namespace to the
root directory.
If you created "unknown" subdirectories under /usr/tests/, they would be
skipped by the tools. (Actually, the goal would be for the tools to
eventually run arbitrary test programs regardless of whichever libraries
they use.)
I also find it contradictory how you refer to "atf-run", "atf-report"
and "a collection of tools", which have "atf-" in their name.
Contradictory with what? You certainly don't want a tool named "run" in
your path. There, of course, would be an "atf" dispatcher that accepted
subcommands and executed atf-run and atf-report on your back (much like
git does). But there isn't.
Surely the testing framework would not be renamed regularly. And
software authors, who would use this framework for writing own tests,
would need to be able to rely on an interface for retrieving filesystem
path details anyway. For example, a pkg-config file that can be queried.
Or a similar atf-config script that returns various paths.
Such a thing (atf-config) already exists, although it cannot tell what
the testsdir is. It actually should not, because the location of the
testsdir is a packaging issue and not something the tools care about at
all. atf-run cares about what's in the current directory and won't try
to load stuff from /usr/tests by itself, for example.
In fact, Kyua,
the replacement for ATF, is able to execute these same unmodified tests
and can also execute tests written other frameworks. In a future world,
packages would install arbitrary tests into /usr/tests written using
their preferred libraries (e.g. pyunit, autotest, etc.) and a single
tool (Kyua) would run them all. Encoding the tool name in the directory
introduces a dependency on the tool that does not necessarily exist.
Implementation details here would need to be discussed. The names of the
tools are irrelevant so far.
Exactly; the names are irrelevant.
Let's say you start by putting all the tests into
/usr/tests/atf/<package>, which would be the case if the name mattered.
This directory contains the top-level Atffile, and each subdirectory
contains an Atffile of its own. atf-run can only be executed from
within /usr/tests/atf.
Now, a new tool comes along (let's call it "magic") that is able to
interpret atf-based test cases but can also run pyunit tests. This tool
needs its own Magicfiles. How do you make the previous /usr/tests/atf/
directory tree work? Surely putting Magicfiles within /usr/tests/atf/
would not be appropriate, because this is in the atf namespace. But you
actually need to have those files in there to let the magic tool know
what to do.
Before you say "well, the tools should auto-discover the tests instead
of reading Magicfiles et. al.": no, they should not. These files
include much more than just a list of test programs. They provide
metadata, such as what "framework" the test programs are written with,
or arbitrary configuration properties for the programs.
The various testing frameworks would need to
be compatible somehow and meet the requirements of the controlling tools.
One couldn't dump arbitrary tests into a shared directory and e.g. mix
automated, semi-automated and non-automated tests.
Why not? As I said above, what the "controlling tool" runs is defined
by a file in the tests directory (Atffile). Any of these incompatible
files would not have a matching Atffile, and thus atf-run would not see
them.
Leaving FHS issues aside, in general it is not nice to let any package
"occupy" directories, which have very generic names, without very good
reason. That's not limited to /usr, and it's not a "first come, first served"
thing either. Imagine, a second group of testing framework authors would
also want to use a global /usr/tests in order to store incompatible tests
in there.
Such other systems do not exist, and I seriously doubt any will. If
they ever do, they'd have to play well with existing practice, just as
you ask any package to respect the subpackage-directories layout in
/usr/share. And asking such imaginary systems to respect the proposed
layout would be trivial for them, so it doesn't sound too unreasonable.
Lastly: please keep in mind that all I want to do is package an existing
piece of software, not redesign it. Comments of how things should be or
should not be done within the tool are not relevant in this discussion.
(But I'll gladly take suggestions if they are raised where they
belong: in the upstream mailing lists.)
--
packaging mailing list
packaging@xxxxxxxxxxxxxxxxxxxxxxx
https://admin.fedoraproject.org/mailman/listinfo/packaging
