Autoconf developers,
My organization develops scientific software. Currently for
regression testing we have a set of custom scripts that build the
source code nightly, run a series of simulations, compare the
resulting data with accepted values, and email the status of these
tests to the development team. We recently decided to refactor this
setup to improve ease of use and functionality. It was decided to
see if Autotest could be used for this refactoring.
I chose Autotest for this project for several reasons. We already
use Autoconf, Automake, etc. extensively in the company, so Autotest
would be using our existing expertise. Also, using an existing,
actively developed OSS project seemed better than continuing to
maintain and update our from-scratch solution. However, as we've
worked through the project, we are reconsidering the choice due to
feature requirements.
Specifically, the test suite needs to do the following:
1) Run nightly as a cron job to test the entire package
2) Allow developers to easily create and and new tests
3) Serve as an example of running the software for end-users
Unfortunately, some of the requested features have proven difficult
to implement using Autotest, particularly that of being an example to
end-users. The issues I've encountered include:
1) Naming of test subdirectories. Currently, Autotest runs each test
in a separate directory; these are labeled sequentially based on the
number of the test, i.e. 1/ 2/ 3/ and so on. The dev team would much
rather see them named after the name of the test, i.e. foo/ bar/ test1/
2) Hierarchy of tests. Currently, tests are grouped into
subdirectories by function. We would like them to be executed in
directories named in the same fashion, i.e. testsuite.dir/foo/
fooTest1/, testsuite.dir/foo/fooTest2, testsuite.dir/bar/barTest1, as
opposed to testsuite.dir/1/, testsuite.dir/2, etc.
3) Serving as example to users. In addition to having cron test the
software nightly, we want the tests to show novice users how to use
the software. Due to the complicated nature of the code, this
involves several steps, including creating an input file, running the
software itself (using some scripts to deal with serial vs. parallel
machines, etc.), and processing the results in some fashion (usually
by calling external applications). Currently, I do these steps in a
series of AT_CHECK macros, but these are difficult to read for the
average user or developer. Additionally, users need a script that
they can run completely independently of the regression tests, unlike
the 'run' script created by Autotest that is tied into the test
suite. In other words, if a user is working on a problem of type
foo, we'd like to point them to the regression test foo, which would
show them how they should go about setting up and running their problem.
4) Developing in testsuite.dir. Developers have requested the
ability to debug and develop new tests in testsuite.dir itself. That
is, editing an input file, and then re-running that single test
again. Currently, the solution is to have an AT_CHECK command that
creates a symlink to the input file so it can be edited. If the test
fails, the symlink allows a developer to edit the input, and they can
then re-run the test with the 'run' script created by Autotest.
However, this removes and re-creates the directory, meaning they have
to do a 'cd .' to refresh the filesystem handle. My dev team hated
this.
With the exception of the directory/hierarchy naming, I think all of
these problems can be worked around, mostly by creating shell
scripts. These would be user-readable and called by Autotest.
However, doing that I lose benefits of using Autotest in the first
place, that of testing the return code of each step, being able to
run extra commands on success or failure, and auto-generated,
detailed log files.
The main issues seem to be centered around the tests serving as
examples for the user. As a tester/maintainer, I find that Autotest
works fine for my needs. However, it doesn't work the way the dev
team would like it to in terms of developing new tests or debugging
test failures due to the issues above. So, I suppose my question
really boils down to, "Is Autotest the right solution?"
Thanks for any and all feedback, comments, and suggestions.
--
Nathaniel D. Sizemore, Technical Staff
Tech-X Corporation | 5621 Arapahoe Ave Suite A | Boulder, CO 80303
E: sizemore@xxxxxxxxxx P: (720) 974-1846 W: http://www.txcorp.com
_______________________________________________
Autoconf mailing list
Autoconf@xxxxxxx
http://lists.gnu.org/mailman/listinfo/autoconf
