So, we have a few "unit" tests for parts of userspace. I use quotes because many
of them aren't unit tests at all but tests wrapping around a giant black box
masquerader as a unit test (e.g., all the linker/expander tests).
Some problems with the current tests are 1) they aren't run by default, 2) they
use a unit test framework now in yum, 3) they have interdependencies within the
tree and 4) a lot of the things in the repo are really hard to unit test.
We'd like to spend some cycles here doing unit tests but we need to address the
above problems first.
The requirements I'd throw out are:
1) Each component (libsepol, libselinux, etc) must be testable outside of the
main repo (so that tests can be run from rpm, portage, etc)
2) Must not require anything outside of yum
3) Each test should test a single unit (see
<http://www.javaranch.com/unit-testing.jsp> for an explanation)
4) Must be run in the default build target
5) Be useful tests
For unit tests to be useful they must get run, unfortunately Fedora doesn't seem
to like to package unit test frameworks so we'd need to put one in the tree and
use it directly.
We could:
- write our own and put it in the repo
- use a simple single .c file solution like CuTest
- try to get one packaged up in Fedora
- use something more powerful and add it to our repo
All varying levels of bad.
Further, to meet requirement #1 the test framework would have to be duplicated
once per framework.
Now, to throw more wrenches in the machinery, a huge amount of the libraries
(particularly libselinux) aren't directly unit testable. They'd need to use a
framework that can mock functions (eg., to test is_selinux_enabled() you'd need
to mock the fopen() and read() functions). This makes the more simple solutions
not as useful.
I've recently worked with a framework called test-dept
<http://code.google.com/p/test-dept/> that can do mocking and assertion testing.
It is a pain to use and requires a bunch of headers, Makefiles and a program to
manipulate the object files to run.
After thinking about this quite a bit I don't know if we can have reasonable
test coverage without using something capable of mocking. I've also looked at
other frameworks that provide mocking and they aren't suitable for a variety of
reasons.
So... this is starting to sound like rambling but the only thing reasonable I
can come up with (and reasonable is a stretch) is to remove all the CUnit tests,
start doing unit tests with test-dept, checked in to the test directory of each
component. We could start with simple tests and eventually graduate to mock
tests. The reason I'd rather do this than start with something like CuTest and
move to test-dept when we need its capabilities is because I did that very thing
on a recent project and the two systems are very incompatible and we now have 2
completely separate sets of tests and difficulty finding out what is tested and
where and by what.
Opinions? Compromise? Should we just give up on unit testing altogether?
--
This message was distributed to subscribers of the selinux mailing list.
If you no longer wish to subscribe, send mail to majordomo@xxxxxxxxxxxxx with
the words "unsubscribe selinux" without quotes as the message.
