On 12/13/10 2:59 PM, "Joshua Brindle" <method@xxxxxxxxxxxxxxx> wrote: > So, we have a few "unit" tests for parts of userspace. I use quotes because > many > of them aren't unit tests at all but tests wrapping around a giant black box > masquerader as a unit test (e.g., all the linker/expander tests). > > Some problems with the current tests are 1) they aren't run by default, 2) > they > use a unit test framework now in yum, 3) they have interdependencies within > the > tree and 4) a lot of the things in the repo are really hard to unit test. > > We'd like to spend some cycles here doing unit tests but we need to address > the > above problems first. > > The requirements I'd throw out are: > > 1) Each component (libsepol, libselinux, etc) must be testable outside of the > main repo (so that tests can be run from rpm, portage, etc) > 2) Must not require anything outside of yum > 3) Each test should test a single unit (see > <http://www.javaranch.com/unit-testing.jsp> for an explanation) > 4) Must be run in the default build target > 5) Be useful tests > > For unit tests to be useful they must get run, unfortunately Fedora doesn't > seem > to like to package unit test frameworks so we'd need to put one in the tree > and > use it directly. > > We could: > > - write our own and put it in the repo > - use a simple single .c file solution like CuTest > - try to get one packaged up in Fedora > - use something more powerful and add it to our repo > > All varying levels of bad. > > Further, to meet requirement #1 the test framework would have to be duplicated > once per framework. > > Now, to throw more wrenches in the machinery, a huge amount of the libraries > (particularly libselinux) aren't directly unit testable. They'd need to use a > framework that can mock functions (eg., to test is_selinux_enabled() you'd > need > to mock the fopen() and read() functions). This makes the more simple > solutions > not as useful. > > I've recently worked with a framework called test-dept > <http://code.google.com/p/test-dept/> that can do mocking and assertion > testing. > It is a pain to use and requires a bunch of headers, Makefiles and a program > to > manipulate the object files to run. > > After thinking about this quite a bit I don't know if we can have reasonable > test coverage without using something capable of mocking. I've also looked at > other frameworks that provide mocking and they aren't suitable for a variety > of > reasons. > > So... this is starting to sound like rambling but the only thing reasonable I > can come up with (and reasonable is a stretch) is to remove all the CUnit > tests, > start doing unit tests with test-dept, checked in to the test directory of > each > component. We could start with simple tests and eventually graduate to mock > tests. The reason I'd rather do this than start with something like CuTest and > move to test-dept when we need its capabilities is because I did that very > thing > on a recent project and the two systems are very incompatible and we now have > 2 > completely separate sets of tests and difficulty finding out what is tested > and > where and by what. > > > Opinions? Compromise? Should we just give up on unit testing altogether? My vote is to go the CuTest route. My primary reason is that writing CuTest tests is easy. We've been really bad about creating unit tests so far, and I doubt that will change. Something as painful to use as test-dept will just encourage developers to not write unit tests. I realize that this will mean we will never get near 100% test coverage for things like libselinux due to the lack of mocking support, but I think it's the lesser of two evils. Given our lack of diligence in this area historically, I'd rather make things easy to do than shoot for full coverage. Note that we've been using CuTest in CIL, and it is very simple and easy to use. There's almost no spin up to using it. Chad -- This message was distributed to subscribers of the selinux mailing list. If you no longer wish to subscribe, send mail to majordomo@xxxxxxxxxxxxx with the words "unsubscribe selinux" without quotes as the message.
