On Sun, 23 Jul 2006, Rolf Schumacher wrote:
The larger problem: As an assessor for safety critical software of embedded systems I'm regularily faced with statements like: "we only have made a little change to the source code we do not want to retest all modules" The problem is, even if a change is made to one module only, how can I demonstrate that all other modules are unchanged?
The short answer is "You can't. Not even in Principle." The longer answer is (greetings from Werner by the way!)... * A common form of bug in C/C++ is an uninitialized variable. 9 times out of ten, if the value "zero" or whatever the previous value was is acceptable, the bug escapes the notice of the testers, no matter how rigorous. However, change the memory layout, change the activation pathway of the code, and the uninitialized memory can hold something nasty. * Bugs infest the boundaries of state space. In a smallish system you can easily have 2000 variables ie. more states that there are atoms in the universe! ie. Your previous testing, no matter how rigorous, did not explored the full state space. In fact explored only tiny fraction. So a small change entirely external to your "unchanged" module can cause your "unchanged" module to move through a different region of state space. An untested region. So far more useful than checksumming I would recommend.... * Compile -W -Wall -Werror and perhaps use www.splint.org as well. * Design your system in strict layers. * Decrease the size of the state subspace in each layer. * Decrease the number of "user configurable items". * Decrease the number of product variants. * Place a good suite of automated Unit tests at each layer. Test Driven Development is Very Good for this. * Brace the interior of the code well with asserts. This is like testing from inside the code, both during bench test and production test. * Think your assert handling policy through so you run exactly the same code under test as in production. * Have an automated suite of functional tests around your product as a whole. * Run all tests on all builds. * Preferable run all the non-timing critical test under something like valgrind as well. http://www.valgrind.org/ All this will do _way_ more to increase safety than any checksumming scheme.
Unchanged source code is easy to demonstrate. But this is not the whole story.
I was recently called in to track the difference between too apparently identical bits of source code resulting in different executables & behaviour... Answer? The __FILE__ macro (commonly used in asserts and the like) includes the full path, one guy had built his code in /home/joe and the other in /home/fred resulting in different string lengths! (And hence different offsets everywhere!)
If I change the source of one module and recompile it....
All too true. Perhaps if they were separate processes / executables you in the nice state of being able to byte for byte verify that they are same. The Unixy notion of separate processes separate virtual addresses spaces is not just a pretty face. It gives a lot of really good, really hard safety guarantees!
The only way to find these (unlikely) errors nowadays is to retest all modules. Or to ignore this problem with some unknown risk. But: retests are too expensive and risk negligence has its probability.
If I were to push (hard) on some aspect of the problem I would push (hard) on the cost of retest.
The idea is to have a checksum for each module that has passed all tests successfully and this checksum is unchanged even if any other module becomes changed.
Checksum the individual .o object files. Best balance for difficulty of doing vs safety. Not in anyway a guarantee, a test is still better, this would be cheaper and better than nothing. Notes about checksumming the .o files.... They include debug data which has absolute file paths. They include time stamps. Probably need to use the objdump (or maybe objcopy/ readelf) utilities to read out the sections you care about and checksum those. eg. Beware of $Version Control$ tags ending up in strings. They also caused spurious diffs. John Carter Phone : (64)(3) 358 6639 Tait Electronics Fax : (64)(3) 359 4632 PO Box 1645 Christchurch Email : john.carter@xxxxxxxxxx New Zealand Carter's Clarification of Murphy's Law. "Things only ever go right so that they may go more spectacularly wrong later."
From this principle, all of life and physics may be deduced.
