Thanks, John.
I see you're experienced what you're talking about.
And I like to thank you for the examples.
Splint looks good. Valgind too, but the majority
of software I'm dealing with is non-Linux, RT-based.
I'll give it a second look.
You're absolutely right with your examples.
But think twice: if the only alternative is:
1. take the code with a few immediate ad hoc tests or
2. supply some rules that reduces the probability of
failures, do not change the amount of effort needed.
I'd like to supply the rules to end up with more safety,
that is: less probability for unacceptable failures.
John, we're living in a real world. I can tell:
you can't say: retest 100000 lines of code
upon a small change. Believe me.
We have to come up with some better idea.
For now I'd like to focus solely on dynamic errors.
Errors to happen while compiling, linking, loading and running.
For now, forget about dangerous errors in one module
that were covered so far by the unchanged other one.
For now, forget about programming errors
(even if this is the most likely source).
An error I'd like to uncover is: I'm linking on a PC/XP
and somehow a bit changes just before the linker
packs the object to be written to the disk. Checksum is ok,
the object is bad.
If I had a checksum from last linking and made no change
I could point to the failure immediately, e.g. at load time.
In the first place I'm not thinking about a checksum
to be generated while the object is on disk (this would be nice, though).
I could have a MD5 algorithm to run at initialization time.
I just have to look at the code as an (or some) array(s) of bytes,
calculate the sum and do not start the software
if it is not as expected. I'll do that on three computers
at the same time with some tricky hardware voting.
I have to do this anyway on a regular base
in order to check whether my software has changed
where it shouldn't. This is no cost.
I guess you're right in thinking of each module
as a separate task that takes messages to perform
functions and supplies messages as results.
That points to the right direction.
Than the question is, what's the simplest messaging system,
that leaves the receiver code invariant
upon changes in the sender and vice versa?
Should look familiar to programmers that think in
function calls.
The point that I'm asking is,
John Carter wrote:
On Sun, 23 Jul 2006, Rolf Schumacher wrote:
The larger problem:
As an assessor for safety critical software of embedded systems
I'm regularily faced with statements like:
"we only have made a little change to the source code
we do not want to retest all modules"
The problem is, even if a change is made to one module only,
how can I demonstrate that all other modules are unchanged?
The short answer is "You can't. Not even in Principle."
The longer answer is (greetings from Werner by the way!)...
* A common form of bug in C/C++ is an uninitialized variable. 9 times
out of ten, if the value "zero" or whatever the previous value was is
acceptable, the bug escapes the notice of the testers, no matter how
rigorous.
However, change the memory layout, change the activation pathway of
the code,
and the uninitialized memory can hold something nasty.
* Bugs infest the boundaries of state space. In a smallish system you
can easily have 2000 variables ie. more states that there
are atoms in the universe!
ie. Your previous testing, no matter how rigorous, did not explored the
full state space. In fact explored only tiny fraction.
So a small change entirely external to your "unchanged" module can
cause your
"unchanged" module to move through a different region of state space.
An untested region.
So far more useful than checksumming I would recommend....
* Compile -W -Wall -Werror and perhaps use www.splint.org as well.
* Design your system in strict layers.
* Decrease the size of the state subspace in each layer.
* Decrease the number of "user configurable items".
* Decrease the number of product variants.
* Place a good suite of automated Unit tests at each layer. Test
Driven Development
is Very Good for this.
* Brace the interior of the code well with asserts. This is like testing
from inside the code, both during bench test and production test.
* Think your assert handling policy through so you run exactly the
same code
under test as in production.
* Have an automated suite of functional tests around your product as a
whole.
* Run all tests on all builds.
* Preferable run all the non-timing critical test
under something like valgrind as well. http://www.valgrind.org/
All this will do _way_ more to increase safety than any checksumming
scheme.
Unchanged source code is easy to demonstrate.
But this is not the whole story.
I was recently called in to track the difference between too apparently
identical bits of source code resulting in different executables &
behaviour...
Answer? The __FILE__ macro (commonly used in asserts and the like)
includes the full path, one guy had built his code in /home/joe and the
other in /home/fred resulting in different string lengths! (And hence
different offsets everywhere!)
If I change the source of one module and recompile it....
All too true. Perhaps if they were separate processes / executables you
in the nice state of being able to byte for byte verify that they are
same.
The Unixy notion of separate processes separate virtual addresses spaces
is not just a pretty face. It gives a lot of really good, really hard
safety guarantees!
The only way to find these (unlikely) errors nowadays
is to retest all modules.
Or to ignore this problem with some unknown risk.
But: retests are too expensive and risk negligence
has its probability.
If I were to push (hard) on some aspect of the problem I would push
(hard) on the cost of retest.
The idea is to have a checksum for each module that has passed all
tests successfully and this checksum is unchanged even if any other
module becomes changed.
Checksum the individual .o object files. Best balance for difficulty of
doing vs safety.
Not in anyway a guarantee, a test is still better, this would be cheaper
and better than nothing.
Notes about checksumming the .o files....
They include debug data which has absolute file paths.
They include time stamps.
Probably need to use the objdump (or maybe objcopy/ readelf) utilities
to read out the sections you care about and checksum those. eg.
Beware of $Version Control$ tags ending up in strings. They also caused
spurious diffs.
John Carter Phone : (64)(3) 358 6639
Tait Electronics Fax : (64)(3) 359 4632
PO Box 1645 Christchurch Email : john.carter@xxxxxxxxxx
New Zealand
Carter's Clarification of Murphy's Law.
"Things only ever go right so that they may go more spectacularly
wrong later."
From this principle, all of life and physics may be deduced.
