Test details
We are often asked
how we test our own products. Following a rigorous code inspection of
all changes, (its around 120 KLOC at the moment), the Safer C toolset
is then subjected to the following dynamic tests which together contain
just over 279,000 test cases:-
|
Test
|
Requirement
|
FIPS160 regression
test
|
The same suite
as NIST use for validating C compilers. It contains well over
a 1000 files and several hundred thousand lines of code. Our parsing
engine must correctly parse the entire suite for syntax and constraints.
Apart from two issues where we disagree on the interpretation
of the standard, it does.
|
Internal
regression test
|
A series of
nearly 300 files containing an eclectic mixture of strange C fault
modes from around the world we have collected over the years.
Correct parsing and warning for each fault mode required.
|
Oakwood Computing
MISRA-C test suite regression test
|
As high a
detection rate as possible for the required and advisory rules.
Currently running around 95% for the required rules on our internal
suite although the new official conformance suite will affect
this.
|
Numerical
C regression test
|
A regression
suite of several hundred files based on the well-known Numerical
Recipes C suite by Press et. al. Correct parsing and warning for
each fault mode required.
|
Stress testing
|
Amongst other
things, no core dumps when given crazy inputs, (for example an
object file). We also give it to Dr. Tim ("Wrecker")
Hopkins of the University of Kent Computing Laboratory, the best
stress tester we have ever found. This includes the customary
'sitting-on-the-keyboard' test and many other novel and highly
sophisticated test techniques.
|
GUI testing
|
Identical
scripts for each supported platform are exercised and each option
checked. This is manual only (and takes ages). :-(
|
The source code
is internally annotated against the requirements and the parser must
also pass itself with no statically detectable code defects.
Finally, when all
else fails, we also use several beta testers who can generally break
the best products within a very short time, (its a calling) and install
the product several times on as many platforms as we can get our hands
on. The depressing thing is that its a complex product and after all
this, there will still be bugs in it, but hopefully not very many.
Current post-release
defect density:-
Parsing engine:-
0.26 per KLOC (1000 lines of source code)
Interface:-0.93
per KLOC
Back
to top
