Course synopsis

This 2-day course gives a complete overview of relevant testing techniques. It is suitable for both inexperienced and experienced programmers as well as programming management and forms a sound, rigorous basis for software testing particularly for embedded control systems. It covers inspections, test planning, test management, test coverage, test case design, estimating testing quality and system diagnosis.

The course is copiously illustrated with data from real systems containing many surprises and important clues on how to improve software testing. Numerous workshops and self-assessments are included along with links to many other sources of relevant material.

Day 1 (Testing overview)

• The current state of the art in testing

Under the following headings, we gather together and discuss real data from all over the world to guide us in designing the best test strategies.

• Preparing the ground

  • The importance of good testing

  • The goals of testing

• Categorising defects - know your enemy

• Where and how do defects occur historically ?

• A salutary workshop in test case design

Many software engineers and managers believe that testing is fundamentally tedious but simple. In this series of workshops, we dispel this belief. The workshops illustrate a number of important principles including the effects of test-case design on architecture.

• Test types

It is important to understand test types in order to piece together an appropriate testing approach. Here we discuss the various types and their limitations in sufficient detail that they can form part of the tester's armoury.

• Static testing

  • What can and what can't be found

  • Inspections, manual and automatic

  • Estimating untestable components without testing them first

    • Relevant software measures

    • Supporting evidence

• Dynamic testing

  • Test case design

    • Black box v. White box

    • Equivalence classes

    • Boundary values

    • Other techniques

  • Unit, integration and system testing

  • Regression testing

• Estimating test quality

Estimating test quality is important not only from the viewpoint of producing a reliable product but also from the viewpoint of assessing the risk of system failure. Since one of the primary goals of testing is to find defects, at the same time we study the crucial issue of constructing a defect database.

• Test coverage

• Software fault injection

• Test effectiveness

• Defect databases -what to record and how to analyse it.

  • Pitfalls in measurement

  • Defect analysis and feedback

• Test planning

All this activity has to be managed somehow. Here we discuss how.

• The management of testing

• Concurrent evolution of test plans

• The relationship with design

Day 2 (Advanced topics)

• Portable test harnesses using scripting languages

A key step in improving test quality is some form of automation. Test automation can itself absorb a significant amount of resources. Here we discuss state of the art techniques to allow portable test harnesses to be developed. This was not easily possible even as recently as 2 years ago.

• Introduction to scripting languages

• Using Tcl/Tk and Perl to write portable test harnesses

• Inspections revisited in depth

There is overwhelming data to suggest that inspections are the most effective way of removing defect from software systems that has ever been found. Here, using a series of workshops, we concentrate on understanding and building these essential skills.

• Data on inspections

• The psychology of inspections

• Building inspection skills

  • Reading tests

  • Inspection tests

• Organising inspections

• Measuring and improving inspection quality

• Testing and diagnosis

In testing we find defects but what if there is insufficient information to correct the defect ? Then, the defect continues to fail, and we have that most undesirable of all failure modes - the repetitive failure mode. Repetitive failure modes are essentially avoidable and their presence leaves engineering shops wide-open to legal scrutiny with searching questions such as "Why did you continue to let that happen ?". Here we define the requirements for useful feedback from testing in terms of diagnostic quality and show how testing can be instrumental in improving this critical area.

• Diagnostic distance and diagnostic quality

• Improvement of diagnostic quality by test feedback

• Challenges in consumer system testing - the techniques in practice

The essence of testing is to apply knowledge to real systems. Some of the most challenging systems to test are those found in consumer electronic systems. Here we discuss techniques in practice in these challenging areas.

• Embedded systems

• Safety-critical systems

• Coping with the unavoidable

Nobody knows how to produce a defect-free system, so everything we release is overwhelmingly likely to contain defects. We must prepare the user for this by analysing the risk of such a defect being present and its possible effects on the end-user of the system.

• Risk assessment

• Risk management

• Course summary - putting it all together

Back to top