Automated Testing in an Agile Environment

Lever Switch

Let’s have one final test. Throw the switches

Recently, we needed to undertake unit testing for agile development, and continually needed to test separate modules. As we integrated the separate modules into our central repository we of course find the need to check that new additions to the repository have not altered the work already completed. Because we were verifying an embedded C system we chose to use Unity (not the game engine) for the embedded project.

More about Unity can be found at www.throwtheswitch.org and https://github.com.

To make this as efficient as possible, in the Unity test harness we created tests that consistently rerun. We do this by following design patterns, and from these produce tests scripts that are derived from an examination of the codes logic. The key techniques that we use to design test cases are:

  • Equivalence Partitioning;
  • Boundary Value Analysis (Supplementary to Equivalence Partitioning);
  • Independent Path Coverage; and
  • Decision Coverage.

The Unity test harness is a lightweight C test harness for embedded systems, and provides the tester the ability to express how the Function Under Test is required to behave during the white box testing.

Unity Test Harness

The Unity test harness provides a number of really useful features, these include:

  • The ability to express test cases and their expected results in a common language;
  • Configuration Management of the test cases and the results;
  • The flexibility to run test cases needed to provide evidence of PASS/FAIL depending on the module being considered; and
  • It provides detailed reports of the test run.

Unity provides the ability to produce assertion statements that we expect to be true about our Functions Under Test, and this supports the use of Test Driven Development.

Regression Testing and Continuous Integration

We currently conduct regression testing, this is the process of testing software to ensure that it still performs with the same expected results after further work has been carried out. With the test scripts we created it is a natural progression to use the development practice of Continuous Integration. This is an automated process which requires the recurrent integration of tested code to the stable branch of our project.

The code base contains shared resources such as mocks that provide simulate behaviour for complex system elements, definitions of global constraints, and Unity framework to support our regression testing. We found that this continuous integration approach builds confidence in the robustness of the modules under test, and where further development is essential, our team gets a really good understanding of the projects overall progression, as well as the next obstacle. It also provides confidence that the changes introduced into source control have not disturbed the function under test or the test scripts.

Lessons Learnt

This approach has the ability to reduce the cost of regression testing, and provides confidence, through continuous integration, that the final product is compliant. It goes without saying that automated testing is quicker than manual testing, allows for verification of different test permutations within a short span but requires constant tending to ensure the effectiveness of the agile verification process. This method does also require an initial investment cost, however this is soon regained.

In the context of testing using Unity, I have found that writing the tests scripts in the same language as the code under test is written in removes the need to ‘context switch’ to another tool/language and way of thinking in order to test.

I believe that this is an effective environment to continuously test any system with a clear understanding of the health of a software project in real time. We will be using this proven approach for other projects.