• Poor test design – the root of all evil!

Poor test design – the root of all evil!

titles Mind the Gap (Large)

During my many years in testing one aspect has stood out amongst all others as costing significant time, money and quality; poor test design. Not only does test design take a great deal of time the end result of it is often poor in terms of duplicate testing, gaps in coverage and inaccurate and poor test procedures. If we assume that we now have a set of inefficient, incomplete and incorrect test cases then obviously we are going to throw good money after bad when it comes to test execution. Take it one step further and we are going to invest in activities such as automation that will simply execute our poor tests faster i.e. more poor investments in testing.

Finally I found a product, AgileDesigner, which automatically converts structured requirements into detailed, accurate and maintainable test cases via a range of coverage techniques. The end result being less time to create fewer test cases covering more requirements and the ability to keep them up to date and relevant.

Let’s take a look at the traditional situation. A change is ordered and the Requirements Analysts create a text based requirements document. The requirements are reviewed by test but due to other priorities and the volume of text the review process was not so effective in finding issues and clearing up all misinterpretations. Test now have the task of designing an optimal set of accurate test cases that fully cover the requirements. Test then ask the Requirement Analysts to review their test cases but due to the volume of text and repetition in them only skim read and then approve.

So what just happened? Well we took a long time to create two sets of independent documentation, requirements and tests, that no one had the energy or time to review and in all likelihood contained a great deal of errors that will affect us later on. In addition, based on recent research at some major institutions, the test cases probably only covered around 25% of the requirements and had duplicate testing up to 30 times.

Summary is that we have taken too much time creating too many test cases that cover too little, have many inaccuracies and will take a lot of effort and time to execute.

The problem does not, unfortunately, end there. Every time there is a change we receive updated sets of requirements, which need to be tested. We also need to assess how that change has affected our existing test cases. Which test cases need to be modified and how? Which test cases are no longer relevant and shall be removed? Which test cases contain duplicate testing of the new test cases we just created? Updating test cases can be extremely time consuming as one change can affect the steps of many test cases and if not updated the detailed steps are no longer accurate.

Summary is that we again have spent too much time creating even more test cases, have not aligned our existing test cases with the change and have introduced increasing levels of inaccurate and duplicate testing. In the end we have out-dated test cases, which we continue to execute and have lost control of what they represent in terms of coverage.

Agile Designer allows teams e.g. Business and Requirement Analysts, Testers, Developers etc. to collaboratively draw graphicaland common requirement models. Graphical models are so much more powerful than text in explaining what is actually being requested i.e. more effective static testing and hence find defects earlier in the development process.

Of course there are many products that allow you to build graphical models but building the model in Agile Designer is just the beginning. In Agile Designer these models can then be automatically converted into a complete set of test cases, with detailed test cases steps and expected results, through a range of different coverage techniques. You can choose to test all nodes, all edges, all pairs, full coverage etc. and even have different coverage settings for different start and end points of the process.

In addition Agile Designer can also be used to automatically find and allocate relevant test data so that testing can commence directly and will not fail downstream due to test data issues. Finally when the requirements change the model is simply adjusted and then used to automatically take care of new test cases, updates and removals.

Summary is that we now spend less time creating a common graphical requirements model, rather than independent text rich documentation, that can be used to automatically generate fewer test cases of higher quality that cover more requirements and ensure that our test cases and steps are always aligned with change.

For me this product addresses all three goals of time, cost and quality ensure that more accurate development and testing are performed in significantly less time and effort. Only when we have a true understanding of what we are testing and are perfectly aligned with that should we start to implement other improvements such as simulations and automation.

Go to http://www.agile-designer.com/ to read more and start a free trial.