Thursday, November 25, 2010

When to do written test cases?

Recently I've had several talks with various testers, whom I tutor, about use of written test cases. Since the subject has gained remarkable feedback lately so it looked well-cooked enough to publish.

Doing manual testing with written test cases tends to be one of the most expensive form of testing. Why so? Some of the reasons are

If doing the test and/or repeating it isn't either expensive nor extremly time-consuming then usually writing down in details what to test takes a lot longer than actually doing setup for test and running it. So from testers perspective it's not very wise choice of cost vs value. My personal worst (I'm not proud of it, but my that-time manager required it) was around 5 hours to write down the test case into really fine details while it took 5 minutes to run it. What makes it "my personal worst" - the test case was written for very testable functionality taking only 1 minute to setup and easy/cheap to run it again as many times as required. So in general I spend over five hours for a less than 5 minute easily repeatable test.

Someone with higher level of qualification should usually write the test case and review it several times since mistakes in test cases can cause various problems from false-negative bugs (bugs that aren't real and appear only because of the mistake(s) in test case) to false-positive bugs (bugs that are considered correct behavior because misinterpretation of test case instructions). That means that higher level specialist's time, what is usually more expensive, is used for this task. In addition it takes lot of time to think up and write down every possible combination how to test something since there are almost unlimited amount of possible test cases. And worst part is that usually there are too many unknowns about the product to reduce number of possible combinations to reasonable level.

If the Test Case is prewritten after some other document (Use case, requirement document), then often some areas are left uncovered in test case either because all possible combinations can't be thought at that time, some things are still uncertain at the time of writing the test case or some requirements have changed between writing the document and actually testing done. My experience I usually find majority of bugs from things not written down on test cases or in any other documentation.

If detailed test cases are written afterwards for someone else then it's usually good idea to find out for what purpose and what is done with them afterwards. If they are done for someone to review what's been done, then there are usually much cheaper ways to do it. For example to use screenrecording tool to record done tests. It's even to organize them like documents with versions etc. 1 GB of data storage in Hard Drives cost less than 0,1 USD. Or to write just test report describing test coverage at the end of each time-cycle or mission.

But then when to do written test cases?
Written test cases are the most formal ways to test and should be used when the highest possible level formality is required. The reason can be business decision or obligation from contract, but in any case the additional time required must be considered.

Also written test cases are useful if you can run tests only limited number of times and must make sure to cover every possible and required scenario with really limited reruns of tests. And the need for extra resources must be considered again.

Another reason for written test cases might be needed if something additional is done with the test cases. For example if certain test are needed to be automated for any number of reason. Then the test cases serve as use case documents for automated tests. In which case they need to be maintained and updated as any other product documentation along with automated tests in order to use them again in future properly.

My personal experience is that most software development projects doesn't have neither the need nor budget for fully formal testing to do all testing in written test case form.

No comments:

Post a Comment