Automated Testing: Myth vs. Reality

By Jeff Gainer

(Author's note: This essay originally appeared in the 20 June 2001 issue of the Cutter IT Journal E-Mail Advisor.)

Automated software test tools have a high disappointment rate. After an initial flush of excitement of recording test scripts, the tool often becomes "shelfware," and testing returns to the old manual methods. The keys to successfully implementing an automated test tool are preparation, training, and realistic expectations. Here are answers for some of the common pitfalls and misconceptions.

Testing is Not a Silver Bullet
The benefits of automated testing are legion: repeatability, maintainability, automatic reporting, and the ability to run unattended regression tests. But automated software testing is not a cure-all, and it is certainly not a solution for having too little time. Expecting a test automation tool to act as a "silver bullet" to circumvent staffing, schedule, or budget issues virtually guarantees failure.

Test Early and Often
Software testing activities should be conducted during the entire lifecycle of a development project. Correcting a defect late in the development cycle is considerably more expensive than correcting one caught early in the lifecycle. The most effective and economical method is to test early and often.

Record and Playback Is Just a Tool
The myth that automated test tools simply record keystrokes and play back the tests unattended has probably caused more disappointment that any other feature of these tools. First, recording a test case presumes the application is working as designed. Second, recording test cases encourages building a library of undocumented, unmaintainable code. Recording a test case is a good place to start developing a test, but it's just an aid to developing the code for a more powerful test suite or utility.

Don't Use Overly Complex Test Cases
On the opposite end of the automated testing spectrum, sometimes testers can become so enamored of the tool's capabilities and robustness of its language that they spend excessive time creating complex test cases when simpler, more straightforward cases would do. Resist the urge to write a program to test a program.

Invest in Training
Software testing and automated test tools require training. The tutorial in an automated testware package is rarely enough to expose the true power of the tool. Often, users without sufficient training use the tool for a brief time to record and play back test scenarios, then discontinue using the tool without ever taking advantage of its more powerful capabilities. Yes, training costs money and time. But the payoff is real: I've had students in my classes who have never written a line of code in their lives, yet after four days they were writing reusable functions, data-driven test cases, and creating new classes and methods.

Don't Automate Everything
Some tests will always be manual. Although you can create an automated test to verify if a report was printed, you can't replace the manual task of looking at the report and verifying that the information is correct and properly formatted. Also, you can't automate a test if the results are not predictable. There are some tests that may be executed infrequently or would be too cumbersome to automate; these kinds of tests are also best left to manual methods.

Don't Use Part-Time or Inexperienced Testers
Some wrongly view software testing as a training ground for novice developers or, worse, a dumping ground for poor developers. Software testing is a legitimate subdiscipline of software engineering, as are programming and project management.

You should also avoid accepting part-time testers. Interested stakeholders may offer to "loan" staff to test software a few hours a week. Later, you may find that the part-time testers' other duties interfere or their interest wanes - suddenly, you have no staff. Also, it is possible that the testers may be "on loan" simply because they don't supply much value in their current positions. Do you want someone else's castoffs on your team?

Set Realistic Expectations
Successfully deploying a test automation project requires planning and time, as well as political skills of the part of the project manager to get (and keep) upper management support by setting realistic expectations. The benefits of automated testing accumulate over time - there is never an immediate payoff, but the long-term benefits are very real.

--Jeff Gainer


(c)2001 Cutter Information Corp. All rights reserved. This article has been reprinted with the permission of the publisher, Cutter Information Corp., a provider of information resources for IT professionals worldwide.

This article originally appeared in the Cutter IT E-Mail Advisor, a supplement to Cutter IT Journal.

Return To