Wikipedia.org explains software testing this way:

“Software testing provides an objective, independent view of software to allow the business to appreciate and understand the risks of software implementation.”

Software testing is an often-neglected, or at least underachieving, part of the software development lifecycle or change management process.

There is a serious need to improve testing. Consider this. After a study commissioned by the Department of Commerce’s National Institute of Standards and Technology (NIST) was completed, NIST released a report that suggested that software bugs were costing the U.S. economy $59.5 billion annually.¹ According to their study, a third of this cost or more could be avoided if better software testing was performed.

Testing can be very important. Most people understand this. However, the fact remains that most organizations struggle in this area. Why is that?

We believe one contributing factor is a lack of supporting tools.

Consider this common approach to performing testing. A BA, Architect, Stakeholder or Project Manager produces a Test Plan as a Word Document. When it comes time to perform testing, the test team gets a copy of the Test Plan Document and decides how they want to split up the testing duties. Each person begins to test in their area, checking back in from time-to-time with someone accumulating test results, or possibly with each individual posting bugs or issues into a bug tracking system.

So, what’s wrong with this? Several things.

First, it’s very loosely controlled. Documentation on test coverage and results is only as good as the individuals involved.

Next, there’s no visibility into test progress as the test is being performed. One can’t assess progress as it happens, and there’s no easy way for everyone, testers and developers and other stakeholders, to follow along and collaborate.

Further, at the end of the day, documenting the result of the test is very difficult. There’s nothing that encapsulates the effort for posterity, allowing the results to be recorded in change management, or for post analysis or benchmark for comparison.

All of this leads to a lack of scalability of the testing process, a tendency to ‘wing it’, and at the end of the day, results that aren’t as good or productive as they could be, yielding a waste of both time and money.

So, what is to be done about this? There are several helpful approaches to improving software testing.

For a decade or more, software vendors have produced various tools for automating testing. This is the holy grail of testing. Just imagine - we start the test, it runs quickly and comes back with exactly what’s wrong! Awesome! Sadly, this approach almost always disappoints. Why? There are too many cases, too many variants, and eventually, maintaining the automated tests becomes as expensive, cumbersome, and yes, unreliable as doing the tests manually. In software of any magnitude, it’s just not a useful, or at least it's not a complete solution to the testing problem, and if anything, it only provides superficial comfort that testing is being done for those not very close to the reality of test quality.

Recent industry efforts have placed emphasis on test-driven-development. This is good and helpful because built-in tests can find issues with modules earlier in the development cycle, when it's less costly to correct them. However, this isn’t a total solution because it’s commonly focused on unit testing (testing parts, technically), and it doesn’t commonly address a myriad of issues that emerge due to integration, 3rd party libraries (especially, with newer techniques like inversion of control and dependency injection), systems or environmental dependencies and, of course, the biggest issue of all, the user experience.

Developing expertise and/or adopting testing methodologies can help, too. The industry has been a-twitter about “centers of excellence” for the last few years. Certainly, focusing efforts can help. In the area of software quality, and testing in particular, one can develop knowledge, insight and skills that aid both the understanding of what can go wrong (i.e., what needs to be tested) and how to screen for potential issues (i.e., how to test). However, this doesn’t change the fact that without good tools, testing becomes difficult to explain, scale, track, assess and repeat consistently.

Here, an attempt has been made to provide insight into why the software industry struggles to produce high quality, well-tested deliverables. Each organization has its own challenges in this regard. Risks related to software defects vary widely. Not every environment or software product necessitates high-quality deliverables. After all, not all software performs functions that are vital to human life or critical to high value transactions. On the other hand, any time there’s an investment in software development, there’s an opportunity to minimize cost while maximizing value. Better software test tools are needed to reach this implicit goal.

¹ http://www.abeacha.com/NIST_press_release_bugs_cost.htm, U.S. Economic impact of software errors per NIST report