By PATRICK
THIBODEAU
Computerworld, June 25, 2002
WASHINGTON -- Software bugs are costing the U.S.
economy an estimated $59.5 billion each year, with more than half of the cost
borne by end users and the remainder by developers and vendors, according to a
new federal study.
Improvements in testing could reduce this cost by about a third, or $22.5
billion, but it won't eliminate all software errors, the study said. Of the
total $59.5 billion cost, users incurred 64% of the cost and developers 36%.
The findings were released yesterday in a 309-page study (download
PDF) conducted by the National Institute of
Standards and Technology (NIST), a federal agency that conducts extensive
research on technology issues.
There are very few markets where "buyers are willing to accept products
that they know are going to malfunction," said Gregory Tassey, the NIST
senior economist who headed the study. "But software is at the extreme end,
in terms of errors or bugs that are in the typical product when it is
sold."
A product of 18 months of research that included extensive feedback from end
users, the study examined the impact of buggy software in several major
industries -- automotive, aerospace and financial services -- and then
extrapolated the results for the U.S. economy. The study was conducted for NIST
by the nonprofit Research Triangle Institute in Research Triangle Park, N.C.
For instance, one case study in the automotive and aerospace industries
involved interviews with 10 software developers and 179 users of computer-aided
design, manufacturing and engineering systems and product data management
software. That study found that users of these software design tools spent
significant resources responding to software errors. Approximately 60% of those
surveyed said they had experienced "significant software errors" in
the previous year. The total cost in these sectors from inadequate software
testing was estimated to be $1.8 billion.
Similar results were found in the financial services sector. In the case
study of that sector, four software developers and 98 software users were
interviewed. According to the study, developers agreed that an improved testing
system was needed that could track a bug back to the point where it was
introduced and show how it influenced the rest of the production process. The
ideal testing infrastructure could remedy problems in real time rather than
requiring developers to wait for the product to be fully assembled, the report
said.
The total cost on financial services from inadequate software testing was
estimated at $3.3 billion, the report said.
"No one thinks you can get all the errors out of software," said
Tassey. Vendors are under pressure to get products to market quickly, and there
are diminishing returns on testing: The more effort you put into it, the fewer
the bugs that are found. "However, the general consensus seems to be [that]
the current state of the art with respect to testing is poor and can be
sufficiently improved," he said.
The study didn't propose specific actions for improving testing but called
for the development of testing standards, noting that today's testing tools
"are still fairly primitive."
The study also said that standardized testing tools, scripts, reference data
and metrics, among other things, "that have undergone a rigorous
certification process would have a large impact on the inadequacies" now
found.