Study Shows a Shift Away from Overtime as a Remedy For Poor Estimation

January 9, 2009 - Arlington, Massachusetts

Performance of Poorly Performing Organizations Has Declined Even Further

October 30, 2008 - Arlington, Massachusetts

E.M. Bennatan


One of the most costly results of poor estimation skills is often the complete cancellation of a project.

Cutter Consortium recently examined the extent to which software organizations have abandoned or cancelled projects over the past three years due to significant budget or schedule overruns. This survey effort studied software project estimation at more than 100 software development organizations and was analyzed by Cutter Consortium Senior Consultant E.M. Bennatan.

The first area we examined was comparative performance; how are projects estimated today compared to six years ago? We defined success by the ±10% rule: success means hitting the mark within 10%.

Estimating Project Schedule

Organizations were asked: In the past three years, what would you say is the percentage of your organization's software projects that are developed within plus or minus 10% of the original estimated time?

Reports Bennatan, "Just 33% of surveyed software organizations report a success rate of more than 70%, which means that more than 70% of their projects were developed on time [8% of these reported a success rate of more than 90%].

This effort was a repeat of a 2002 study, also analyzed by Mr. Bennatan, that exposed software estimation as "a tough beast to control."

"It is interesting to note that the numbers are not radically different from the 2002 survey, when 35% reported a 70% success rate [of which 14% had a success rate of more than 90%].

"In the mid-to-lower end of the success scale, the differences were more profound. The current survey found that 36% of organizations had a schedule success rate of between 50% and 70%. This is up from just 26% in 2002. 24% had a success rate of less than 50% (an improvement over the 36% in 2002). Total failure -- no on-time projects -- was reported by 8% of organizations (3% in 2002)."

"Our 2008 survey shows a small shift toward the center. This means that while schedule estimates today are not as poor on the low end of the scale as they were in 2002, they are also not as good on the high end."

Figure 1

In the past three years, what would you say is the percentage of your organization's software projects that are developed within plus or minus 10% of the original estimated time?

Estimating Project Cost

Bennatan next examined the survey findings for budget estimates and compared them to the 2002 findings. According to the data, 37% of software organizations had a budget success rate of 70% or more (of which 10% had a greater than 90% success rate). "This is surprisingly similar to the 2002 survey findings, where 38% had a 70+% success rate [with 10% of them reporting a 90+% success rate]."

Here, too, the mid-to-lower end of the budget success scale showed more significant differences. The new survey found that 32% had a success rate between 50% and 70% (24% in 2002), and 24% had a less than 50% success rate (34% in 2002). Total budget failure (no project completed within budget) was identical to total schedule failure, and was reported by 8% (4% in 2002). "Here again," says Bennatan, "we found a shift toward the center, similar to the schedule estimate findings."

Are Estimate Disasters Now Less Common?

When asked if, in the past three years, any of your organization's software projects been abandoned or cancelled due to significant cost or time overruns, just over half (52%) of surveyed software organizations report having had no projects cancelled or abandoned; in the 2002 survey, the figure was 56%. A further 29% report that up to 10% of their projects were cancelled (also 29% in 2002), and 17% state that between 10% and 25% had been abandoned (11% in 2002). Beyond that, the numbers were minuscule. Just 2% report that the proportion of cancelled projects was between 25% and 50% (3% in 2002), and no surveyed companies had more than 50% of their projects abandoned (it was 1% in 2002).

Says, Mr. Bennatan, "It was quite surprising to see that project disasters have actually increased (for companies with more than 10% of project failures). In other words, the performance of poorly performing organizations has declined even further.

Figure 2

In the past 3 years, what percentage of your organization's software projects have been abandoned or cancelled due to significant cost or time overruns?

Only 37% of software organizations believe that their ability to estimate software projects has improved over the last three years, down from 51% in 2002. 58% see no change (43% in 2002) and 5% actually see a decline in their ability to estimate (6% in 2002). "Either estimation skills are leveling out," muses Bennatan, "or software projects are becoming more difficult to estimate."

So, in six years, what has changed? Says Bennatan, "Most areas of computer technology have improved immensely in the past six years. In software development, we have learned the benefits of shorter, more agile projects, and even for larger project, we have adopted a more evolutionary approach. But while new tools, techniques, and methodologies abound, the survey findings do not indicate that they are having any appreciable effect on our ability to estimate."

To request a copy of the Agile Product & Project Management Executive Update (Vol. 9, No. 18), containing E.M. Bennatan's analysis, or to schedule an interview with Mr. Bennatan, contact Kim Leonard (+1 781 641 5111 or kleonard@cutter.com).

More findings.

About Cutter Consortium

Press Release: Performance of Poorly Performing Organizations Has Declined Even Further

Why Choose Cutter?

Cutter challenges the traditional analyst model with its revolutionary approach to research, consulting and training.

Learn More »