The Elusive Definition of Success with Software Projects

November 27, 2009

What is the state of software projects today? Are software projects more successful today than they have been in years past? Think about any software project that you know of that is just getting started. How confident are you that the project will be considered a success?

One problem with software projects is determining the definition of success. Consider the following, classic definition of a successful software project.

Delivery of software:
  • On Time
  • On Budget
  • Of High Quality
  • Containing the Expected Features
Does this meet your own definition of success? More importantly, just how is the software industry doing in terms of projects being completed and meeting this classic measure?

A 2006 Chaos Report from The Standish Group indicates that software projects are improving over time, as demonstrated by a similar study that they conducted in 1994. Let’s take a look at the numbers and how they contrast over the twelve-year span.

According to The Standish Group, 19% of the projects reported on in 2006 were outright failures, which is certainly an improvement over the 31.1% failure rate reported in 1994. A look at the rest of the figures reveals a different problem, where projects move from failure to “challenged.” A challenged software project is defined as failing to meet the classic definition of success in one or more ways, such as incurring a budget or schedule overrun, or having features trimmed from the final delivery, but the project itself is not deemed to be a failure.

The 2006 Chaos Report states that 35% of the software projects succeeded in meeting the on time, on budget, and expected features criteria during 2006, relative to only 16.2% in 1994. (While not explicitly stated, the assumption is that the quality objectives were met as well.) If we remove the 19% of outright project failures for 2006, we arrive at a significant, 46% figure for software projects that were in the challenged category during 2006.

Although these figures paint a dismal picture of the software industry, they only represent a piece of the puzzle. In fact, there is – and should be – a modified view of success. This was illustrated in a 2008 survey conducted by Scott Ambler, who had an overall goal to explore how Information Technology (IT) professionals define project success.

There were 279 respondents to Scott’s survey, and their definition of success in comparison to the classic definition is as follows:
  • On Time: 58% believe that delivering when the system is ready to be shipped is more important than delivering on schedule.

  • On Budget: 70% believe that providing the best ROI is more important than delivering under budget.

  • Of High Quality: 82% believe that delivering high quality is more important than delivering on time and on budget.

  • Delivered with the Expected Features: 83% believe that meeting actual needs of stakeholders is more important than building the system to specification.
After reading the survey and noting the responses to the classic measure of success, I reflected on past experiences and conversations that I’ve had with others about the nature of software success. In general, the measure of success to a great degree is a function of organizational culture. I’ve talked to some frustrated project managers and developers who have told me that in their company, teams that deliver on time are recognized as successful over other teams who deliver as little as a week late.

The real rub is that teams that delivered on time is still regarded as successful in spite of the fact that their software required substantial defect-fixing months after delivery, while the teams that were only marginally late delivered with higher quality, and did not need to allocate substantial time and effort to support a product of poor quality – and were able to move on to more productive ventures.

I decided perform a modest cross-check of the results of my own by posting a question on LinkedIn in the Software Development category. I cited the survey and asked: “What should the definition of success be? Are we being more successful today than yesterday because we are changing the definition of success?”

Needless to say, I received a range of responses. There were answers that qualified success as meeting the business objectives of the project, while other answers were in the “it depends” category. Overall, while there was agreement with the classic definition in principle, there was a lot of deferring back to what the stakeholders of the project defined as successful.

This leaves us in a situation where not only are software projects challenged, but the very definition of success is also being challenged. It is certainly in the best interest of everyone to have a candid conversation about the definition of success before taking on a project. Above all, acknowledge that there will be challenges, such as people changing their minds about the requirements, or details surfacing that change everyone's understanding of the original requirements.

And don't expect to have it all. The classic success criteria is about predictability, and software projects are rarely predictable. I wrote about this in my Can Software Projects be Predictable? post.

Scott Ambler’s December 2008 Software Development Project Success Survey

Standish Group Report: There’s Less Development Chaos Today by David Rubinstien, March 1, 2007

Software Magazine: Project Success Rates Improved Over 10 Years