Agile Development Can Improve Your ROI

June 29, 2010

Introduction
In my post, What is Better About Agile Development? (a reprint of the article Top 10 Reasons to Use Agile Development that I wrote for DevX.com), I listed Superior ROI as the first reason for using Agile development. This post will explore the topic in greater detail.

I set a condition in my article that Agile projects provide a superior return on investment compared for other software projects where “the business is forced to wait for the completion of the entire project before it can begin deriving a benefit from that investment.” I’m therefore comparing traditional, plan-driven projects to Agile.

I recognize that there are hybrid solutions in use, but I’m using these comparisons to highlight the benefit of Agile development against traditional development in the purest sense – to examine what is working that we should all be considering, regardless of the development methodology in use.

Just to keep definitions clear, a traditional, plan-driven approach is one that operates with formalized phases of phases of requirements, design, construction, testing, and maintenance, completing each phase before moving to the next. These types of projects strive to be predictive. Conversely, Agile development is an umbrella term that encompasses several actual methods (XP, Scrum, etc.), but in general Agile projects are incremental and adaptive.

And the Survey Says...
Leading off, Scott Ambler’s December 2008 Software Development Project Success Survey specifically asked respondents about their ROI and included this in the results. Scott used a formula to derive an effectiveness score, and Agile development’s ROI score was 3.9 versus 0.2 for traditional development on a scale of -10 to +10.

In addition, a detailed treatment of various studies has been performed by David F. Rico in a couple of articles, and he includes a long list of references (and commonly cited sources) that serve as a handy reference if you want to examine a variety of Agile development studies for yourself.

In his article, What is the Return on Investment (ROI) of Agile Methods?, David Rico’s stated purpose was “to examine and identify the ROI of agile methods. More specifically, its purpose was to identify the ROI of using agile methods in their entirety, not just some of the tools and techniques of agile methods like pair programming.”

His conclusion: “The benefits of using agile methods range from 10% to 100% for increased cost-effectiveness, productivity, quality, cycle-time reduction, and customer satisfaction. The use of agile methods as a new product development approach does result in increased ROI. This begins to dispel the notion that agile methods result in lower ROI than traditional methods.”

In another article, What is the ROI of Agile vs. Traditional Methods? An Analysis of XP, TDD, Pair Programming, and Scrum (Using Real Options), David Rico’s goal was “examine scholarly studies of Agile Methods and survey the range of quantitative costs and benefits associated with the use of Agile Methods. Data were compared to costs and benefits of Traditional Methods such as CMMI®.”

David concluded that, “On average, studies of Agile Methods reported 29% better cost, 91% better schedule, 97% better productivity, 50% better quality, 400% better satisfaction, and 470% better ROI than CMMI®.”

He also made another good observation: “The latest trend is to mix-and-match Scrum and XP to tap into practices like Pair Programming (PP) and Test-Driven Development (TDD) to increase productivity and quality.” One Agile method alone is not the recipe for success, and like a lot of things, we need to look under the covers to discover what is really working. Some of these are practices that can be employed on any project.

The Numbers...
In terms of working the numbers, John Scumniotales has a nice post, Why incremental development is better - An ROI perspective, that demonstrates how delivering early and often improves ROI. John also provides an ROI/NPV Excel spreadsheet calculator that can be downloaded so you can play with the numbers for yourself, as your mileage may vary. John’s sample project contrasts a traditional (non-incremental) approach versus an Agile (incremental) approach:


Dennis Stevens made a solid observation in the comments section of John’s post, stating, “Some features are more valuable than other features… Higher earlier benefit actually creates a bigger spread.”

These comments address another observation that I made in my article and something that is important for improving results: “Agile does ask for discipline and participation from the business as part of the deal, such as rigorously prioritizing the features…” and, “In return, the business gets its highest-valued features delivered early…”

Our Experience...
We’ve experienced superior return at my own company as we’ve transitioned to Agile. (My posts don’t delve too far into the specifics of my own company to avoid any entanglements with competitive disclosure or breeching of confidences.)

In the past, we’ve used the traditional plan-driven development approach. We performed the classic requirements elicitation, refinement and development, and then moved on to design and build out the architecture in its entirety prior to plugging in the business features. I’ve been involved with projects that have required – based on the defined feature set and staffing – over one year to complete before we could release and begin generating a return.

In comparison, we initiated the development of a brand new product after our adoption of Agile development. This product was released much earlier, allowing us to begin generating returns much sooner than would have been possible with a traditional approach. In fact, we delivered this product to one customer before our first, official release. We’ve benefited greatly from an early return on investment along with obtaining early feedback.

The feedback that our customer provided us allowed us to make adjustments to features that we thought we had right, but didn’t. We thus obtained two benefits in one: we generated a return very early in the product cycle, and our official product release was stronger as a result.

We’ve also taken advantage of another aspect of Agile development: adaptability. We appeared to have a solid feature set defined, but feedback from customers have led us to adding new, compelling features and re-prioritizing our product backlog to work on those valuable features now. We aren’t measuring ourselves against “planning the work and working the plan”; instead, we’re adapting and responding to the needs of the market.

How Does Your Project Portfolio Look?

ROI from a single project is one thing, but many companies have a portfolio of projects, and it is worth considering what the return of the entire project portfolio. A failed software project obviously provides no return and can easily drain significant capital from a company, placing a greater emphasis and need for other projects to succeed.

Every year, the much quoted Standish Group Chaos Survey reports outright project failure or “challenged” projects – projects that fail to meet the classic definition of success in one or more ways, such as incurring a budget or schedule overrun, or having features trimmed from the final delivery, as noted in the table below:

2009
2006
2004
2002
2000
1998
1996
1994
Successful
32%
35%
29%
34%
28%
26%
27%
16%
Challenged
44%
46%
53%
51%
49%
46%
33%
53%
Failed
24%
19%
18%
15%
23%
28%
40%
31%

The Chaos Survey paints a dismal picture of the software industry, but it is worth noting that the very definition of software project success can and is disputed, as demonstrated in Scott Ambler’s December 2008 Software Development Project Success Survey. There were 279 respondents to Scott’s survey, and their definition of success in comparison to the classic definition is as follows:

  • On Time: 58% believe that delivering when the system is ready to be shipped is more important than delivering on schedule.
  • On Budget: 70% believe that providing the best ROI is more important than delivering under budget.
  • Of High Quality: 82% believe that delivering high quality is more important than delivering on time and on budget.
  • Delivered with the Expected Features: 83% believe that meeting actual needs of stakeholders is more important than building the system to specification.
Given that definition of success is disputable, it is certainly in the best interest of everyone to have a candid conversation about the definition of success before taking on a project! But the main point is that one disputes that delivering all of the software features, on time, with high quality, and on budget is a problem. Scott Ambler’s survey simply challenges the definition of success. (Projects that are challenged can also generate more than the anticipated benefits, but nothing is certain until your product is in use.)

Standish Chairman Jim Johnson was quoted about the improvement in success rates saying, "The primary reason is the projects have gotten a lot smaller. Doing projects with iterative processing as opposed to the waterfall method, which called for all project requirements to be defined up front, is a major step forward." (This comment was made prior to 2010, and I don’t have anything definitive about the dip in success rates. Does anyone have any insight on this?)

Focusing on the industry averages of Agile project success rates, the success rates are not perfect, but they are very good. The 2009 State of Agile Development Survey sponsored by VersionOne, includes data from 2,570 participants from 88 countries and reports that 23 percent of the respondents indicated that they had not experienced a failed Agile project. This is an improvement from 17.4 percent who reported no Agile project failures in VersionOne’s 2008 report, and in contrast to the Chaos Report’s downturn in overall software project successes.

I also took a look at the figures from Scott Ambler’s survey. Success in Scott’s survey was defined by the respondent, and not subject to a stricter definition such as the “on time, on budget, with quality and with the expected features” criteria used by the Chaos report. I’m fine with this, since a great many are arguing that the Chaos criterion is flawed.

I’ll stick to the Agile versus Traditional (plan-driven) comparison. In keeping with the spirit of “what’s working,” Scott’s survey included examining success rates by location – the closer the team, the greater the odds of success – summarized as follows:


Success Rates
Agile
Traditional
Co-located
79%
73%
Near located
73%
69%
Far located
55%
55%

Co-located means that everyone, including primary stakeholders, are in the same room.

Near located means that everyone is in the same geographic area, and could be on different floors, in different buildings, or working at home, but could potentially get together each day for a group meeting if need be.

Far located means that people would need to plane to attend a physical meeting of the entire team.

Overall, these studies indicate that nothing is perfect, but Agile improves your odds of success, and greater project success contributes to the overall return of your organization.

Conclusion
Summarizing what’s working in general for Agile software development projects, I’d have to say that the findings support that smaller, co-located projects that deliver incrementally allow a greater and earlier return on investment, and have a slightly greater chance of success than larger projects and/or distributed project teams. Finally, practices such as pair programming and test-driven development should be utilized to achieve productivity gains.

What is your take?

6 comments

I think the observation that teams (including stakeholders) working closer together tend to be more successful than others makes sense. I believe it has to do with the importance of communication and trust.

I suppose communication is one of the main reasons why agile and lean methodologies perform better than BDUF (waterfall and co.) as they allow the stakeholders to provide feedback faster and steer the project to the right direction.

I guess distributed development (virtual organizations) in the realm of communal open source projects might provide some form of exception to the rule. In this case developers themselves are stakeholders.

This changes things quite a bit as you are dealing with different kind of project dynamics. Of course you have to measure success differently in this case. Conventional metrics (goals, time, budget, quality, etc.) just won't cut it always.

June 29, 2010 at 2:21 PM
Dave Moran said...

Juho,

Yes, close proximity seems to enable the right communication at the right time, and perhaps even more communication, that leads to success.

I also agree that things change quite a bit as you deal with different kinds of project dynamics (and different people with different experience, strengths, etc.).

June 29, 2010 at 9:22 PM
kousalya said...

What an exciting experience!/Hilarious! Delightful! True!/wonderful stuff! thank you!

Software Product Development

April 9, 2011 at 2:47 AM
Dave Moran said...

@kousalya -- Thanks for reading. and I' glad that you enjoyed it!

April 14, 2011 at 7:58 AM
Anonymous said...

I think a next article topic could be,"Quality differences between agile and other traditional approaches". I have seen Agile work really well, and seen it perform miserably. One thing that all of the latest agile projects, that I have seen, have in common is bad quality. Bad user experience, no checking for anything except the happy path, file not found errors, etc... If you only do unit testing with a quick run through the happy path and say it is tested, you get software that looks like it was made in China.

January 4, 2012 at 3:59 PM

Post a Comment