Agile Development Increases Productivity

July 9, 2010

In my post, What is Better About Agile Development? (a reprint of the article Top 10 Reasons to Use Agile Development that I wrote for DevX.com), my fourth reason for using Agile development is that Agile development increases productivity. This post will expand upon this topic.

Before I begin, let me state that I understand that traditional projects can be successful and productive. Many people point out – rightly so – that many aspects of Agile development are not new. However, the question of productivity as it relates to Agile development is worth exploring. Even if you don’t want to be an Agile shop, perhaps there are aspects that you can adopt that will improve your situation…

Surveys and studies have attempted to determine to determine if Agile development is improving– or least providing a strong perception of improving – productivity. While everyone is not reporting productivity improvements, a high percent certainly are. Let’s take a quick look at some percentages:

82 percent reported a “somewhat higher or much higher productivity improvement” in Dr. Dobb’s February 2008 Agile Adoption Rate Survey Results.

73 percent reported “improved or significantly improved” productivity in VersionOne’s 2009 State of Agile Survey.

83 percent reported an improvement in productivity at Lockheed Martin, as reported at the Agile Vancouver: Much Ado About Agile, Sep 2, 2007 by Mike Zwicker in a presentation titled War Stories: Fighter Jets and Agile Development at Lockheed Martin.

Reaching further back, 93 percent stated that productivity was “better or significantly better” in the Shine Technologies 2003 Survey.

How is this happening?

Practices
Scrum is taking the lead as the Agile method of choice, but there is growing acknowledgement that a Scrum/XP hybrid provides more along the lines of total productivity improvement. My article focused attention primarily on the autonomous, teamwork orientation that Scrum provides – although I admit that I should have mentioned something about technical practices.

Some key practices of Agile teams employ that improve productivity:

Automated Testing: Ideally, these are small tests that are capable of being executed many times a day and can confirm that a newly-developed feature is working correctly as well as confirming that pre-existing features are continuing to function as expected.

Automated Unit Tests/Test-Driven Development (TDD): Agile development insists upon developer unit testing, and the best way to accomplish this is through automated testing. TDD is a “test first” approach where a unit test is written before the code is written; the test fails and the developer must code to make the test pass. This approach has the advantage of forcing developers to think more about their interfaces, helping to make the software more understandable and self-documenting.

We’ve been increasing our attention on automated unit tests, and the first benefit we obtained was that our developers were delighted that they could re-factor with confidence. And QA was more than grateful that automated tests quickly and easily proved that the behavior of the software did not change, with no manual testing required on their part. In the end, our velocity improved, allowing us to develop additional features that we could not have completed without the initial investment to build our suite of automated unit tests. (This investment was made gradually as we built out new functionality – although we did need to invest time in creating tests for code that was already created without automated unit tests.)

Code Reviews/Pair Programming: The benefits of code reviews have been known to the industry for many years; equally known is that the earlier a defect is detected, the cheaper it is to correct. Catching coding problems is best accomplished by a code review. Pair programming is one way to accomplish a real-time code review.

A recent Forrester study, The Value and Importance of Code Reviews, published in March, 2010 (commissioned by Klocwork) found that code reviews are considered to be of high value by those who conduct code reviews. The benefits cited:

  • Bugs are uncovered earlier in the life cycle.
  • Best practices are shared.
  • Refactoring and code simplification are encouraged.
  • Reduction in wasted time (later in the project).
There is a way to combine the development of automated unit tests and code reviews. We have not tried this out in our office (yet), but I did attend a recent training course where we conducted a side session that combined Test-Driven Development and Pair Programming. One programmer wrote the test, and the other programmer implemented the code. There was continual dialog about the business feature and the design and implementation as we worked through the example. The end result was a shared understanding of the code, solid, simpler code (we simplified the instructor's sample code), and an automated unit test. Does anyone have real-world experience with this technique that they would like to share on a guest post?

Continuous Integration: Instead of keeping portions of a system separate until the end of a project, Agile projects integrate and deliver working software early and continuously. This avoids problems of late-stage integration that some projects suffer, and helps to build a greater degree of quality into the system through the testing of features through the architectural layers.

Insistence on Quality: Agile development has a philosophy that quality comes first. Features should not be considered done until they are of high quality; defect-fixing is not an activity that occurs late in the project cycle. In the event that a defect is found with a feature that was declared as done, work on a lower-priority feature should be deferred in favor of making sure that the higher-priority feature is of high quality.

Teamwork
As I noted, my article focused on the team-oriented aspects of how Agile improves productivity. I made the statement that, “Agile development focuses the team's attention like a laser on delivering the highest-priority, highest-valued features in short increments of time using the most productive mechanisms to accomplish the work.”

Part of the “most productive mechanisms” are the practices noted above. Focusing on the highest-priority, highest-valued features is just that – a concentrated, team-based effort. We’ve found through our own experiences that teams should limit their work in process to maximize productivity.

We use two-week sprints our organization, and teams make commitments to accomplish certain work within that two-week period. It is tempting for teams to start as much of that work as early as possible, but I advise caution. There can be side-effects:
  1. Less collaboration occurs, and you end up with a team that is a collection of individuals working on their own tasks, leading to point number 2.
  2. Teams may end a sprint with a lot of work in process, but nothing actually completed.
Limiting work in process forces teams to truly discuss the highest priority items and finding ways to collaborate and truly share and own the work as a team. You might get pushback on this, as people tend to react with “there isn’t anything that we can all work on,” but there is usually ways in which people can collaborate.

We’ve also experienced the benefits of establishing a truly collaborative team environment. We had one team that did nothing other than change how they worked. We had a classic case of a “team” that was simply a collection of individuals working individually on their tasks. There was little to no collaboration, functional hand-offs, and plenty of overtime.

We started over by grooming the backlog and discussing – really discussing – each feature. We found that some features were vague, and others were unnecessarily complex based on the goals that the business had in mind. By having a real dialog with the Product Owner, we were able to make changes that met the business needs more effectively, and in a way that was simpler to implement from a development standpoint.

The day-to-day work of the team changed as well. Everyone tasked out each User Story, discussed the validations, and collectively estimated the work (including discussing any deviations in the estimates). And during the course of implementing a User Story, the team collectively discussed the impact of any change – yes, issues surfaced as the team got into the details of some features. But there was a collective understanding and full team ownership of each and every feature.

The result? Without adding or changing any technical practices, focusing solely on teamwork and building true collaboration, the team achieved a little over 30 percent increase in productivity. Our measure was based on the team’s estimate of when they would complete the work based on their old approach and velocity. We simply beat it, and this included eliminating the overtime that the team was working.

This brings me to another point that I made in the article. Productivity increase comes from teams operating autonomously, where the teams make informed decisions about their day-to-day work without the need for constant managerial direction. Not only does this expedite certain decisions by keeping them localized, employees who have more control over their day-to-day work are more energized and engaged about their work.

A final note about improving productivity through teamwork that I did not mention in my article (one that we’ve also proven in our experience), is that small teams are more productive. This has also been proven in other studies, such as those made by Quantitative Software Management (QSM), a company that maintains a comprehensive database of software development project metrics.

Here’s a pop quiz for you, based on QSM data: There are two software teams that must deliver a project with a size that is estimated to be about 40,000 lines of code. One team, Team Large, has 29 resources compared to Team Small, with 3 resources. How much faster do you think Team Large will be?

According to the QSM data, the difference would be a mere 12 calendar days! The other significant differences would be the fact that Team Large would consume 151 more person-months more Team Small, and Team Large (on average) would create six times as many defects! The additional lines of communication and overhead associated with large project teams create more problems than are solved.

Hyper-Productivity
Jeff Sutherland coined the term Hyper-Productivity, defining it as a 400 percent higher velocity improvement. It’s an impressive marketing term, but does Hyper-Productivity exist?

It’s possible, based on studies. In his article, An analysis of XP, TDD, Pair Programming, and Scrum (Using Real Options), Dr. David F. Rico arrived at the following figures:



Low
Median
High
Productivity Improvement
14%
122%
712%
Quality Improvement
10%
70%
1,000%
Schedule Improvement
11%
71%
19%

I don’t think that everyone can improve by 400 percent or more; my personal opinion is that this depends entirely on where you start from. If you have a good sense of software development and you are following a sound methodology that includes good technical practices, I still feel that you can improve, but not to a dramatic degree that places you in the Hyper-Productivity category. What are your thoughts about Hyper-Productivity?