Agile Development Should Be Sustainable Development

July 13, 2010

In my post, What is Better About Agile Development? (a reprint of the article Top 10 Reasons to Use Agile Development that I wrote for, my fifth reason for using Agile development is that Agile development creates a sustainable development environment. This post will expand upon this topic.

As a software development manager, my responsibility is to make sure that our development organization produces working software day in and day out, year after year. And like many other managers out there, it seems that every year I’m being asked to do more with less. I can accomplish this in one of two ways:
  1. Squeeze my staff for all they are worth (and replace them out when they burn out).
  2. Work smarter.
I personally like creating a collaborative, fun, sustainable work environment that is focused on retaining good employees and not working them into the ground to meet imposed, aggressive (ridiculous) deadlines that are based on little more than hopes and dreams. Call me crazy, but I like treating professional knowledge workers as professionals. And I enjoy seeing people develop and grow – it’s truly rewarding to see people reach new heights!

Sustainable software development is all about meeting the needs of the business on an ongoing basis, which is typically translated into meeting project schedules. There are other ways of meeting the needs of the business, which I discussed in Agile Development Embraces Business Agility. The issue of software project schedules and the usual overtime response in order to meet those schedules is worth examining in term of productivity and results.

In my article I stated that, “Software projects rarely meet their initial schedule. Overtime with the mandate to ‘work harder’ is frequently used as it becomes apparent that a project will be running longer than anticipated.”

This has been an all too-common experience for many software professionals, and a great deal of anecdotal evidence and various studies that support that schedule overruns are frequent:

In a Dynamic Markets Limited Independent Market Research Report commissioned by Tara Consultancy Services in August 2007, 62 percent of the IT projects failed to meet their anticipated project schedule.

The latest Standish Group report in 2009 report shows that only 32 percent of IT projects were considered successful – meeting the schedule, budget, and delivered with the expected features.

In his book, Software Runaways: Lessons Learned from Massive Software Project Failures, Robert Glass found that schedule overruns were very common, pegging schedule overruns at 89 percent. (The book profiled 16 of the largest and most publicized software failures of the 1990s.)

Using overtime as a mechanism to deal with software project schedule issues creates far more problems than it solves, and generally inhibits the creation of a viable, productive, long-term organization. The longer-term risks of burnout, turnover, and overall sustainability are commonly acknowledged (and frequently ignored), but the short-term effects are rarely mentioned.

In the first place, the productivity of knowledge workers (like programmers) plunges with heavy doses of overtime. Research from The Journal of American Epidemiology – studying people who work greater the 55 hours – found that individuals had problems with memory, problems with functioning appropriately and problems with reading and comprehension.

From a software development standpoint, a major short-term issue associated directly with overtime is quality. In a paper, Impact of Overtime and Stress on Software Quality by Balaji Akula & James Cusick, Baleji and James focused on the impact of project teams working on an aggressive schedule, studying four projects over a two-year period (three of which had an aggressive schedule).

The chart below demonstrates a dramatic difference in defect rates when no overtime is involved:

Project #1
Project #2
Project #3
Project #4
Estimated person hours
Overtime person hours

I’m quite sure that I know why defect rates are low when no overtime is present; I’ve experienced this many times as both a programmer and as a manager. Unrealistic schedule pressure and overtime not only introduces fatigue that produces more defects, programmers stop thinking as deeply and carefully about their work in response to the pressure to meet the deadline. Corner-cutting begins to creep in at every turn as output becomes a major focus – at the expense of practices that help to produce more reliable code.

Overtime has other problems as well. Since programmers are knowledge workers, they need to putting knowledge into their “knowledge banks,” something that will pay dividends later. Routine overtime on routine project tasks eliminates the time for the professional growth that will be of future benefit. (And in other cases, people are doing themselves a great disservice by failing to continually acquire new knowledge, as Pawel Brodzinski pointed out in a post, People Don’t Want to Learn.)

Then there is the whole work/life balance issue. My wife, Lauri-Ann, has referred to herself as a “software widow” at times when I’ve been very involved – and I mean working some very long hours – on software projects. You can tell when you are tipping the scales in favor of work too much when your wife pleads with you to put the computer aside to watch your daughter play soccer on a Saturday morning. (True story, Lauri-Ann almost had to hit me over the head.) In the long run, this isn’t healthy…

Sustainable Development
I support Agile development because it promotes sustainable development as one of its 12 Principles, as found in the Principles behind the Agile Manifesto. The principle states, “Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.”

Agile development doesn't try to predict the future by estimates alone, let alone try to make that future happen via overtime. With Agile development, the future can be predicted based on actual team velocity. Agile teams size User Stories in points and tracks team velocity – the User Story points that are completed in each 2-4 week iteration.

Stories are considered done when they are delivered as working software with high quality. Over a period of time, it becomes crystal clear how much work can be realistically accomplished by a team on a sustainable basis. Future delivery can be predicted by contrasting the team velocity against the active product backlog and the size of the User Stories contained in that backlog.

Once you have a clear picture of what is realistic and sustainable, how should you seek to improve team performance? It shouldn’t be about working harder, it should be about working smarter. How is the collaboration? Are there personnel issues or friction? Are practices such as Test-Driven Development being utilized? I’ve witnesses Agile teams working very productively, accomplishing more than they were when they were working overtime through the proper implementation of Agile development. The output was higher, it was sustainable, and morale was better.

When it comes to having developers maintain their skills and working smarter, one contribution that Agile development brings to the table (if you choose to use it) is the concept of slack as a technique to provide for research time each iteration. I first encountered this concept in the book, The Art of Agile Development by James Shore and Shane Warden. They state:

"To keep up with their constantly expanding field, programmers must continually improve their skills. In doing so, they will often learn things that enhance their work on the project.

"Dedicated research time is an excellent way to encourage learning and add additional slack into your iterations. To introduce it, set aside half a day for each programmer to conduct self-directed research on a topic of his choice. Be completely hands-off. I recommend only two rules: don't spend this time on project stories or tasks, and don't modify any project code."

What about teams meeting their commitments? Shouldn't they work overtime to meet what they committed to? If you have a team that struggles to meet its commitments every now and then, you actually have a great situation! It means that they are taking on work that tests and stretches them. It should be a judgement call whether to roll the work into the next iteration or deal with it now. It really depends upon the urgency and circumstances.

If there is a critical customer deliverable with a tight timeframe, a good team will recognize that and get the job done, and they will set aside their research time to make things happen. And yes, there are times that as a manager I've had to remind teams that a customer was waiting on us, and that yes, working over the weekend was necessary. There is nothing wrong with a spike of activity from time to time.

There are other circumstances when it is OK to let the work slide. Software development isn't highly predictable, and what appears simple when it is estimated can turn out to be more difficult in the actual implementation. In this case, why punish people by mandating overtime? Since it is a difficult problem, taking the extra time to deal with it – keeping people fresh at the same time – will improve your odds that it will be dealt with correctly the first time.

I've seen comments on LinkedIn and on other blogs that lead me to believe that there are some very wrong implementations of Agile development out there. Instead of sustainable development, the term "Agile" is being misused and abused to drive a series of short-term, "mini death marches" where the goal is to code quickly (without thinking) and to drive the output of the team in unrealistic and unhealthy ways. Unfortunately, this is the very issue that Agile development seeks to overcome.

Don’t be misled, though. Agile development takes discipline. It takes knowledgeable, capable, engaged people. It takes a supportive management team. It takes training and time. It is work and it is change, but in the end I believe that it is worth it. Regular use of overtime should be viewed as a serious problem.