Leverage Evidence in Software Estimation

June 24, 2011

Agile development and business users/sponsors are at two opposite ends of a spectrum in one respect. Developers want the business to be entirely flexible on scope or dates in reaction to death-march projects (and these are extremely painful) and the seemingly continual expectation that developers spend their days, nights, and weekends writing software – with no life whatsoever – to “satisfy the business.” The business side, naturally, wants what it wants, for a set price and delivered by a specific date.

Business people don’t care to invest their time in understanding why software projects are complex and uncertain, either. At least not in any great detail. What is one approach that you can take?

Go where the evidence tells you. With software projects, it’s all about the evidence of past experience; the evidence of industry experience and the evidence of your own experience.

Industry Experience
There seems to be a “30 percent rule” involving software projects: You’re going to spend 30 percent of your time figuring out what you’ve gotten yourself into.

Winston Royce is credited with describing what is known today as the Waterfall model in his paper, Managing the Development of Large Software Systems, published in 1970. Royce did NOT use the term Waterfall, in the paper (although he did include what is considered a classic Waterfall diagram in that paper), nor did Royce recommend using a single-pass Waterfall model.

In fact, as far back as 1970 Royce was recommending using iterations and spending roughly 30 percent of a project’s time in prototyping. The goal was to learn and apply information to the actual development effort. This 30 percent ballpark figure has shown up in other research.

Steve McConnell points out in his book, Estimation: Demystifying the Black Art, that software projects are most unpredictable in the early stages of a project. McConnell observes that software projects typically reduce their variability at about 20 to 30 percent into the project because of the learning that has taken place. McConnell graphically represents this in his Cone of Uncertainty:

As you can see, the cone is largest the earlier in the traditional project life cycle, but it narrows from uncertainty to certainty as different stages are reached – as those on the team learn more about what they’re involved with.

McConnell makes another equally important observation. In an effort to obtain an accurate end date, someone might ask, “What if we give an extra week to work on your estimate, can you refine it so that it contains less uncertainty?” Unfortunately, improving estimates requires working out the variability that drives the uncertainty, which is best done by performing the actual work of the project, so the answer is “no.”

You can project dates and costs based on the desired functionality, but until those on the project team rolls up their sleeves and gets to work on the project, you won’t really know if those estimates are reliable; and the odds are, they aren’t. The great part about Agile development is that you’re routinely assessing your projections against the evidence of actual, sustainable velocity and the frequent delivery of features. It’s a great reality check.

The Evidence of Your Own Experience
One question that I’m sure comes to mind is, “What about customer contracts? We can have greater flexibility with our internal projects, but what about contracts that involve fixed bids?”

Fixed bids always make anyone in the software industry cringe. They are the least desirable option, but in today’s world the expectations of internal stakeholders consistently gravitate towards the cost, scope, and schedule Triple Constraint (with quality assumed). As nice as it is to avoid this, it’s a reality that needs to be faced more often than not.

If you are bidding on a project, you must commit to a date and a cost. Use the evidence of past projects. Unless you are operating in a domain where you lack prior experience, you have some past experience – and evidence – to draw upon. But don’t rely on just the actual completion times and costs, there are other factors to consider.

Do the present customer attributes match those of comparable past projects? If not, what are the deviations and how do those factors influence your bid? It is important to combine your evidence of prior, related projects with an assessment of the customer in order to arrive at a reliable bid.

Some customer-assessment questions to consider:
  • Do you have prior experience with this customer, and has trust been established as result?
  • Does the customer appear flexible or demanding? Is the customer seeking a true partnership, where you can work together for mutual benefit, or is the customer negotiating for the “best” possible terms related to scope, cost, and schedule?
  • Does this customer have a crisp, well-articulated idea of what they want, with their own equivalent of a “’single, wring-able neck?” (Decision-maker)
The evidence of past experience of similar projects and the type of relationship that you have with the customer dictates your approach. If the customer is willing to be an active partner with you, then greater flexibility is possible. And keep in mind that finding a way to drive a quick win early on helps to establish trust for the more complex work that comes later.

Other customers may want to keep you at arms-distance and be more “business-like,” where you are on the hook to deliver what you said, when you said, and for how much you said it would cost. Of course, changes invariably find their way into the mix, causing a re-negotiation (change control, the savior of many projects).

In addition, some customers don’t have clear lines of authority established. If you are a vendor in this scenario, expect to invest more time and effort! Other customers have an individual who clearly possesses the authority to act and decide, and this makes working with a customer much easier.

The final factor is the evidence of the skill sets and capabilities of the current project team relative to the skills and capabilities of prior projects. Are there vast differences? If so, which way are the scales tilted, and what impact should this have on your bid?

Ultimately, it’s a business decision to enter a contract or not. There may be yet other factors involved, such as the willingness to lose money on a smaller project to land (potentially) a bigger deal down the road or to gain experience in a new market. Or the business is going to the son-in-law of your CEO. It’s up to you to put in a responsible bid (estimate) and manage the risk.