The Challenge of Metrics

August 1, 2009

Now that I’ve articulated my model of software development (the DNA of Software Development), I feel that it is important to discuss the subject of metrics, examining what is essential to measure in order to understand how well any software development organization is performing.

There are a number of challenges when it comes to metrics and software development. It is all too easy to define a large number of metrics to “get the full picture,” which in turn drives a lot of administrative overhead, increasing costs and adding time and frustration to those involved. Another danger is to use metrics inappropriately, such as gauging a programmer’s job performance on a single metric such as the number of lines of code “produced.”

The holy grail is to measure the output of teams and individuals using completly objective and quantifiable criteria. Unfortunately, a generic software widget that can be readily defined and used for comparison doesn’t exist. The type of project, the relative priority, and the people involved introduce a high degree of variability.

As a manager, I strive to maintain consistent expectations and evaluate job performance using the same yardstick for everyone, but the problem of objectively and quantifiably comparing software “output” between different people or project teams remains elusive.

There are metrics that are common in software, such as lines of code (LOC) and Function Points. These are good for gauging the relative size of a project, or examining how well you and your team compare against others in the industry, such as comparing your defect rate per 1K LOC against the industry norm. What they aren’t good for is measuring – and helping – everyone to understand how well teams are doing right now.

To drive the point home about the variability that people alone introduces, does this phrase sound familiar? “I put my best people on my most difficult problems.” When those challenging projects or problems come around, when you need absolutely need to succeed, and need to do so under tight deadlines, who do you select?

Given a choice, would you staff a critical project with junior programmers and hope that they pull a rabbit out of the hat, or, given the option, would you assign at least a couple of senior programmers along with a few junior programmers to the project – the senior programmers who have demonstrated that they are motivated, talented, and capable of delivering results? Do you even need to consider this for more than a couple of seconds?

Expectations – and pay – are very different for the senior and junior programmers as well, aren’t they? A typical software organization is comprised of different people who are evaluated and paid at different levels. Not only that, there will be differences in strengths, weaknesses, individual preferences, communication styles, and experiences that everyone brings to the table.

Variability is also introduced with the type of project assigned to one project team in contrast to another. For example, adding well-defined features to an existing product is far simpler than a project where some exploration is being undertaken to create a new product to meet what is essentially a broader, looser, unmet demand in the marketplace. Can there be accurate, objective comparisons between these very different projects in terms of output?

The challenge of defining essential metrics is that I want them to be as simple as possible, requiring little administrative overhead, be tied to the critical aspects of producing software, and accurately reflecting the state of progress and readiness of the software being developed.

What are these metrics? Next post, I’ll delve into the topic!