The Theory of Everything for Software Development

July 25, 2009

Jurgen Appelo, Chief Information Officer at ISM eCompany and the author of the blog NOOP.NL wants a Theory of Everything for Software Development. When I read about that desire, it got me thinking. Does a Theory of Everything for Software Development exist? If so, what would it look like? This is my attempt at providing a theory.

One of my goals is to distill the complex down to its simplest form – without losing meaning or applicability. The challenge in creating one model that describes the entirety of software development is that it must allow for the multitude of software development processes that are already in existence. And it must stand the test of time, allowing for new approaches that haven't been invented yet to fit equally well.

To me, this means that the model must be general in nature; it should appear to be simple, yet capture the complexity and realities of software development. This took a bit of thinking on my part, but who said that this was going to be easy?

My model was developed from the observation that there are really two constants involved with software development. First, there are some fundamental steps involved with software development. Second, there are people involved in the software development. Nothing new under the sun here. I categorize both as:
  • Software Development Activities
  • Software Development Personnel
Software Development Activities
Software development activities are the steps that are involved with writing any software, starting with the inception of a project to release, whereby the product can either be enhanced with new features or maintained by fixing defects. The basic activities of software development, regardless of the software development methodology being used are as follows:

Before any code is written, there should be definition about what the software is expected to do. This information is utilized in the initial design, both of which informs and guides the actual construction and testing of the software, resulting in working code.

Of course, there is always planning and estimating that can occur at various stages. This includes any re-planning and re-estimating because the situation is changing or reality is setting in, and everyone is beginning to realize that the DATE that was thought to be so achievable is suddenly not so achievable.

Software also lives beyond any project – unless the project is killed. In fact, software (particularly large-scale systems) will spend more time in maintenance than in actual development. My position is that a system test at the end of a project has effectively placed the software in maintenance mode.

Note that there are no arrows or implied flow between the activities. Think of them as cards on a table that can be moved around as needed. For example, a waterfall approach to development will strive to have design complete before any coding, and the process will flow from left-to-right as shown. On the other hand, the use of Test-Driven Development (TDD) will require that a test be written before any actual coding is performed.

Obviously, all of these software development activities require that someone is performing those activities...

Software Development Personnel
Software isn’t about satisfying the needs of computers, it’s about meeting the needs of the customer – the user(s) of the software. Businesses, which oddly enough are comprised of people, provide the initial input into the process by articulating their needs. Some type of development process kicks into gear that captures those needs and produces working software, all through the talents and efforts of people.

Depending upon the size of the project, there could be as few as one to as many as thousands involved in the actual production of the software. The people side of software development can be characterized as primarily dealing with the attributes and interactions associated with the individuals, teams, and the organization as a whole; this includes things like business knowledge, programming skills, critical thinking skills, communication skills, teamwork, and motivation.

The Model
While software development activities and personnel are separate entities, they do not exist in isolation. In fact, they are inexorably linked and dependent upon one another for achieving maximum results. This leads me to my model…

Yes, using a DNA analogy to describe the “theory of everything for software development” works perfectly! (Well, for me it does.)

DNA is often referred to as a recipe or code, something that carries the genetic information used to construct other things. Similarly, the strands of software development activities and software development personnel are intertwined and bonded together; neither can exist without the other, and output (working software) cannot be produced without this fundamental arrangement.

The inputs can be anything that informs the project team, from the business vision, priorities of the feature set, constraints such as the budget, information about the competition, and influencing factors such as the tone and environment that the project team works in.

Closing Thoughts
There are limits to my model. While any software development methodology can be used, the model doesn't explain which one to use, and under what circumstances one methodology should be chosen over another. The model is really describing the core, genetic software development material that can take many forms in actual practice.

When it comes to selecting a methodology, my feeling is that you need to make that call based on the goals and objectives of your project. There are times when the extra expense and overhead of finely-detailed requirements, design, and considerable testing are worth the effort. Being off a dollar or two on an insurance quote is one thing, but being off a degree or two with a nuclear missile is another. "Good enough" isn't always good enough.

The intent of my model is to strip away the methodology from the essence of software development. A methodology is a means to an end, one way of creating software, and each has something of use to offer. What is important is to understand the true nature of software development and the types of problems that a given methodology is attempting to address through its unique approach.

I also feel that it is important to recognize that methodologies alone aren't the only consideration when dealing with software development. The people that are involved with software development are a vital component and a major variable.

Different people have different strengths, weaknesses, and preferences. Everyone has different communications styles and approaches to their work. And when people are combined in a team, the interaction dynamics that occur can amplify strengths and magnify weaknesses, creating unique situations that can increase productivity or plant the seeds for outright failure.

The DNA of Software Development model and this blog are about confronting these realities. I would love any feedback that you may have!

The Quest for the Theory of Everything

July 23, 2009

My day-to-day life in software management involves interacting with technical and non-technical types, which places certain demands on me. Part of my challenge is to explain software development to non-technical people in the simplest way possible.

I have found that this is both a challenge and a useful exercise. Distilling a complex activity into its simplest form so that the explanation has meaning and relevance to everyone concerned is not particularly easy! But it does have advantages. It forces you to think about what is truly important, to examine what those essential truths that are core to productive software development, regardless of the software development life cycle model that is being used.

In doing so, I find that I am able to evaluate programmers and coach them effectively. The same applies for development teams as well; you are able to look for those critical interaction points to help teams get into the “flow.” It also helps you to focus on the metrics are the most important – not to use as a weapon at review time, but to coach and guide people and teams in improving.

Since I’ve been writing the blog for several months now, I thought this would be a great time to reflect on my thinking. And come up with something to write about.

Interestingly enough, I came across an informative and entertaining software development writer in my Internet travels: Jurgen Appelo, Chief Information Officer at ISM eCompany who is the author of the blog NOOP.NL. I gravitated towards this blog because I enjoyed Jurgen’s writing style and because he is a fellow manager writing about software development, something that not particularly common in cyberspace.

Jurgen posted a blog entry in 2008 that states his objective: his quest for the Theory of Everything for Software Development. (His post inspired me to shamelessly steal – I mean incorporate – his title for this blog post.)

In his post, Jurgen states that he doesn’t care about any specific methodology. He cares about how they differ from one another, and that he wants to broaden his mind to understand how all solutions fit together in one big model.

I couldn’t agree more! Agile, RUP, Waterfall, Spiral – they’re a specific means to an end. They each define a unique approach towards achieving the end result of building working software that delights the customer. The question of the day is this: Is there a theory of everything for software development?

I think that I’ve got one. I’ll share it in a few days once I’m satisfied with how I’m explaining it.

One Key to Successful Collaboration: A Unifying Goal

July 17, 2009

In my last post Accountability and Teamwork: Is Everyone Pulling Their Weight? I noted that I recently read the book Collaboration: How Leaders Avoid the Traps, Create Unity, and Reap Big Results by Morten T. Hansen. This book is a great read, and not particularly long – I’ve certainly read much longer! But it is packed with great advice and insight on the subject of collaboration.

At my company we’ve been faced with driving performance from multiple teams across our organization. If you’ve ever been faced with this scenario, I’m sure that you have at least had the thought run through your head, “Let’s have teams compete against each other.”

I’ll confess to this thought. I’ve played a fair amount of sports in my day, and I always relished a challenge. Growing up, I would compete with a good friend of mine on the most trivial things. Sometimes we competed against ourselves, seeing if we could break our old record of passing a football back and forth (across his parent’s living room) without dropping it – or breaking anything.

While competition is a good thing, Morten’s book points out that you want to target competition outside of the company, not inside. Pitting people against each other from within your company will foster unhealthy behavior that will divert focus from where it should be, and that is collaborating with each other to beat the other companies that are your true competition.

A major component of unifying people, according to Morten – and with no disagreement from me – is to create a unifying goal. The great example cited in the book is one that is often used in management literature when talking about creating an exciting, motivating mission that people can be passionate about: The goal of landing a man on the moon.

In the book, Morten reports about an exchange between NASA head James Webb and President Kennedy. Webb had a goal of establishing preeminence in space. President Kennedy, however, wanted a very clear goal. Webb argued that “there is a wide public sentiment coming along in this country for preeminence in space.”

Kennedy’s response: “If you’re trying to prove preeminence, [landing a man on the moon] is the way to prove preeminence…”

Kennedy was right, his goal was clear and specific. It not only proved preeminence, and it guided other decisions that a vague goal like “establishing preeminence” would not. In fact, a goal like “establishing preeminence” would require further debate and scrutiny about projects because it lacked a concrete objective. “Landing a man on the moon” was specific enough that anything not related to that objective was clearly lower priority.

The “landing a man on the moon” goal created clarity, automatically prioritized many decisions, and unified everyone involved. It’s got me thinking more about the goals our organization needs, and I’ll make sure that I’ll never, ever, even entertain the thought of having internal teams compete against each other again!

If you haven’t read this book yet, I highly recommend it.

Collaboration: How Leaders Avoid the Traps, Create Unity, and Reap Big Results by Morten T. Hansen.

Accountability and Teamwork: Is Everyone Pulling Their Weight?

July 10, 2009

When I was in my late twenties, I went with a group of friends on an overnight camping/white-water rafting trip down the Penobscot River located in here in Maine. When we reached the water we found that we had too many for one boat; a couple of us needed to ride in a boat with another, smaller group.

I was good with this, since I had been down the river a few times already. I also thought that it would be great to watch the others who hadn’t been down the river yet – particularly since the Penobscot River boasts class III-V rapids and a 12-foot waterfall. A mischievous part of me wanted to see how they handled things – from a safe distance.

As luck would have it, our boat crew included some first-timers as well: a couple of dads with their 15-year old daughters. Needless to say, I had some reservations about the 15-year olds and how much they would pitch in. But I figured the dads would make up for any shortcomings.

I was wrong! The 15-year olds were fine. Well, except for that time when one of them jumped back unexpectedly as we were navigating some difficult rapids and hit the guide square in the nose with her paddle, knocking him out of the raft. I was in the front and didn’t notice that he was gone until we were through the rapids, and I looked back and saw an empty spot where he should have been.

Never fear, the guide came swimming up to the raft and hopped back in. He bled for a minute, but was otherwise unhurt and in surprisingly good spirits.

Things that make you pause on a rafting trip:
When you hear another guide shout, “Good save!”
to your guide after making it through a difficult
section of the river.

Towards noon, our boat fell significantly behind the other boats being guided down the river during a calm, flat stretch of water as we headed towards our island lunch. I was still in the front left of the boat, opposite from my friend and content to dig in to make sure that I was pulling my weight.

At one point my friend gestured to me just as our guide was pointing out the large distance between us and the other boats, exhorting us to “get our butts in gear.” (I forget his exact words, but you get the idea.) I then noticed that my friend looked a little miffed. He gave me a subtle “ease up” sign with his right hand so that the others wouldn’t notice.

I then took stock of our situation. I realized that the 15-year old girls weren’t going to do a whole lot – but that was in line with my expectations. The dads, however, weren’t exactly compensating for their daughters, either. In fact, while they were sticking their paddles in the water, their “paddling” was more show than substance. They weren’t contributing towards moving the boat forward at all!

My friend was staging a short of silent strike, as he realized that we were working our butts off and giving these people a free ride. Needless to say, as we eased up, it became obvious that everyone needed to put in a little more individual effort if we were ever going to get to lunch.

As I mentioned, this story took place while I was in my twenties. I’m now in my forties (my how time flies!), so this was a good twenty years ago. What made me think of this now? Well, it’s been raining a lot in Maine so far this summer, and I’ve been getting some extra reading in. The book that brought this little memory back is called Collaboration: How Leaders Avoid the Traps, Create Unity, and Reap Big Results by Morten T. Hansen.

Towards the end of the book, Morten states, “People working in teams can shirk and get by because individual output is not being measured, only team output.” Morten notes that there is a vital problem in collective, collaborative work: when people can hide they often do. It’s called social loafing.

Morten advocates cultivating what he calls T-shaped management: people who simultaneously can deliver results on their own job and deliver results through collaboration. Not an easy task! This requires great prioritization and time-management skills.

As a manager of software developers spread across several empowered, self-directed teams, I am paying attention to the output of the teams as well as the level of participation and contributions being made by the individuals on the teams. I've already discovered some things the hard way on this front.

For a start, everyone isn’t cut out to understand how to collaborate and participate effectively on teams, not without some coaching and guidance. Some people take to it like a duck to water. Others prefer the comforts of having others assign tasks to them; they have a difficult time proactively taking hold of work without the nod from the “boss,” no matter how often they are told that they are part of a self-directed team.

Some people have low self-confidence and need coaching. There can be personality conflicts that cause people to disengage from teams, such as one over-powering personality or extreme mismatches in terms of personality traits (introverted versus extroverted) and approaches to work. And I’m sure that others fall into the social loafing category.

What is your experience? How do you handle accountability with teams and individuals?

Expedite Right

July 3, 2009

“This is now our number-one priority, and this issue is causing me some real pain. Can you expedite this?”

Does that question sound familiar? If you’ve worked in the software world even for a short period of time, you’ll be confronted with an appeal like this to resolve a defect, overcome a performance bottleneck, or enhance existing software in some way. And while my general rule is to keep people focused on agreed-upon priorities, there are still times when priorities need to be adjusted.

A development lead who worked for me was faced with the need for a customized report as part of an ongoing project with a customer. Of course, this was elevated to a critical status a few days before a schedule delivery to the customer, and that delivery did not include this report. In reality, this was not a new report. The customer was requesting that we change an existing report, modifying the output from a PDF format to an Excel spreadsheet.

Everyone in our organization empathized with the customer’s needs, since it helped them balance information between two disparate systems that was critical to the customer’s business. Of course, there were some in our own organization who felt it was necessary to “lobby” on behalf of the customer, and did so by talking directly to the lead developer on the project team.

As you might expect, the project team was doing all that they could to meet a delivery date that was only a few days out, and they did not have any time on their hands to add this request. This lead developer didn’t exactly have free time on his hands, either.

However, because the team was already stretched, the lead developer opted to take this task on himself. Developers are people who want to do a good job, and they want to be helpful. This particular individual happens to be a highly motivated person who didn’t want to let anyone down.

He quickly determined that the tool that we were using to generate our PDF reports had the capability of exporting data to Excel, with relatively minor changes required. He added this capability over the weekend so that other people in our organization – those closely tied to the customer – could look over the report, which was ready quite literally the day of delivery.

Needless to say, I started hearing about a major problem with this report as soon as others began looking at it. The concern was phrased along the lines of, “How could we possibly think that this was acceptable?” Naturally, all of this activity was under my radar until the crisis hit.

Since I had people in my office complaining about something that I previously knew nothing about, I promised to look into it right away. (Yes, people were satisfied that I was going to check into the problem, even if I wasn't immediately chiming in on the "How could we?" bandwagon.) I asked the lead developer about his take on the situation and to let me see some sample output so that I could understand the complaints.

When I looked at the Excel output, I noticed one thing immediately. The Excel output looked exactly like our PDF output, right down to the first page that printed off information such as the report title, criteria used to generate the report, and the report date. I also noticed that the actual report placed the data on the Excel spreadsheet like it would a PDF report, complete with all of the white space.

My initial reaction was, “If I requested Excel output, I would expect the data to be organized a little differently.” Not that what I was looking at seemed like a big deal.

The major complaints were as follows:

  • One of the columns wasn’t wide enough to see the data. You had to manually widen the column.

  • A heading was in one column, and the actual data was in a column right next to it, instead of directly underneath the heading. There was no heading where the data actually was found, it was just placed differently than it could have been.

  • Some data (like longer names) ended up in two rows instead of one.
Okay, the Excel spreadsheet met the literal request of the customer, but lacked polish. I get it. A short discussion with all involved determined that polish would be a good thing, and that since this report could easily be delivered at any time, we opted to proceed with our delivery and then send the polished report a few days later.

A short while after this transpired a couple of us were talking with the lead developer, and he felt a bit down. He had tried to keep the burden off of the team by taking on a task during his own time over a weekend, and basically got trashed for it. I supported him, but pointed out one thing that I’ve learned the hard way myself.

Expediting does not mean cutting corners. Work the process, but don’t circumvent the process – whatever that process happens to be. In our case, we knew enough to define what success looks like for a product feature, but we failed to do so in this case. We should have worked with others to be clear about how data in the spreadsheet was organized. It wouldn’t have taken much time.

A dialog about the situation would have revealed another important fact that became obvious after the lead developer spent a weekend on this. We could have (and most likely would have) determined up front that this particular piece of work didn’t have to be included with the scheduled delivery, especially since it was a late-breaking request.

Out of curiosity, I looked up some definitions of expedite:
  • Speed up the progress of; facilitate; "This should expedite the process." (Italics mine.)
  • Process fast and efficiently; "I will try to expedite the matter."
  • To artificially accelerate an order ahead of its regularly scheduled counterparts.