Succeeding with QA Automation

October 30, 2009

As I mentioned in my last post, a change in the date and venue for our annual user conference was a curve ball in terms of a presentation that I was slated to be a co-presenter for: QA Automation. Instead of co-presenting, I was the sole presenter, tasked with creating and delivering the presentation! I’ll cover the highlights of my presentation, with one caveat. I structured this to talk about some general observations about QA Automation and then conducted an open discussion. I wanted to test my observations and learn from others.

I opened with the following questions:
  • What should be automated?

  • Can manual testing be eliminated by implementing automated testing?

  • Can QA Automation improve productivity?
What should be automated?
The following diagram depicts the areas and focus of testing that can and should be automated.


Can manual testing be eliminated?
We are an Agile/Scrum development shop, and our experience has proven that automating within a sprint is not advantageous. From an efficiency and effectiveness standpoint, features need to be complete before they are automated. Manual testing can and should focus on validating the new features and developing good test plans that can be automated later.

Can QA Automation improve productivity?
There are a few objectives that QA Automation can achieve.

Reducing the “QA bottleneck” that is invariably present at the end of software projects, and also where the squeeze to make deadlines frequently occurs.

My other observation is that automated tests take less effort to run than manual tests, and as a result they are likely to be run often. This alone can increase your confidence in the software.

When I look at QA automation, my philosophy matches my opinion of manual QA, and is that you don’t want to be in the business of “testing your way to quality.” If you feel that you have a large number of defects that are occurring in the system test phase, look for other areas to improve in your software development process. Perhaps you need to improve your requirements definition, or improve on design and code reviews – preventing defects from finding their way into the code in the first place.

QA and Development Role in QA Automation
We have experimented with using traditional QA and Software Developers paired together. This has proven successful, and I think that pairing leverages strengths that both professions bring to the table. The first area that I explored was the subject of unit tests. There are those who state that the developer is the best qualified resource to write unit tests, because the developer knows the code. I don't share this opinion!

Yes, the developer knows the code, and therein is the danger. Most developers that I know aren’t good at testing. One of the reasons for this is that if you know the code, you are likely to be thinking in terms of the “happy path,” – how the code should function – and not in terms of the fringe cases that a good QA tester will consider. Pairing can produce better tests.

I asked about experiences that those in attendance had with Test-Driven Development (TDD), where you start with unit tests that fail and then write code to make the test pass, but no one in the audience had any real-world experience to share.

My take is that TDD is useful, particularly since the book The Art of Agile Development noted that this forces developers to think in terms of interfaces first and implementation details second, providing a design that is more comprehensible. I would advocate the QA could help strengthen the tests here as well, for the same reasons noted above.

Conversely, we’ve found that Development can help QA. They can help QA with things like code organization and source control. Developers naturally have a lot of experience with the design and development of common functions, and can bring practices like design and code reviews to the QA Automation table.

A group discussion yielded some more perspective on this, the main takeaway being the realization that it was possible to get into QA Automation without considering code design and management. And just like any software, you could end up with spaghetti-code that is difficult or impossible to maintain, neutralizing any gains from your investment in QA Automation.

Conclusion
The bottom line for me: QA Automation is an investment, and you need to consider the cost of the tools, training and mentoring, script (or code) organization and management. Target those areas where there will be a need and a return for the automation dollar. Ultimately, you need to answer the following questions related to QA in order to determine if your software is truly “done”:
  • How much of the product is being tested through regularly-executed, automated testing?

  • Does manual testing adequately cover the remaining bases, or are there gaps?

  • Is your failure rate is known and defensible?

Change and Our Annual User Conference

October 23, 2009

At the end of August, I attended a user conference as a vendor. This was nothing unusual, since we are always the featured vendor at an annual conference in the fall that was essentially built around our company and product.

This year, however, we made some changes. Change was due, and it wasn’t a question of what, but more of a question of when. For more than one reason, our conference attendance has been declining, and low attendance doesn’t make you feel great as a vendor. There is simply less energy and excitement to get your blood flowing.

For us, conference attendance has been dwindling for GOOD reasons. We develop enterprise-level software for the insurance industry, and we have made tremendous strides in improving the quality of our releases in the last several years. We also do a much better job of addressing critical issues and adding enhancements on behalf of our customer base.

Needless to say, one unfortunate reason that we had higher attendance in years past was because our customer base wanted to beat us up! They felt that they needed to corner us and collectively press us to address our quality issues and their needs. As we’ve improved our delivery and paid more attention to servicing our customers, they have had less to complain about, which in turn reduces the urgency that customers feel about attending a conference.

Over the years we’ve also increased our on-site customer visits and leveraged the Web, conducting Webinars, posting information to a new Community Site that we developed, and generally have been more open and communicative. Our customers understand what we’re working on, what the priorities are, and what our challenges are because we communicate better. They also appreciate our challenges and the work that we do as a result. And again, this gives them less reason to attend a conference.

They also understand and appreciate that we have a certain capacity to continually add new capabilities to our product line. We’re not a large organization, and these days we are sized appropriately in terms of the revenue that we generate. This does place constraints on us, but our customers agree with our overall vision and direction; the big question now is: How fast can we deliver on our vision?

In general, our customers enjoy the time and dialog with us at the annual conference. But it has become very tough for us to fill up an entire conference with compelling new content on an annual basis. At the beginning of the year when we began planning for the conference, we understood that the economy was down, and we expected that our conference attendance to be down as a result both of these factors.

Fortunately, we had another option! We used to be an independent operation, but now that we are a part of a larger company, and it was advantageous all the way around to combine into one conference that represented a larger suite of sister products under one one corporate umbrella.

Of course, this change had some impacts on us. The date for the combined conference was ahead of our usual conference date, so many of us scrambled to prepare presentations and demonstrations. Not that we didn’t scramble in years past – after all, does anyone ever have as much time to prepare as they would like? I found myself scrambling in August, the month when I take a week off to enjoy the short summer that we get in Maine.

My role at these conferences is always as a speaker on one or more topics, and I typically am involved in putting together rolling PowerPoint presentations for our booth. I’ve also done my share of booth-duty in years past, demonstrating products and answering questions.

This year, the change in location and the moving up of the date changed our original planning. I was slated to co-present on QA Automation. As a development manager, I have been involved with QA Automation efforts at our office this past year, in part because I allocated a developer to assist in this effort.

Changing of the timing and location of the conference left me in to my own devices – I was suddenly the only presenter! Fortunately, I have given my share of presentations over the years, so this did not produce any anxiety. I did need to spend some time thinking about just what the heck I was going to present, however…

Next post, I’ll cover my thoughts about QA Automation, using material that I pulled together to present at the conference.

Are You Committing Your (Business) Tanks?

October 16, 2009

This post is off-topic from my usual software development-oriented posts, but given the state of the economy and that scaling back is the order of the day, I thought I would branch out and discuss tactics for managing/leading through tough times.

“I’m committing my tanks!”

This quote is from the World War II movie The Battle of the Bulge, about the siege of Bastogne. As American field commander General Grey makes this statement, I noticed that he did so in an animated, excited, and delighted manner. Why would a general, who is committing to a battle where people in his command will die, be delighted?

Before going on, let me acknowledge that this film was NOT critically acclaimed by movie war buffs. I do happen to enjoy war movies myself, and watched this for the first time in years because I picked up the movie on the cheap. I’ll confess that I did enjoy watching the movie more as a kid than I did as an adult, but I digress.

General Grey wasn’t delighted about committing his tanks because he was being a blood-thirsty jerk. It was because the time for analysis and speculation was done.

For the record, I do not have any military experience, but based on the military history that I have read, generals – like business leaders – do not always have complete information. They have to rely on the information that they have at hand, the opinions of their subordinates, and their own experience and judgment.

And most importantly, they need to make a timely call. To delay and wait for full information can cost them a battle. Indecision – and the time wasted as a result – can cost armies the value of surprise and position along with providing the enemy time to strengthen its reserves and better position itself while they are scouting you for weaknesses.

A lack of commitment looses battles. If you are going to attack, attack to win. This is why General Grey was delighted. He recognized that he had all the information that he was going to obtain within an acceptable time frame to engage the enemy. Furthermore, he was done deciding what action needed to be taken. He had committed himself and his army in a decisive direction – now he could focus on making that direction a success.

From a military perspective, battle is chaotic, typically lacking the clarity that would make most of us feel both comfortable and assured. The business world is no different. You need look over the landscape, collect what information that you can, and then provide the troops with focus and commitment. And I’m not the only one who thinks this way.

In his book: Andy Grove The Life and Times of an American Business Icon, Richard Tedlow quotes Andy Grove about the tendency for people to hedge their bets when faced with uncertainty. Andy Grove doesn’t pull any punches: “Hedging is expensive and dilutes commitment.” Grove states that you need “exquisite focus.” The observation being that without commitment, “you will always be looking for a way out rather than a way to win.”

What is your business doing in this uncertain economy?

Now is the time to be investing and positioning for the future. Downturns in the economy cause people to react cautiously, cutting expenses and reducing investments to improve today’s bottom line. This can cause long-term loss of market share and strategic position, creating an obstacle that is more difficult to overcome later. Here’s a quick, 5-point list on ways to take advantage of a down economy to strengthen your business.

Keep the R&D Dollars Flowing
Don’t sacrifice tomorrow for today! Assess the return on investment on everything that you are doing, and keep your business focused and committed on those activities that drive your economic engine and competitive edge. Cutting “investments” to save a few dollars today is really nothing more than hedging your bets – and slitting your own throat.

Don’t Skimp on Sales and Marketing
Because companies are looking to save on every expense possible, you can grab business from your competition by offering a lower price. Doors that have been previously shut because companies “already had a solution from another vendor” may open if you can provide goods or services at a lower cost. Keep your sales force active and knock on those doors!

Likewise, advertising budgets get slashed by many companies in bad economic times. You can use your competition’s cuts to your advantage – by keeping your business visible while they are not.

Now is the Time to Hire
Layoffs have created a large pool of great candidates. This is something that will not continue indefinitely for more than one reason. First, the economy WILL turn around, and jobs will be created. Secondly, and in the United States in particular, the baby boom generation is getting older and looking towards retirement. There are simply less good people that will be a part of the labor pool in the years to come.

According to research by McKinsey, "finding talented people (is) likely to be the single most important managerial preoccupation for the rest of the decade.” The problem is that "too many firms still dismiss talent management as a short-term, tactical problem rather than an integral part of a long-term business strategy."

Provide Great Customer Service
Give customers a reason to stay with you – and recommend you to their friends and acquaintances. Treat your customers like gold, and provide them with exceptional service. This will pay dividends in today’s connected world; consider all of the Facebook and Twitter word-of-mouth that can spread in the form of guerrilla marketing if customers have a great experience with you – including recommending you if their friends have negative experiences elsewhere.

Look for Acquisitions
Now is a great time to shop for acquisitions that can strengthen your business. Acquisitions always present risk, but there are great opportunities at bargain prices today relative to a few years ago, and this represents opportunity. The right acquisition could substantially improve your competitive dynamic.

Some related links if you are interested in more:

Starting Up in a Down Economy

Taking Advantage of a Down Economy With Investment Hiring

5 Strategies For Business Prosperity in a Down Economy

M&A: A Smart Strategy in a Down Economy

Can Software Projects be Predictable?

October 9, 2009

I’ll answer with a qualified “yes.” But should you really desire something else?

Predictability can be achieved, but not without cost. And the larger the project, the higher the price. (And the price is not linear, as noted later in this post.)

Large-scale projects have more requirements – and thus more product features. More features mean more components, more interfaces between the components, more behavior expected of the software, more testing, and more sleepless nights than smaller projects. After all, getting a large number of features correct, and operating correctly with one another, is a challenge in and of itself.

When talking about software project predictability, the most common definition of success is really about predictability, the delivery of projects:
  • On Time
  • On Budget
  • Of High Quality
  • With the Expected Features
Am I talking about a waterfall project? Let's say that I am.

Given all of these constraints (including the choice of a waterfall methodology), predictability becomes a function of:
  • Clear, unambiguous, well-understood requirements that will not change during the course of the project.
  • Estimation and work performed by experienced, competent professionals who can and will work together as a team.
  • Project scheduling performed by experienced, competent project managers.
Note that I’ve made the target that the project team needs to hit crystal clear right out of the gate. In addition, the software requirements need to be clear; they must be expressed clearly to the satisfaction of the project team.
This alone requires a skilled job performed by those responsible for crafting the requirements, plus an investment in time to ensure that the project team understands the requirements.

I’ve also added the rule that the requirement cannot change during the course of the project. Getting the requirements right, ensuring that they are understood by the team, and managing change is a big part of meeting the predictability/success criteria. This requires an up-front investment of time that before a project actually begins; and the more requirements to be addressed by a project, the greater the investment in time required.

It shouldn’t be too difficult to see that I’ve also stacked the deck with experienced, competent people – the “secret sauce” to any successful project. Good, experienced, competent professionals will not take undue risks, and they can be counted on to work a project to completion, hitting their dates with quality. I’ve also stipulated that they can and will work together as a team, something that is easier said than done.

What else drives unpredictability?

One of the problems with predictability in software projects is that they are the most unpredictable in the early stages of a project. This phenomenon is reported by Steve McConnell in his book Software Estimation: Demystifying the Black Art. It is known as The Cone of Uncertainty.

The cone is largest the earlier in the project life cycle you are. The cone narrows from uncertainty to certainty as different stages are reached. Steve McConnell points out that a significant narrowing of the cone occurs during the first 20-30% of the total calendar time for a project.

What happens when someone asks you to take extra time to work on your estimate, so that it contains less uncertainty? Nothing that makes the estimates any better than the already are. Improving the estimates requires working out the variability that drives the uncertainty, and this means performing the actual work of the project.

What else contributes to software project uncertainty?

In a word: People.

Let’s face it; everyone has their strengths, weaknesses, preferences, and idiosyncrasies. Some mixes of people work better than others, and no matter what it takes time for people to understand each other and balance out each other’s strengths and weaknesses so that they can work effectively as a team.

In the real world, you will likely have a mix of seasoned and less experienced personnel on a project. It simply isn’t economical or feasible to staff each and every project with people highly experienced in the position required for a project. People need to learn and grow, and they will need new experiences to help them grow.

Giving people new experiences and challenges is a great investment; your organization gains bench strength and providing new opportunities and challenges keeps people happy and motivated. This will sacrifice some predictability, a risk that can be mitigated through mentoring and additional oversight.

If you attempting to predict one project's completion based upon another project, keep in mind that studies have proven that software projects are not linear in nature. Referring back to Software Estimation: Demystifying the Black Art, Steve McConnell points out that if a 10,000 lines of code system required 13.5 staff months, a 100,000 lines of code system should require 135 staff months, but in reality it requires 170 staff months.

The amount of people involved with large-scale projects introduces extra overhead, part of that being extra lines of communication. And when there are more people and more lines of communication, the greater the opportunity for miscommunication to occur.

Is predictability all that you care about? Consider this: Time will likely change your perspective, especially in today’s fast-paced business climate. What you think is a priority today many not be one tomorrow.

Instead of being predictable – like predicting what you need software to do one year from now – another option is to be fast and adaptable. Be prepared to embrace changing conditions and circumstances by building smaller software projects that focus on the high-value, high-return features first, deferring on the rest.

Smaller projects enable a faster return on investment; the quicker that the software is in use, the sooner that a return is provided. Decisions to continue to enhance the software to provide even greater return can then be made. In a worst-case scenario, a smaller project enables you to fail fast, and with less expense incurred.

Improving Results Through Better Collaboration

October 2, 2009

This is the final post covering how one of our software teams made some modest changes and went from struggling to high-performing, all within a short period of time.

Understanding the requirements was the first important improvement. The next two improvements related to estimating and collaboration – beyond the improvement already achieved between the product owner and the team.

Estimating
One clear reason that this team struggled with meeting its dates was because they were underestimating. A good understanding of the features (backlog) and streamlining some of the tasks helped, but they needed to make a couple of other adjustments.

The first was that there was no team estimation process. For example, individual developers would go off and determine estimates for the items that they planned on – or thought that they would be – developing. They also were estimating work in fairly significant (for Scrum) chunks of time. I asked that the team use tasks that were no greater than 2 days in size, and that they collaborate and discuss their estimates.

I’m a big fan of team estimation, and the experience with this team supports this concept. Once the developers began discussing their tasks collectively and exploring how they were going to approach each user story, they gained a much greater understanding of what was truly required. The combination of smaller task sizes and greater collaboration increased their confidence that they could meet their objectives.

Collaboration
As the team saw the benefits in collaboration as part of their planning and estimation process, they were receptive to moving towards greater team collaboration across the board. While the team was using Scrum development, it really looked like a mini-waterfall project where Quality Assurance would develop test plans and would wait for Development to be complete before any testing would begin.

Over a period of a few weeks or so, the team moved towards collaborating across the board. For example, each time a developer finished even the smallest piece of functionality, he reviewed it immediately with QA. And if the team encountered one of those inevitable “we didn’t consider this” scenarios, the team huddled together to discuss what that meant to the user story, to their acceptance criteria, and their commitments.

Did you catch the subtle change? The team took ownership. It wasn’t QA’s acceptance criteria, it was the team’s acceptance criteria. The team also started collaborating this way on their own; they had seen the benefits of collaboration and embraced it.

The Result
This team moved from merely performing individual tasks to being an engaged, motivated, collaborative team. They started to own all aspects of their work – as a team. As they moved from being a collection of individuals to a true team, you could almost feel the spirit and the flow. The output of the team visibly increased, overtime ceased, and everyone looked much happier about their work. Being successful will do that.

Looking back, I got a little lucky. One of the factors in this team’s rapid success was the willingness of the team to try new things. They were competent professionals, but they were unable to implement the key concepts of Scrum development – despite the fact that they intellectually grasped what “it” was.

Sometimes people get set in their ways or lack the ability to see how to apply concepts in their day-to-day work. Changing methodologies can necessitate changes in expectations, and these new expectations must be clearly communicated. Other times, people are simply skeptical.

In our case, it was a combination of factors that included some skeptics, but they put aside their skepticism and tried something new. And they ended up loving it. It was a pleasure to be a small part of this, and definitely one of those "Yes!" moments to see them succeed.