Too many unit tests? It Always Depends

The correct answer to all non-trivial questions in software development (and a great many other things) is "It depends". Knowing this shows experience. But the correct question is really "What does it depend on?". Knowing the answer to that shows true wisdom.

Let's take a topic which has received some attention by august luminaries recently about the extent to which one should write unit tests. Your first reaction may be, "Really, people are still debating this?" Well yes they are.

Joel Spolsky and Jeff Atwood are debating this.

As much as Joel makes me chuckle and as much as he clearly has experience shipping quality software (full disclosure: the company I work for, Atlassian, happily competes with his flagship product), Joel has seemingly just levelled up or something and is going around with Jeff Atwood and challenging what I call "blind completeness" in unit testing practices.

Well I'm sure anyone on my team (JIRA) could come up with some plausible practical scenarios where additional unit tests do not repay the time spent on them. Unit tests are to catch regressions and to act as a jig keeping the code still as you maximise the design for maintainability. They also act as an index of working features (as opposed to fictional specifications). If the software won't ever be modified, then the time spent on unit testing will be waste. Just like all risk management work carried out where the maladies don't arise. And insurance policies on which a claim is never made.

It's possible that a turn of events will occur such that a given set of unit tests were unnecessary. It's not theoretical, it's practical. And it's wasted time, and for a business owner like Joel, that's wasted money. It completely makes sense that he's thinking about this. He needs to run his software business and he needs to use this fact about the possibility of wasted money to his economic advantage. Go Joel!

So what action should one take with this practical knowledge?

Here's what Jeff Atwood says:

The longer I think about this, the less I care about code hygiene issues (laughs). Because the unit of measure that really matters to me is, how quickly you can respond to real business needs with your code. And by that I mean, how well are you serving the people that are using your code. To me that's what it's all about.

Absolutely. Except over what time frame? Just this week? Don't unit test at all then.

Code quality, specifically effective unit test coverage is a leading indicator of the future ability of that code to respond to real business needs. That's why this apparent front-loading is so central to agile practices. It's the only kind of optimisation that cannot be premature if you care about the future cost of change. Optimising for maintainability is the single most critical practice for enabling change. It is not necessary to understand this about agile processes in order to get value from unit testing, however.

As an aside, watch what magnificent apoplexy happens to Kent Beck when his You Aren't Going To Need It (YAGNI) mnemonic is turned on his own advice about up-front testing.

So the question in the terms of my opening statement is:

"How much should I unit test?"

...and the answer will not surprise the attentive (or experienced):

"It depends".

And...

"What does it depend on?"

There are two factors:

1. How much does the test cost to create and maintain? Like all code, tests are relatively cheap to create. At least when you compare to the typical total cost of maintenance. For some reason the figure that stuck in my brain from reading some oft-cited book that I've long-forgotten (Peopleware?) was a ratio like 4:1 maintenance. If anyone can find me a citation I'd be grateful.
2. What is the cost of bugs in production? You could put this another way: what is the benefit of the test catching a bug before it gets into production. This cost is not a dollar value. It's a dollars over time curve.
   

The cost of bugs in production to a software shop like Joel's is the loss of customers or goodwill due to a customer's bad experiences with the software. And to make matters worse, you may not ever find out that the reason a customer didn't purchase or renew a license was their experience with bugs. There may be some potential for getting sued, but to my knowledge this doesn't happen in the development tools product world. Medical software? Perhaps. So there's at least one thing it depends on.

For completeness I'll mention some second order effects of bugs in production. Sometimes the bug never manifests but the assumptions evident in the code lead maintainers to draw incorrect conclusions about the system invariants. This can lead to bugs that do manifest. Don't make the mistake of thinking this is a theoretical problem.

The biggest problems with measuring the cost of bugs in production are that the cost is not immediate and often not simple to trace. There's a huge time delay. Bugs can sleep for years. And of course, if your customer base is growing then your potential loss due to the impact of bugs is also growing. If your customer base is tiny (as with fledgling start-ups and many internal IT department projects) then the cost of bugs in production is sometimes much smaller than for an established software product with millions of users. There's another thing it depends on.

Some software is sold on features, others on reputation or marketing. If you are a big name you can make money selling poor quality software at a high price because the purchasing decision-makers do not personally evaluate it nor are they fired for buying from a big name. Of course while business people can be perfectly happy shovelling this kind of shit, good developers rarely are. But there's another thing it depends on.

Further, the mechanisms for the impact of the cost of bugs in production often lean on non-development parts of the organisation like support and sales. It may be a cost coming from someone else's budget! Perhaps the project sponsor doesn't care about that budget! I would bet Joel does care because it's his company. From the point of view of the project sponsor, it depends on this too.

Clearly the economics of bugs is all about knowing your business, your customers business and the role of the software in it. And your market. And your pricing. And your customer's purchasing decision process. It depends on all this stuff.

You need to know the time frame over which you are measuring the cost-benefit trade-off of being at your sweet spot on the quality curve. This may also change over time. Sometimes it's critical to "get to market". Other times it's super critical. In my experience getting to market only ranges between these two.

All this stuff is just basic cost-benefit analysis and there is no magic formula that will work across all software projects. There's too much variation. Of course this doesn't stop pundits from continually declaring which size fits all.

So now in terms of Robert Martin's blog post, Quality Doesn't Matter That Much -- Jeff and Joel. My response is. It bloody depends. Software developers need to be able to speak to their project sponsors in terms such that they can negotiate the optimal quality balance for the circumstances.

Finally, if you are a software developer working on a project for whom the cost-benefit trade-off that has been decided by your project sponsor leaves you with a quality level that robs you of job satisfaction, then drop me an email.