06 April 2011

Automated Testing in Games

So I clearly upset a few game developers with my last blog post. Sorry about that! I do find it a bit surprising how strong some of the emotions were but I probably deserved the beating I got on reddit.

Second my post was too long and based on my stats, it looks like less that 1.0% of visitors read the article. Unless they can read a couple of thousand words in 20 seconds. Again, that was my fault.

But what was interesting was the responses to my article were pretty clearly divided into two main groups (disagreeing and flaming) and one small group (agreeing):

  1. You are on crack, we do this stuff (automated testing, etc) better than anyone.
  2. Game developers are moar awesome than normal developers and unit testing, agile and whatever else you are pushing doesn't apply to the game industry because of [one of]:
    1. How can you unit test games anyway that doesn't even make sense!
    2. Deadlines. We don't have time.
    3. Games have hundreds of gigs of assets.
    4. Console contracts / monetisation models etc.
  3. I agree that the game industry suffers from some poor tool and process adoption habits. 
Those who use these practices I've been talking about - and to keep things specific I think automated testing is an excellent candidate to focus the debate - these people generally didn't seem too upset with me. In fact I know plenty of game developers in this category. 

Those who believe that their game studios are backward generally cannot speak publicly about ANYTHING with the only few exceptions being people who have left the industry, like munificent who says:

I worked at EA for eight years. My running joke-that-was-not-a-joke was that we were twenty years behind the times, but I was probably exaggerating. Fifteen is a pretty good estimate.
At EA, I think the problem was that the company culture was built like this:
  1. A bunch of bedroom hackers in their teens started studios and shipped games ten to twenty years ago.
  2. Those games got successful, and the studios scaled up. The founders hired young people under them.
  3. More staff churn. Fresh grads come in, burned out experienced coders go out.

Munificent went on to qualify and contain his own statements as generalisations (as I did with mine).

Tool and process adoption in development teams is of particular interest to me and I have an ongoing conversation with several developers about the barriers to solving problems they have in this area.

As for the middle group, the question boils down to this:

If automated testing is to some degree not suited to the game industry, then why?

If the answers are things like "How could you test Starcraft 1?" or "You can't test for things like game balance" then I'm sorry that's not a convincing reason! How is it obviously impossible to automatically test Starcraft? I'm sure game developers heavily using automated testing (perhaps even Blizzard) will call bullshit and say "yes you can automatically test it".

You Can't Write a Test for X

I've seen so many of these debates and had them myself since the 90s and I've come to believe slowly by being convinced on a case by case basis that you can test the vast majority of what you think you can't test. I've learned this working in a number of different software projects in different industries.

If the answer is the cost benefit trade-offs on this project lead us to conclude it's not worth doing here and there then that's interesting. I'm looking for examples because it may well be a tooling problem.

I'm not saying that everything can be tested automatically, and perhaps the up-front cost of writing the tests is never outweighed by the benefits of having the tests because of game industry specific business features or whatever, but I think many of those things are currently subject to rapid change.

Game Revenue

WOW is a good case study. This is the sort of game that couldn't be funded 15 years ago. It plays similarly to the MUDs of the early 90s with a lot better graphics etc. but the market was too small for such a grand vision then. We were on 14.4k modems (or worse) and the number of internet users was tiny. WOW has recurring revenue. This is a feature unlike traditional games but not unlike traditional and modern non-game software.

With ongoing revenue you can fund ongoing development in a more continuous way. Updates can be done more frequently and codebases are encouraged to have longer lifespans. Longer lifespan codebases mean legacy code. There's nothing to encourage you to invest more in automated testing like having to maintain legacy code!

I think these are important factors, driven by the modern consumer internet and the market forces that come from it. These forces push game code towards a scenario where automated testing has a bigger benefit.

That doesn't address the cost side. You might think that the costs of automated testing, particularly unit testing, are fixed. Well in my experience of unit testing for the past 12+ years, the biggest barrier to automated testing is bad design. Here's what munificent (who has since moved on to Google) had to say about that:

Unit testing would be awesome, but the thing is... you need units for that. A lot of the game codebases I've seen are closer to "giant hairball" than "collection of components", which isn't amenable to unit testing.

When I was writing software in the 90s and 80s I never wrote automated tests. I thought testing had to be manual and had to be done when the system was "finished". I don't think that any more. When someone tells me that they have to test that way I ask why and I think I'm right to be suspicious of weak, vague answers.


  1. I found for many testing problems that you need to be an expert in other domains to actually test things.

    For instance, audio code or random processes require signal processing knowledge or knowledge of non-parametric statistical tests.

    So here's a test you should try to write:
    * prove statistically that a random generator or encryption algorithm produces uniformly random values.

    If you have the knowledge you can solve it but you might need a lot of code to do it.

    So you can't test X if you don't have enough knowledge to get it. Heaven forbid you're not good at signal analysis or statistics.

  2. test-some-stats, that's an interesting case.

    I think that if you're going to write a random number generator, you had better not rely on manual testing or you will go mad!

    True you need knowledge to write the test, but then how are you supposed to implement the thing without that same knowledge? How would you know if you've written something that meets the needs unless you have quite a specific idea of what the right behaviour is?

    The problem is really just how do you implement that specific idea in code.

    Formal verification has always struck me as clearly being superior to the empirical automated testing alternative. I am not knowledgeable enough in it to know how to make it pay as well as unit tests for any of the code I write, however I did once exhaustively test a hash function to demonstrate that it was perfectly collision free on all possible inputs. That is not always possible or desirable to do.

  3. In my experience, there is nothing in game software which makes it inherently less testable than "regular" software.

    Even things like rendering engine can be a subject to automated regression testing (yes, we are doing it, as a part of continuous integration process, and yes, our studio is listed at atlassian.com/game-development).

    I am more or less with you on the majority of the sentiments (including the ones from the previous controversial post), but personally would avoid doing plain generalizations here, especially if I was an outsider to the game industry.

  4. Ruslan,

    Thanks for the comment - I will certainly avoid these kinds of generalisations in future!