Second my post was too long and based on my stats, it looks like less that 1.0% of visitors read the article. Unless they can read a couple of thousand words in 20 seconds. Again, that was my fault.
But what was interesting was the responses to my article were pretty clearly divided into two main groups (disagreeing and flaming) and one small group (agreeing):
- You are on crack, we do this stuff (automated testing, etc) better than anyone.
- Game developers are moar awesome than normal developers and unit testing, agile and whatever else you are pushing doesn't apply to the game industry because of [one of]:
- How can you unit test games anyway that doesn't even make sense!
- Deadlines. We don't have time.
- Games have hundreds of gigs of assets.
- Console contracts / monetisation models etc.
- I agree that the game industry suffers from some poor tool and process adoption habits.
Those who use these practices I've been talking about - and to keep things specific I think automated testing is an excellent candidate to focus the debate - these people generally didn't seem too upset with me. In fact I know plenty of game developers in this category.
Those who believe that their game studios are backward generally cannot speak publicly about ANYTHING with the only few exceptions being people who have left the industry, like munificent who says:
I worked at EA for eight years. My running joke-that-was-not-a-joke was that we were twenty years behind the times, but I was probably exaggerating. Fifteen is a pretty good estimate.At EA, I think the problem was that the company culture was built like this:
- A bunch of bedroom hackers in their teens started studios and shipped games ten to twenty years ago.
- Those games got successful, and the studios scaled up. The founders hired young people under them.
- More staff churn. Fresh grads come in, burned out experienced coders go out.
Munificent went on to qualify and contain his own statements as generalisations (as I did with mine).
Tool and process adoption in development teams is of particular interest to me and I have an ongoing conversation with several developers about the barriers to solving problems they have in this area.
As for the middle group, the question boils down to this:
If automated testing is to some degree not suited to the game industry, then why?
If the answers are things like "How could you test Starcraft 1?" or "You can't test for things like game balance" then I'm sorry that's not a convincing reason! How is it obviously impossible to automatically test Starcraft? I'm sure game developers heavily using automated testing (perhaps even Blizzard) will call bullshit and say "yes you can automatically test it".
You Can't Write a Test for X
I've seen so many of these debates and had them myself since the 90s and I've come to believe slowly by being convinced on a case by case basis that you can test the vast majority of what you think you can't test. I've learned this working in a number of different software projects in different industries.
If the answer is the cost benefit trade-offs on this project lead us to conclude it's not worth doing here and there then that's interesting. I'm looking for examples because it may well be a tooling problem.
I'm not saying that everything can be tested automatically, and perhaps the up-front cost of writing the tests is never outweighed by the benefits of having the tests because of game industry specific business features or whatever, but I think many of those things are currently subject to rapid change.
WOW is a good case study. This is the sort of game that couldn't be funded 15 years ago. It plays similarly to the MUDs of the early 90s with a lot better graphics etc. but the market was too small for such a grand vision then. We were on 14.4k modems (or worse) and the number of internet users was tiny. WOW has recurring revenue. This is a feature unlike traditional games but not unlike traditional and modern non-game software.
With ongoing revenue you can fund ongoing development in a more continuous way. Updates can be done more frequently and codebases are encouraged to have longer lifespans. Longer lifespan codebases mean legacy code. There's nothing to encourage you to invest more in automated testing like having to maintain legacy code!
I think these are important factors, driven by the modern consumer internet and the market forces that come from it. These forces push game code towards a scenario where automated testing has a bigger benefit.
That doesn't address the cost side. You might think that the costs of automated testing, particularly unit testing, are fixed. Well in my experience of unit testing for the past 12+ years, the biggest barrier to automated testing is bad design. Here's what munificent (who has since moved on to Google) had to say about that:
Unit testing would be awesome, but the thing is... you need units for that. A lot of the game codebases I've seen are closer to "giant hairball" than "collection of components", which isn't amenable to unit testing.
When I was writing software in the 90s and 80s I never wrote automated tests. I thought testing had to be manual and had to be done when the system was "finished". I don't think that any more. When someone tells me that they have to test that way I ask why and I think I'm right to be suspicious of weak, vague answers.