Automated game testing

Something that is very cumbersome (and boring) to do for every developer out there is the software testing. To combat that, you usually develop a number of methods that helps you track down bugs, for example automatic bug-reports and unit-testing.

One of the more interesting ones is automated testing. For games, this usually means that you have a system that simulates the player and automatically generates input for it. For my latest project Hyena. I’ve had the rare opportunity to play around with an application where the controls are so basic that it was actually possible to write a system that played the whole game for me.

For Hyena this was obviously extremely simple, since controls only require you to choose from of a number of predefined choices. The game-book I used for testing was Flight From The Dark, which has 350 pages and each page has on average about 1.6 choices, so fully testing the game was definitely possible. Some preliminary runs did prove very helpful. First of all, it pointed up some show-stopper bugs in sections like this where you ended up with all choices disabled due to a problem in the LUA-script. The other kind bug it pointed out appeared when I looked at the pages that never was displayed, and could trace that to another LUA-script bug.

Also, doing simple analyses of such runs are also trivial and can be a great tool for game-design. You might remember hearing of such analysis from the development of the original Half-Life. Basically, they built a system that tracked stuff like the health of the player throughout the levels, and having this activated during focus-testing allowed them balance the game better. We (Starbreeze) asked Valve about that a few years ago while working on Riddick, and while they still were very enthusiastic about the idea, they also told us that it never got to the point where it was really useful. My guess is that the big problem for most games is that you still need to rely on real players for the data to be useful.

Then again, that’s not equally true for Hyena and game-books in general, since there isn’t that much of a difference between a simulate player and a real one. The only limitation my simulated player has is that it never goes to the action chart and actively uses items (healing potions for example). Of course, it also happily chooses a really stupid-sounding choice as often as a sane one. Still, the analysis gives a lot of information about how well the game-book is structured. Take a look:

http://www.collectingsmiles.com/hyena/analysis/fftd_analyze.jpg

This is a flowchart generated from a simulated run of 40139 playthroughs (yes, it took a few hours) and shows for example that:
• The game is  completed in 7.5% of the cases
• The Gourgaz kills you in 57% of the cases
• The most dangerous enemy are The Doomwolves, who kills you in 74% of the cases

It also displays a couple of “flaws”:
• There is this really big path that takes off from near-death experience in this section that is shown in less than 0.1% of the cases.
• A couple of sections are never displayed, either because of the problem above, or because of this one, where a fight extremely seldom lasts 4 rounds.

I intend to expose this analysis functionality in Hyena, so this tool will hopefully be available to anyone who wants to write his own game-book.

4 thoughts on “Automated game testing

  1. Hello Mr. Andersson,

    I am very interested in automatic software testing with computer games. I read your article and I asked me if you have more information about that topic. I would like to write a paper for my study and I am searching for interesting stuff to do that.

    Thank you
    Daniel

  2. Hey Daniel,

    Cool, it’s quite an interesting topic I think, but it might be hard to find too much information. I don’t really have any more information for you either, as I haven’t really pursued it much further. I did more experimentation with more complicated systems, but I was more after interesting ways to visualize the flow of a game which could help the design processes. It worked really well for the choose-your-own-adventure type of branching story used in Hyena, but with more complicated setups it started to fall apart.

    The idea was that through sampling a massive amount of playthroughs generate a graph showing how player choices led to consequences later on. The game that I worked on was very complicated in this aspect (time passing was a key factor in what happened in the game) and I failed to visualize this complicated system in a good way. It worked well for bug-finding though where it helped find some crash bugs and also some progression blockers.

    From my experience in the games industry, people are spending most of their efforts in metrics/telemetry, where they attach tools to the QA team and focus testing groups to get more accurate information. The extent of automatic game testing in most projects is usually limited to smoke testing where you automatically load up each level in the game and do some basic test. Also, input stress testing is fairly common, where you generate random button presses over a long period of time.

    Not sure if this was any help, but let me know if there is anything else I can do to help. And I’d love to read your paper once it’s done.

  3. I hope as a late comer to this site that it’s still viable to ask – I downloaded the gamebook engine with the intent of writing my own simple stories for fun – thank you for making it! – but I cannot find the hyena scripting language information. Would you be willing to email this to me as the page is gone from your site? Also, I’d love to have access to the tool for checking out how my gamebook might play out!

    Thank you for considering my request.

    Warm regards,
    Elizabeth

Leave a Reply to Jens Cancel reply

Your email address will not be published. Required fields are marked *