I’ve participated in a number of alpha/beta tests for various games over the past few years, both open and closed in nature. Though I don’t think I could pick out one particular test that I found the most productive or most enjoyable, there are some elements that I can definitely identify as being conducive to good testing results. I’m not going to name specific games here, since some of the experiences I’m referencing are technically still under NDA, even if some of the corresponding games have already been publicly released (and others definitely haven’t).

Create a Testing Community

Every single one of the beta tests that I would give high marks for both productivity and enjoyment has done this in some way or another - they’ve gone beyond just handing testers a game and then asking them to fill out a survey after playing around with it some. Instead, they’ve created a sense of togetherness amongst their tester-base. For most beta tests, this means at the very minimum having some form of private discussion boards where testers can chat with each other. Not only is it a good idea to have boards for testing purposes, but throwing in some form of an “off-topic” area is almost a necessity if you really want people to feel like they belong - otherwise, the forums wind up feeling too much like a job (or you just get off-topic posts in your feedback section, which can be rather distracting).

Say What You’re Looking For

Some developers do this better than others. I’m sure it’s probably tempting to any company to drop players into the game, let them play to their hearts’ content, and then ask them for any feedback they might have from their experiences. There’s a couple of problems with this approach.

1. For the majority of the testing period the game isn’t going to be finished - so a good portion of the feedback you’d wind up getting will become moot as soon as whatever feature you already had planned to address a particular shortcoming finally makes it into the live build.
2. It’s unlikely that you’re going to be working on every part of the game simultaneously - some portions will reach a final or near-final state quicker than others; some won’t even begin to be implemented until later in the development cycle, and some will get put on hold for stretches of time for one reason or another. As things change over time, feedback you receive on portions of the game you’re not actively working on might become out of date due to changes in how other, more actively developed items interact with the topic of the original feedback.
3. Aside from the occasional wall of text post about some issue or another, most testers tend to give more in-depth or specific feedback when they have a prompt to work off of - it gives them something to keep in their mind while testing, so that various impressions and thoughts regarding that particular topic will stick around for when they go to put their thoughts into words.

Make Your Conversations Two-Way Streets

For basic testing, simply asking some questions and giving testers to give you their answers in the form of observations, thoughts, and opinions can work out fine. To really harness your testers, though, you have to go beyond what amount to simple questionnaires and instead engage your testers in dialogue. Don’t just listen to your testers, talk to them, and talk to them in meaningful ways: whenever possible, try to provide the context surrounding design decisions and changes. While a certain percentage of testers in any test group larger than a handful will probably be off in their own little world, you’ll find that a surprising number of testers are more than willing to sign on to a clearly-explained set of development goals and visions.

When getting feedback, try to respond to it - even if it requires responding to multiple people’s feedback with a general response to all of them combined. If you agree with the sentiments expressed, responses are easy - “yep, we’re planning on doing that” or the like. If you think the feedback is mistaken though, you should still try to respond - if at all possible try to lay out at least the basic reasoning for why the tester(s) might have arrived at a wrong conclusion.

Whenever possible, try to avoid the “trust us, we know something you don’t know” type of response - the best feedback typically comes from those who don’t trust the developers to get everything right, and thus put in the effort to lay out their own reasoning in a clear and complete manner in hopes that the game will benefit from their feedback. While sometimes you’ll probably be forced to stick with “trust us” (temporarily at least, due to not wanting to fully commit to certain aspects of design until later in the development cycle), make sure you have a good reason for withholding information from your testers if there’s even a hint that having the information might help them provide more useful feedback. Also consider ways in which you might be able to convey the general reasoning behind a decision, even if you can’t go into implementation specifics until later.

Get People Involved and Keep Them Involved

This goes hand-in-hand with creating a testing community - have your testers be active. Whether this means hosting goofy events (like a “everyone has dev cheat commands” night) or contests or something else, it’s pretty consistent that people who enjoy testing will keep testing, and keep providing useful feedback. Turnover rates are always an issue in any real beta test (no, the “open betas” which are really publicity stunts don’t count).

When some testers do inevitably become inactive, be aware of it. If there’s a need to let more testers in to keep an active testing community, then don’t hesitate to take what measures are necessary to keep the testing population steady. If need be, contact inactive testers and ask them if they plan to continue testing, and fill the gaps left by those who decline or don’t respond. A sudden drop in tester population can have an adverse effect on the morale of all involved, even among the more dedicated testers still participating.

* * *

Perhaps I’m writing this pointlessly; maybe the only people who read this won’t ever be in the position of managing a beta test. But hopefully, some developer heading in to beta will, by the whims of fate, happen to randomly stumble across this blog entry and take it to heart. If you happen to be that developer… let me know, I want to test your game. ;)

« Back to index