- The Eternal Beta
- Don't Listen to What People Say, Look at What They Do
- Collecting Data
- AB Testing
- Statistics: The Analyst's Toolbox
- The Art of Interpretation
- Gather Data, Test, Analyze
Don’t Listen to What People Say, Look at What They Do
Whether or not Henry Ford actually said, “If I asked my customers what they wanted, they’d have said a faster horse,” what has been proven is that the opinions of your customers aren’t always the best indicator of what they want or are willing to do.
Imagine if you asked your players: “Would you buy fewer IAPs if the price went up?” The majority of your players would say “yes.” They have a bias; they don’t want you to increase prices. However, in reality their actions may not change in the event of a small increase in IAP prices. The reason is that humans are complex, and the choices they make are based on the situation. What people believe they will do isn’t what they’ll actually do.
Think about when you signed up for that gym membership but didn’t go and instead ate burgers while sitting on your couch. You, like your players, have difficulty predicting your behavior.
That’s not to say that speaking with your players is a waste of time; indeed, you should ensure that you communicate with them as much as possible. It helps you understand how they think, plus by making them feel heard and showing they’re appreciated your players feel valued.
However, to really understand their behavior and their actions, you must instead place the emphasis on what they do rather than what they say.
The application of analytics to the iterative design of your game is the same as the application of the scientific method. The scientific method relies on observable evidence to form and give credibility to a hypothesis (Figure 4.1). The process is as follows:
- Form a hypothesis. Develop an idea or concept, such as increasing the price of a certain IAP by $1 will in turn increase monthly revenue by $100,000.
- Test the hypothesis. Create an environment where the hypothesis should occur, such as increasing the price of the identified IAP by $1 in a live game.
- Gather data. Gather empirical evidence relevant to the hypothesis, such as sales and revenue figures before and after the price hike.
- Interpret the data. Build an understanding of what happened in the test using the collected data. Then use this understanding to strengthen the hypothesis or to undermine it and form a new hypothesis. If a new hypothesis is formed, repeat the process.
Figure 4.1. The scientific method.
Constant theorizing and testing creates an ever-increasing framework of understanding around your players and helps you build a better game for them.
To gather the vital metrics needed for your analysis, you need an analytics package.