It had all the makings of a successful product launch: a highly-recognizable brand, a built-in global fanbase of millions, and a great team of game developers with roots in a silicon valley icon—Google. But the launch of Pokemon Go, the highly anticipated augmented-reality mobile game, was immediately followed by a barrage of social media complaints over server outages. And the fact that this wasn’t even a worldwide release further illustrates the critical challenge the online gaming industry faces in scaling their applications to global demand.
While the cause of this particular failure isn’t clear yet, problems with applications that rely on online transaction processing (OLTP), such as those used in online gaming, are often rooted in the underlying database. In particular, companies leveraging relational databases have to think about issues associated with scaling to peak demand, concurrency, and latency. These challenges arise due to the limitations of relational databases commonly used to power online gaming applications—usually MySQL or one of its derivatives—which were architected for single-server implementation only, and don’t allow you to scale out like other cloud applications.
These problems can only be solved by using scale-out relational database technologies that allow you to maintain linear performance and the full data integrity of ACID compliance.
Scaling to Erratic Demand Levels
Life would be so much easier if it were steady and predictable. Of course it isn’t—for people, or video games. Online gaming applications deal with widely varying levels of demand, and typically have to scale to much higher demand when a new title is launched, to accommodate signups. This often involves provisioning computing hardware that can handle the upper ceiling of projected demand.
We blogged about this back in March, noting the roller-coaster-esque life cycle of gaming applications, where at release you don’t know what you’ll really need and often either over- or under-provision computing power. In this case, as complaints piled up from players who couldn’t get into the system just to sign up, it appears that Pokemon may have under-provisioned for higher-than-expected demand.
But, whether you under or over-provision for such scenarios, gaming companies often face a secondary challenge when demand tapers to normal levels: being able to scale down the system so you don’t have to pay for excess capacity. Cloud infrastructures allow you to scale up and down to demand at the application layer, because most gaming applications are designed to “scale out” by adding nodes. But, this isn’t necessarily true of the databases that power gaming applications, due to the aforementioned reliance on pre-cloud relational technologies. This kind of flexibility can only be achieved by relational database technology that is architected to scale by adding or subtracting server nodes.
Millions of People Playing at the Same Time
With the launch of a highly anticipated game, you may be talking about millions of users concurrently running a variety of “write-intensive” transactions that demand ACID compliance. Once again, we run into a limitation of most relational databases. MySQL and many of its derivatives are used because of their ability to maintain ACID compliance, and may afford the ability to scale reads. Amazon RDS, for example, provides 15 ‘Read Replicas’ very simply via their AWS Console—but they don’t necessarily scale in writes.
Marc Staimer of Dragonslayer Consulting pointed out in “Why Traditional SQL Databases Fail to Scale Effectively”, that read-only slaves “cannot overcome the bottleneck of the master SQL database especially for write-intensive applications.” In other words, applications powered by databases that can’t scale writes, under the stress of a high number of simultaneous players, tend to resemble the LA freeway at rush hour: gridlocked.
Database Latency = Slow Reaction Time = Equals Angry Gamers
What would happen to an athlete if their reaction time were suddenly doubled? They’d lose. Similarly, database latency yields slow reaction time, essentially taking the “real time” out of the multi-player experience. It kind of defeats the purpose of the whole thing, and leaves you with a lot of frustrated players. In order to serve a large number of gamers simultaneously, your database has to be able to scale out linearly, ensuring that low latency remains low regardless of how much capacity you’re running.
Once again, we see that MySQL and its various replacements are limiting in this respect, because the one-server architecture makes it very difficult to scale to high demand without sacrificing in terms of latency, or ability to maintain ACID compliance—or both. As we pointed out in our previous blog on this subject, “Sharding” and “Master-Slave” may sound like cool terms from The World of Warcraft, but they’re actually very complicated tactics used to stretch the performance of a relational database beyond its natural limits. In addition to the many problems with these techniques, they also add a lot of complexity to your application, ensuring that it will require more management and overhead for the long term—translating to additional cost, and a greater chance of something going wrong.
Online Gaming Requires a Database Architected for the Cloud
Online gaming is an industry firmly rooted in the cloud, and its applications require a database engine that’s also architected for the cloud. It must provide the ability to scale out by adding server nodes like other cloud applications, while also supporting high concurrency for write-intensive transactions—i.e. potentially millions of simultaneous users—and maintaining ACID compliance and linear performance. That may sound like a tall order, but in reality it’s not, and we’ve been working with top online companies for years to provide these very capabilities.
Of course, when gamers experience downtime, they hang up their controllers and take a break, turning to productive activities like Yoga or helping their spouses with the dishes, right? With this launch, I think Pokemon would beg to differ. Rather, gamers go online and complain about your game–and they can get pretty creative in their complaining!
So do yourself a favor: Keep your gamers happy by powering your games with scale-out SQL that supports high concurrency and lets you flex your computing power to match the erratic demands that your users will place on it.