24/7 Uptime Across Dozens of Sharded Clusters, Supported by MongoDB Enterprise Advanced and Just a Single Administrator
[This article was written by Mat Keep]
Square Enix is one of the world’s leading providers of gaming experiences, publishing such iconic titles as TOMB RAIDER and FINAL FANTASY. As more of its games moved online, Square Enix rapidly hit the scalability limits of relational databases and migrated to MongoDB. By adopting a multi-tenant database-as-a-service, Square Enix has been able to consolidate database instances, resulting in increased performance and reliability. Advanced operational tooling enables the operations team at Square Enix to scale dozens of database clusters on-demand and deliver 24x7 availability to gamers around the world, all with just one administrator.
With this year’s E3 Expo just around the corner, I had the opportunity to sit down with Tomas Jelinek, Senior Online Operations Administrator at Square Enix Europe to discuss how his platforms have evolved to meet the demands of millions of dedicated online gamers around the world.
Can you start by telling us a little bit about Square Enix?
Square Enix is one of the leading and most influential providers of digital entertainment content in the world. We have a valuable portfolio of gaming titles including FINAL FANTASY, DRAGON QUEST, TOMB RAIDER, HITMAN, DEUS EX and JUST CAUSE, which collectively have sold hundreds of millions of units worldwide.
We are always seeking ways to push the boundaries of creativity and innovation by providing high quality entertainment content and products. As gaming is right at the intersection of big data, mobile, and cloud computing, our infrastructure platforms are critical to staying ahead of the market and giving gamers experiences they won’t get anywhere else.
We will be making some really interesting announcements at E3 so it’s an exciting time to be working as part of the Ops team that keep all of our gaming platforms running.
Tell us how Square Enix is using MongoDB.
Originally we ran our games on a platform where data was stored in old-fashioned relational databases. But that wasn’t efficient as our titles multiplied, game complexity increased and datasets grew out of proportions. So we built our multi-tenant Online Suite – a central shared infrastructure used across the company. We deliver MongoDB-as-a-Service to all of our studios and developers. As part of this Online Suite we provide an API that allows the studios to use MongoDB to store and manage metrics, player profiles, info cast information, leaderboards and competitions. MongoDB is also used to enable players to share messages across all supported platform such as PlayStation, Xbox, PC, web, iOS, and Android etc. Essentially, the Online Suite supports any functionality that is needed across multiple games.
Every title also needs to support its own specific in-game functionality, and so each is provisioned with dedicated infrastructure connected to MongoDB to store game state and player metrics, along with specific content and features. For example, Hitman Absolution introduced the ability for players to create their own contracts, and then share those with other players. This is all managed by MongoDB.
We also use MongoDB for inter-game and inter-player messaging. Our web and mobile sites use MongoDB for content management and product catalogs.
Just Cause 3 - dropping onto a platform near you.
Just Cause 3 © 2015 Square Enix Ltd
What were you using before MongoDB? Was this a new project or did you migrate from a different database?
The move to online gaming started back in 2007. We used relational databases to store player profiles and leaderboards, and to run analytics across metrics we collected from the game play. But as our online audience grew, the databases didn’t scale with us. We ended up bringing in teams of consultants to buy us more headroom, but the existing databases just couldn’t keep up.
We started to ask increasingly tough questions of our data. One complex analytics query took 3 weeks to run in our relational database! We knew it was time to start looking for an alternative.
How did you hear about MongoDB? Did you consider other alternatives?
We started our search for a replacement in 2011. We needed a database that could meet the requirements of both our development teams in Montreal and our ops team here in London.
The development teams were focused on how quickly they could build new games and add functionality to maximize game lifecycles. They also needed to ensure all of the operational and analytical functionality they demanded could be supported by the database. So a flexible schema and expressive query language were important to them.
In operations, we needed to validate scalability and robustness of the database. Customer experience is not something we could afford to compromise on, even if the developers loved the database! For ongoing manageability, we also needed to verify that the database could fit into our existing operational workflows and tooling.
Each team embarked on its own evaluations. We built Proofs of Concept and looked at various established and new database technologies, including MongoDB. We hired an external consultant to help guide the process.
All of the teams reached the same conclusion: MongoDB was the best fit for our next generation gaming platform. And that complex query that took 3 weeks to run in our relational databases…we got it down to 2 minutes in MongoDB! So in-game analytics was a go.
In the context of database evolution, 2011 was an eternity ago, and I’m sure a lot has changed in products we evaluated. But we’ve never looked back with MongoDB.
Can you describe your MongoDB deployment?
We mainly run MongoDB 2.6 today. Each game server instance is deployed on VM running Ubuntu Linux, Nginx & Jetty connecting to MongoDB with the Java driver.
Our multi-tenant Online Suite is provisioned across the main 10-shard MongoDB cluster. Each shard is configured with a 3-node replica set running a primary and secondary node and a replica set arbiter. Each MongoDB instance runs on its own physical server in our data centres.
We use a different architecture for the clusters dedicated to other projects supporting individual titles. The load our games place on the backend is very spikey. It’s not uncommon to provision 60+ front-end servers to support a game at launch, and then dial that back as traffic reduces. Our marketing team often runs promotions around individual games, and with this approach, we are able to just add nodes back into the cluster when they are needed. This type of elasticity is essential to ensure we are keeping costs down by avoiding over-provisioning. MongoDB gives us solid persistence layer to support such an approach.
Many of our dedicated games clusters are deployed on AWS, and we have started to use theMongoDB Management Service (MMS) to automate provisioning and configuration, as well as manage upgrades.
I’ve found MMS to be extremely useful – it saves us a lot of time, and means we can scale our infrastructure without having to scale our ops personnel at the same time. It also means I can get a lot more done, and have a life outside of work!
Automated MongoDB provisioning on AWS EC2