Pokemon: Gotta Scale Them All

Pokémon Go is awesome.
Pokémon Go is webscale.
Pokémon Go is life.

Mobile is hard, but scalable architectures are awesome. One of the best things about working with mobile dev teams is having a first-hand view of massive changes in the App Store landscape. As has become apparent over the last week, in the future, all such changes will be compared to the release of Pokémon Go.

We work with Niantic Labs to monitor the performance of both the iOS and Android versions of their apps. At critical times (like when launching an awesome new game!), they use our product to find and fix the issues that affect their users the most. This means that we had a front-row seat to last week's release of Pokémon Go. With adoption and usage of their new app through the roof, we had to quickly work to scale our backend infrastructure to keep up with Niantic's success.

In what our Ops team will forever refer to as "the Pokémon Scenario," our service started processing events from tens of millions of new users in a matter of days.

In the first 24 hours after launch, we saw the traffic on our ingest tier shoot up:

Increase of peak incoming traffic by ~500 requests/second after 24h. This increased to ~1500 RPS by Thursday night.

We're no strangers to big data processing: On a typical, pre-Pokémon day, we handle more than 2.6 billion events (~30k every second). However, over the last week we've tracked a major increase in traffic:

One week after launch: Total increase by ~5500 RPS, or ~475M additional events per day.

Thanks must be given to the team at Niantic for giving us a heads up about the release. Even with advanced notice, though, our Operations team had to work diligently to handle the new load on the system.

Fortunately, we are able to scale our platform horizontally at each tier, and the only impact that Pokémon had on our services was some delayed processing while we scaled out our Kafka and Spark clusters. In a later post, we'll go into the details of how we architected our platform to support such easy scaling (hint: asynchronous processing + λ-architecture for the win).

Hats off to Niantic Labs on their great launch.