The State of Data Streaming With Apache Kafka and Flink in the Gaming Industry

This blog post explores the state of data streaming for the gaming industry in 2023. The evolution of casual and online games, Esports, social platforms, gambling, and new business models require a reliable global data infrastructure, real-time end-to-end observability, fast time-to-market for new features, and integration with pioneering technologies like AI/machine learning, virtual reality, and cryptocurrencies. Data streaming allows integrating and correlating data in real-time at any scale to improve most business processes in the gaming sector much more cost-efficiently.

I look at game industry trends to explore how data streaming helps as a business enabler, including customer stories from Kakao Games, Mobile Premier League (MLP), Demonware / Blizzard, and more. A complete slide deck and on-demand video recording are included.

General Trends in the Gaming Industry

The global gaming market is bigger than the music and film industries combined! Digitalization has played a huge factor in the growth in the past years. The gaming industry has various business models connecting players, fans, vendors, and other stakeholders:

• Hardware sales: Game consoles, VR sets, glasses
• Game sales: Physical and digital
• Free-to-play + in-game purchases: One-time in-game purchases (skins, champions, miscellaneous), gambling (loot boxes)
• Game-as-a-service (subscription): Seasonal in-game purchases like passes for theme events, mid-season invitational & world championship, passes for competitive play
• Game-Infrastructure-as-a-Service: High-performance state synchronization, multiplayer, matchmaking, gaming statistics
• Merchandise sales: T-shirts, souvenirs, fan equipment
• Community: Esports broadcast, ticket sales, franchising fees
• Live betting
• Video streaming: Subscriptions, advertisements, rewards,

Growth and Innovation Require Cloud-Native Infrastructure

Most industries require a few specific characteristics. Instant payments must be executed in real time without data loss. Telcom infrastructure monitors huge volumes of logs in near-real-time. The retail industry needs to scale up for events like Christmas or Black Friday and scale down afterward. The gaming industry combines all the characteristics of other industries:

• Real-time data processing
• Scalability for millions of players
• High availability, at least for transactional data
• Decoupling for innovation and faster roll-out of new features
• Cost efficiency because cloud networking for huge volumes is expensive
• The flexibility of adopting various innovative technologies and APIs
• Elasticity for critical events a few times a year
• Standards-based integration for integration with SaaS, B2B, and mobile apps
• Security for trusted customer data
• Global and vendor-independent cloud infrastructure to deploy across countries

The good news is that data streaming powered by Apache Kafka and Apache Flink provides all these characteristics on a single platform, especially if you choose a fully managed SaaS offering.

Data Streaming in the Gaming Industry

Adopting gaming trends like in-game purchases, customer-specific discounts, and massively multiplayer online games (MMOG) is only possible if enterprises in the games sector can provide and correlate information at the right time in the proper context. Real-time, which means using the information in milliseconds, seconds, or minutes, is almost always better than processing data later (whatever later means):

Data streaming combines the power of real-time messaging at any scale with storage for true decoupling, data integration, and data correlation capabilities. Apache Kafka is the de facto standard for data streaming.

"Apache Kafka in the Gaming Industry" is a great starting point to learn more about data streaming in the games sector, including a few Kafka-powered case studies not covered in this blog post - such as

• Big Fish Games: Live operations by monitoring real-time analytics of game telemetry and context-specific recommendations for in-game purchases
• Unity: Monetization network for player rewards, banner ads, playable advertisements, and cross-promotions.
• William Hill: Trading platform for gambling and betting
• Disney+ Hotstar: Gamification of live sport video streaming

Architecture Trends for Data Streaming

The gaming industry applies various trends for enterprise architectures for cost, elasticity, security, and latency reasons. The three major topics I see these days at customers are:

• Fully managed SaaS to focus on business logic and faster time-to-market
• Event-driven architectures (in combination with request-response communication) to enable domain-driven design and flexible technology choices
• Data mesh for building new data products and real-time data sharing with internal platforms and partner APIs

Let's look deeper into some enterprise architectures that leverage data streaming for gaming use cases.

Cloud-Native Elasticity for Seasonal Spikes

The games sector has extreme spikes in workloads. For instance, specific game events increase the traffic 10x and more. Only cloud-native infrastructure enables a cost-efficient architecture.

Epic Games already presented at an AWS Summit in 2018 how elasticity is crucial for data-driven architecture.

Make sure to use a truly cloud-native Apache Kafka service for gaming infrastructure. Adding brokers is relatively easy. Removing brokers is much harder. Hence, a fully managed SaaS should take over the complex operations challenges of distributed systems like Kafka and Flink for you. The separation of computing and storage is another critical piece of a cloud-native Kafka architecture to ensure cost-efficient scale.

Data Mesh for Real-Time Data Sharing

Data sharing across business units is important for any organization. The gaming industry has to combine exciting (different) data sets, like big data game telemetry, monetization and advertisement transactions, and 3rd party interfaces.

Data consistency is one of the most challenging problems in the games sector. Apache Kafka ensures data consistency across all applications and databases, whether these systems operate in real-time, near-real-time, or batch.

One sweet spot of data streaming is that you can easily connect new applications to the existing infrastructure or modernize existing interfaces, like migrating from an on-premise data warehouse to a cloud SaaS offering.

New Customer Stories for Data Streaming in the Gaming Sector

So much innovation is happening in the gaming sector. Automation and digitalization change how gaming companies process game telemetry data, build communities and customer relationships with VIPs, and create new business models with enterprises of other verticals.

Most gaming companies use a cloud-first approach to improve time-to-market, increase flexibility, and focus on business logic instead of operating IT infrastructure. Elastic scalability gets even more critical with all the growing real-time expectations and mobile app capabilities.

Here are a few customer stories from worldwide gaming organizations:

• Kakao Games: Log analytics and fraud prevention
• Mobile Premier League (MLP): Mobile eSports and digital gaming
• Demonware / Blizzard: Network and gaming infrastructure
• WhatNot: Retail gamification and social commerce
• Vimeo: Video streaming observability

Resources To Learn More

This blog post is just the starting point. Learn more about data streaming in the gaming industry in the following on-demand webinar recording, the related slide deck, and further resources, including pretty cool lightboard videos about use cases.

On-Demand Video Recording

The video recording explores the gaming industry's trends and architectures for data streaming. The primary focus is the data streaming case studies. 

I am excited to have presented this webinar in my interactive light board studio:

This creates a much better experience, especially in a time after the pandemic, where many people are "Zoom fatigue".

Check out our on-demand recording:

Slides

If you prefer learning from slides, check out the deck used for the above recording here.

Case Studies and Lightboard Videos for Data Streaming in the Gaming Industry

The state of data streaming for gaming in 2023 is fascinating. New use cases and case studies come up every month. This includes better end-to-end observability in real-time across the entire organization, telemetry data collection from gamers, data sharing and B2B partnerships with engines like Unity or video platforms like Twitch, new business models for ads and in-game purchases, and many more scenarios.

Gaming is one of many industries that leverage data streaming with Apache Kafka and Apache Flink. Every month, we talk about the status of data streaming in a different industry. Manufacturing was the first. Financial services second, then retail, telcos, gaming, and so on... Check out my other blog posts.

Let’s connect on LinkedIn and discuss it!