Use Cases for Apache Kafka in Retail

The retail industry has completely changed. Consequently, traditional players must disrupt their own businesses to stay competitive. New business models, great customer experiences, and automated real-time supply chain processes are mandatory. Event streaming with Apache Kafka plays a key role in this evolution of the re-invented retail business. This blog post explores use cases, architectures, and real-world deployments of Apache Kafka such as edge, hybrid, and global retail deployments at companies like Walmart and Target.

Disruptive Trends in Retail

A few general trends have completely changed the retail industry:

• Highly competitive markets with thin margins
• Moving from "High Street" (brick & mortar) to online (omnichannel)
• Personalized customer experiences providing optimal buyer journeys 

These trends require retail companies to create new business models, provide a wider customer experience, and improve operational efficiencies. 

Event Streaming With Apache Kafka

Many use cases for event streaming are not new. Instead, Apache Kafka enables faster processing at a larger scale with a lower cost and reduced risk. 

Kafka is not just used for greenfield projects in the retail industry. It often complements existing applications in brownfield architecture. Plenty of material explores this topic in more detail:

• Kafka as modern middleware to replace or partner with other MQ/ETL/ESB infrastructure
• Kafka in conjunction with API Management tools
• Integration with legacy mainframe applications
• Event streaming as the foundation of postmodern ERP, logistics, and transportation systems

Let's now take a look at a few public examples that leverage all the above capabilities.

Real-World Use Cases

Various deployments across the globe leverage event streaming with Apache Kafka. It is the right choice, whether you need to optimize the supply chain, disrupt the market with innovative business models, or build a context-specific customer experience. Here are a few examples:

• Walmart - Real-Time Inventory System: 8,500 nodes process 11 billion events per day to deliver an omnichannel experience that allows every customer to shop how they prefer
• Target - Omnichannel Distribution and Logistics: Data correlation of events from distribution centers, stores, digital channels, and customer interactions enable real-time omnichannel scaling
• Nuuly - Clothing Rental Subscription Service: Different from a typical e-commerce model, it needs real-time event-driven architecture
• AO.com - Context-specific Customer 360: Hyper-personalized online retail experience, turning each customer visit into a one-on-one marketing opportunity via the correlation of historical customer data with real-time digital signals
• Mojix - Retail and Supply Chain IoT Platform: Real-time operational intelligence across the edge and cloud improves inventory accuracy, enables operational intelligence, and supports omnichannel sales

The architectures of retail deployments often leverage a fully-managed serverless infrastructure with Confluent Cloud, or they deploy in hybrid architectures across data centers, clouds, and edge sites. Let's now further explore an example.

Omnichannel and Customer 360 Across the Supply Chain with Kafka

Omnichannel retail requires the combination of various tasks and applications across the whole supply chain. Some tasks are real-time, while others use batch or historical data:

• Customer interactions – website, mobile app, and on-site
• Reporting and analytics – business intelligence and machine learning
• R&D and manufacturing
• Marketing, loyalty systems, and aftersales

The real business value is generated by correlating all the data from these systems in real-time. Kafka is a perfect fit due to its combination of different capabilities for real-time messaging at scale, storage for decoupling and caching, data integration, and continuous data processing.

Hybrid Architecture, From Edge to Cloud

The following picture shows the possible retail architecture for leveraging event streaming. It runs many mission-critical workloads and integrations in the cloud. However, the context-specific recommendations, point of sale payment and loyalty processing, and other relevant use cases are executed at the disconnected edge in each retail store. 

In this video and presentation, I dig deeper into this architecture and talk about the specific retail requirements and challenges solved by Kafka at the edge and in the cloud. You can also check out the following posts to learn about global Kafka deployments and how Kafka functions at the edge in retail stores:

• Kafka is the new black at the edge in retail stores
• Use cases and architectures for Kafka outside the data center
• Architectures for hybrid and global Kafka deployments

Software (Including Kafka) is Eating Retail

Event streaming with Apache Kafka plays a key role in the evolution of retail. Walmart, Target, and many other retail companies rely on Apache Kafka and its ecosystem to provide a real-time infrastructure that makes the customer happy, increases revenue, and stays competitive.

What are your experiences and plans for event streaming in the retail industry? Did you already build applications with Apache Kafka? Let’s connect on LinkedIn and discuss it! Stay informed about new blog posts by subscribing to my newsletter.