Augmented Reality Demo With Apache Kafka and Machine Learning
Mobile shopping with an augmented retail experience.
Join the DZone community and get the full member experience.Join For Free
Augmented Reality (AR) and Virtual Reality (VR) get traction across industries far beyond gaming. Retail, manufacturing, transportation, healthcare, and other verticals leverage it more and more. This blog post explores a retail demo that integrates a cutting-edge augmented reality mobile shopping experience with the backend systems via the event streaming platform Apache Kafka.
Augmented Reality (AR) and Virtual Reality (VR)
Augmented reality (AR) is an interactive experience of a real-world environment where the objects in the real world are enhanced by computer-generated perceptual information. AR is a system that fulfills three basic features: a combination of real and virtual worlds, real-time interaction, and accurate 3D registration of virtual and real objects.
Pokemon Go from Nintendo is one of the most famous AR applications used by millions of people. The adoption of AR increased a lot in the last few years as modern smartphones and tablets support AR out-of-the-box. Fun fact: Pokemon Go is also a great success story for Google and Kubernetes to provide an elastic and scalable infrastructure: Bringing Pokémon GO to life on Google Cloud.
Augmented reality alters one's ongoing perception of a real-world environment, whereas Virtual Reality (VR) completely replaces the user's real-world environment with a simulated one. While the below demo is for AR, the combination with Kafka is possible for VR use cases similarly.
Apache Kafka and Augmented Reality SDKs
Before we jump to the use case, let's be clear AR and VR are not directly related to Apache Kafka. However, both are complementary to implement innovative use cases. Most AR and VR applications require communication with backend services and other users to provide the expected user experience. In real-time. At scale. 24/7. That's why Kafka is a perfect fit for AR and VR, similarly to all the other use cases across all verticals where event streaming helps.
AR and VR applications are built with specific 3D engines. This includes:
- Game engines (that move into other markets more and more) like Unity 3D, Unreal Engine, CryEngine, Lumberyard.
- Physics engines like Bullet, Havok, PhysX.
- Vertical solutions, for instance, Industry 4.0 / Industrial IoT (IIoT) products from PTC, Siemens, or General Electric.
- AR/VR SDK that is either included in the 3D engines or separate technologies such as Apple's ARKit for iOS devices, ARCore from Google, Vuforia, or EasyAR.
Fun fact: Many of these engines and vertical solutions do not just sell the software. They also provide additional services on top of their products. For the same reason, as in other companies, these services' central nervous system is often Apache Kafka. For instance, Unity is a heavy user of Apache Kafka in Confluent Cloud to handle on average about half a million events per second to process millions of dollars of monetary transactions for their in-game purchase and advertisement services.
Use Cases for Kafka and Augmented Reality
This blog post explores a use case from the retail industry: Online shopping with an enhanced user experience and location-based services. Similar examples exist across industries:
- Mobility services: Enhanced customer experience and location-based services for navigation, logistics, ...
- Manufacturing: Education and training on simulated machines, equipment repair.
- Gaming: Augmented real-world environments.
- Smart City: Simulated planning of urban, electricity, water, ...
- Industry-agnostic: Improved planning (before production), ergonomy simulation tests,
The heart of the applications is the AR/VR hardware and app.
The data's central nervous system is Apache Kafka to integrate with other systems and correlate the aggregated data sets continuously in real-time.
The Architecture of a Cutting Edge Retail Example
Kafka is the new black in retail across various use cases. Learn more details about event streaming in retail for omnichannel customer 360 experiences and hybrid Kafka retail architectures with edge deployments in the smart retail store.
The following example demonstrates online shopping with an enhanced user experience and location-based services. Customers can buy anything they see. Anywhere. The customer makes a picture of the thing. The backend processes the picture and augments it with additional information such as a digital picture, product details, and different shops to buy it.
This example shows how important the integration of the AR app with the company's backend services is:
A few notes on the architecture:
- The AR enhancements are possible on the client device or server-side. This depends on the business case and technical environment. For instance, the app could store images for products and only load updated details like the server's price.
- Kafka supports the handling of large messages (like images from the phone camera). Nevertheless, a comparison of the trade-offs is inevitable.
- The example uses Swift on the mobile app and Kafka-native stream processing tools (Kafka Streams and ksqlDB) on the backend side. However, the architecture is very flexible. Any other technology or 3rd party SaaS service is possible. That's the beauty of Kafka's strong support for domain-driven design (DDD).
Live Demo Video
The demo uses Swift, ARKit, Xcode, and Confluent Cloud. Hence, it only supports Apple iOS devices. But hey, it is just a demo to share the idea of building an AR app and integrate/correlate the data in real-time at scale with other backend systems using the Kafka ecosystem. Confluent Cloud provides the serverless SaaS offering so that the developer can focus on building the AR app and business applications in the backend.
Augmented Reality + Apache Kafka = Science Fiction?!
Augmented Reality and Virtual Reality are still in the early stages. Use cases like the online shopping experience - where you can buy anything you see anywhere - are often just a vision. But it is not far away from reality.
Kafka deployments exist across industries and business units already. If you want to build an innovative AR or VR app, it is just another microservices and mobile app to connect to the Kafka infrastructure. The integration of AR/VR into the supply chain is pretty straightforward. Kafka is typically already connected to the payment infrastructure, real-time inventory, and so on.
What are your experiences and plans for event streaming together with AR/VR use cases? Did you already build applications with Apache Kafka? Let’s connect on LinkedIn and discuss it! Stay informed about new blog posts by subscribing to my newsletter.
Published at DZone with permission of Kai Wähner, DZone MVB. See the original article here.
Opinions expressed by DZone contributors are their own.