We recently completed a pilot of a large scale system designed to provide a stateful, real-time view of customers that can be applied in any industry. In this case, we worked with a multi-national network service provider to collect call, and other data in real time, to drive real-time “customer engagement” actions. The actions may be emails, text, outbound calls, or push notifications to a web app.
The goal is to provide a real time stateful view of each customer, to ensure engagement is contextually relevant and delivered in real time. The result should be increased revenue per customer and reduced churn.
A Real-Time Engagement Model
Real-time customer engagement systems collect and filters data sources, recognizes events, and combines events to recognize scenarios and take real time action.
In this case, both customers and their devices can be viewed as “actors.” The actors generate events, such as reboots, calls, web sessions, bill payments, and contract renewals. Combinations of events, often combined with customer profile information, are used to define a “scenario.” The Customer Engagement system recognizes when events meet the scenario and triggers the corresponding real-time action. The model is illustrated below.
EVAM, Apache Spark, and AWS
EVAM is a player in the real-time event processing (Streaming Analytics) field, with over 40 enterprises worldwide using EVAM to drive engagement with over 200 million end customers. The architecture begins with the identification of events that need to be captured, and data integration (listeners) filter and parse data streams and provide the formatted event message to the system. The formatted events are ingested by the EVAM engine, which is hosted on a distributed in-memory cluster of compute nodes, which processes the events, recognizes scenario matches, and triggers the real-time response. The process from end-to-end is accomplished in 10-30 milliseconds.
Apache Spark is an established open source project, with significant real-time data streaming and processing capabilities. Spark includes support for data filtering and parsing, with events handled via independently operated applications. Some of the challenges that limited consideration of Spark for this project includes the challenge of incorporating real-time events into a time based contextual scenario. For example, a wireless network provider will care about dropped calls that occur within a time window of several hours (ie., three dropped calls within three hours). Also, detecting “non-events” is challenging with Spark (ie., a device has not been activated in the five business days following purchase). EVAM provides a robust system for defining complex business driven scenarios that combine real-time events with time windows, non-events, and customer profile data, with a Visual Scenario Designer that is aimed at business users. For these and other reasons this project was implemented using EVAM. We’ll write more about EVAM working with Spark in a future blog.
AWS has launched significant new capabilities and we decided to utilize a range of AWS capabilities to deliver a world-class solution in this project.
EVAM + AWS for Real-Time Event Processing
The system is based on wireless calls within each operating unit (usually defined by a country), which are captured, parsed and event details streamed to AWS Kinesis (Step 1). Other call details are captured as they occur in the API gateway, and those are also streamed in real time to AWS Kinesis (2). Other customer data (billings, payments, etc.) are updated to a RedShift based data warehouse (3). A specialized customer profile is generated from the data warehouse to augment real-time data for engagement purposes (4). The profile is roughly 100 columns of data and hundreds of bytes in size, and can be kept online in the distributed in-memory cache.
The combination of the real time events are processed in the EVAM event engine to recognize defined scenarios, which combine the events with time windows and can include non-events, and customer profile data, to trigger appropriate customer engagement actions. In this case, the resulting actions are outbound emails, text, and push notifications that are routed through the API gateway to the end customer (5). The entire process is easily serviced in 30 milliseconds from event to action.
By keeping track of calls (success and failure), and a view on where each customer stands on contract expiration and renewal can give a network operator a distinct advantage in customer engagement:
- Every day, all customers within X days of needing to repurchase their pre-paid plans can receive appropriate incentives to repurchase
- At any time, a customer who experiences repeated failed calls within a prescribed time window can receive a real-time engagement action
- Devices purchased but either not activated, or not being used, can trigger a follow up with the customer with helpful hints on getting started
Amazon AWS provides a global footprint that lends itself to serving operations in varied geographies.
In this case, the pilot was conducted for a single operating unit, which comprised a total of approximately 3 million end-users. A handful of scenarios were developed, which based on approximately 10 real-time data sources. The event processing engine and in-memory cache were hosted on a single 8 core 64 GB VM. A second VM of equal size was provided to provide high availability.
The field of Streaming Data is evolving rapidly, and is now capable of providing a real-time, stateful view of customers for real-time engagement. Amazon AWS provides a cost effective and capable platform, which can be augmented with solutions such as EVAM to support complex business driven scenario recognition for advanced customer engagement.