Big Data Trends: Real Time, Stateful Customer Engagement

Surveys indicate that enterprise adoption of the public cloud will accelerate in the coming years, as micro service and cloud-native application architectures become mainstream. The move toward Continuous Integration and Agile software delivery is being mirrored in enterprise data architectures, with growing focus on in-memory and real-time data processing.

The Evolution of Enterprise Data Architectures

Enterprise data strategies have been littered with cost and failure in earlier decades.   Data warehousing was known for high costs and failed projects during the 90’s.  More recently, Gartner surveys indicate that Big Data strategies based on Hadoop and map reduce have met similar outcomes, with 49 percent of responses reporting they are “struggling to get value from Hadoop” (Gartner 2015 Hadoop Adoption Survey).   

There are several reasons for these problems.   First, the technologies employed in past efforts have been overly complex and inflexible.   Problems were compounded by the lack of engineering expertise, particularly when matched with immature open source projects.  Finally, many big data initiatives lacked direction and were often justified to record data in the hope of realizing business value.  The result has been heavy investment but often with little impact to the business.

Fortunately, the Big Data landscape is moving to a more practical focus, with an emerging track record of success and return-on-investment.   We’re shifting from systems of record (e.g., “what were last quarter's sales of a product X”) to systems of real-time insight and action (e.g., “what an individual customer likely to buy, and what form of engagement will best influence their behavior?”).  This shift in focus creates and sustains enterprise value and the business brand, by emphasizing customer experience.   Forward thinking businesses are employing Streaming Analytics solutions today to drive value and competitive advantage. For this article, we define “real time” as customer engagement actions delivered within 50 milliseconds of a customer event (fast enough to ensure a web offer is customized to the customer while browsing a web page). 

Real Time, Stateful Customer Engagement

This next phase in enterprise big data analytics has developed over the past decade, with advances in distributed in-memory cache, real-time data ingestion, and cloud-based services such as AWS Kinesis, ElasticSearch, and others.  Robust data integration libraries have been developed to streamline project implementation, with connectors to virtually any enterprise data source.   The resulting applications are diverse, and tailored to the needs of retailers, financial service providers, network service providers, and manufacturers.  Point solutions such as Campaign Management are in decline, as “enterprise-wide” event processing platforms are adopted to address the full range of customer engagement.

Retailers are rushing to implement mobile apps that interact and shape the customer experience.   Offers will reflect not only suitable products, but preferred colors and sizes, shaped by past purchases and current events and social trends. Customers will be recognized when they are nearby, and can be incented to stop by the store. Web sessions will be personalized, with items presented to compliment or accessorize.   In short, a combination of in-store, online, and mobile interaction will shape a more intimate dialogue with customers, even while interactions become increasingly digital. 

A wireless operator needs an effective way to engage with prepaid customers.  Customers cycle continuously from renewed, to approaching renewal, to expired and becoming “churn.” After implementing a real-time view the operator can incent renewal, prompt customers to use their prepaid service, and take extra steps to avoid customers becoming churn.    Since implementing a real-time engagement syse revenue is up over 5% per account, and churn has dropped by 8%. The metrics continue to improve as the engagement strategies continue to be improved.  

Navigating the Journey to Real Time, Stateful Customer Engagement

Most enterprises will struggle to be effective in building solutions based on open source projects, such as Apache Spark and others.   Open source projects are by nature a “project” and simply require more engineering talent than most organizations can apply.   The better route for most organizations is to partner with an expert in the field, who has demonstrated the engineering talent to integrate leading technologies to deliver affordable, easy implemented solutions. 

“Business ready” solutions should integrate with disparate operational systems, including existing Campaign Management Systems, Rules Engines, and Technical Event Collection.   Existing systems can be integrated into a new “enterprise event hub.”   Actions triggered by existing systems will flow through unchanged, but new scenarios can be added, and the enterprise gains a single view on overall customer engagement. 

Another aspect of a “business ready” customer engagement solution is support for easily defined and implemented scenarios and actions.   A Scenario designer enables business people to identify combinations of events that trigger actions, without relying on software developers.  Scenarios can combine events, and importantly non-events, with a time window, with globally defined constraints to prevent a customer being bombarded when multiple scenarios are triggered.   Scenarios also should be flexible to not only trigger actions, but provide input to other scenarios.    The result is a lower-cost, more responsive, and faster-learning customer engagement program.  Example scenarios could include:

• A wireless customer experiences 3 dropped calls within 12 hours.
• A customer recently enrolled in a new banking service, but has not utilized the service (a non-event). 
• A wireess prepaid customer is nearing expiration, and their device has left the network, suggesting a possible competitive switch.   

An example Visual Scenario designer combines business events with flows to define a scenario. 

Finally, a “stateful” system incorporates a complete view of the customer at any point in time.  This is accomplished with a “customer profile” that is derived from a data warehouse and stored in-memory for real-time access.   The online customer profile can include 100 or more columns of data (name, age, address, etc.), to provide context for engagement.    Such a customer profile occupies a minimal memory footprint and is incorporated into scenario decision-making without being “queried,”  and support the delivery of actions within 30-50 milliseconds of triggering events. 

Conclusion

There is an accelerating trend toward enterprise-wide, real time, stateful customer engagement, with engagement delivered to end-users within 50 milliseconds.  These systems integrate with any event source, and integrate with existing campaign management and decision rules based systems, and can be either on-premise or used in the public cloud.   Customer engagement is based on scenarios which include multiple events, non-events, and time windows, and logic to support potentially hundreds of scenarios with prioritization and constraints.   Scenario designers are tailored to business users to define and implement scenarios.  Finally, these systems incorporate a “stateful” view of customers derived from traditional databases that are kept online for real-time engagement.  

The capabilities outlined in this article are provided in varying degrees by a range of vendors, and my firm, EVAM, is a leader in real-time customer engagement. Over 40 enterprises rely on EVAM systems globally, interacting with over 200 million end users in retail, finance, network service providers, and manufacturers, with real-time customer engagement.   

Where Data Warehousing and many Big Data projects have struggled to deliver business value, real time customer engagement systems are typically simpler to implement, high ROI, and low-risk.   Implementation is simplified due to focusing on a subset of total data, with well-identified business events, and easily updated scenarios and actions. Results are immediate and enduring.  

To learn more about real time stateful customer engagement, and explore how these systems can be integrated with Spark and existing enterprise systems, talk with one of our consultants at EVAM at http://evam.com/streaming-analytics-check-up/