Five Reasons Why BAM and CEP Have Failed You

[This article was written by Maneesh Joshi]

Back in 2005, the concept Business Activity Monitoring (BAM) was coined by Gartner to capture the requirement of visibility into business operations. BAM delivered this visibility through aggregation and summarization of information around business activities. The BAM solution would capture event data, aggregate them into a single store, apply some context on top of the data, and then deliver dashboard-based visibility to business operations managers and sometimes executives.

A few questions that a typical BAM solution would answer for the business operations manager at a mortgage lender were:

1. How long does the typical loan application take from submission to approval?
2. Where is the most time spent in the loan application workflow?
3. How many calls does the average call center rep answer per day?

Unfortunately, BAM never delivered on it’s promise. I can think of four challenges that made the BAM solution fall short.

1. Vendor-Focused Solutions

The first challenge with this approach to BAM was that the solutions were typically built by the large enterprise software vendors that offered it as a complement to their own big and wide technology stacks. This approach made the offerings very vendor-centric and often didn’t cover the heterogeneous set of technologies that most businesses owned. The visibility hence was rather restricted to the large vendor stacks and created information silos and narrow visibility.

2. Too Much Customization

The second challenge arose as a result of the fact that no two business processes are alike. There were very few commonalities between business processes for the vendors to build repeatable and scalable pre-packaged dashboards around. In order to build a solution that worked for these processes, heavy customization and services were necessary. For the solution to be meaningful and provide the right kind of visibility, the customization often included intrusive code changes to the application in order to raise business events, whether it was Java code in the Java tier, or the SQL packages in the database tier.

3. No Real-Time Visibility

The third challenge was from the lack of real-time nature of the visibility that the BAM technology delivered. By the time the operational data was collected from the sources, aggregated into a database, applied context on, analyzed, and presented into a dashboard, the visibility provided was already stale. In order to address this staleness problem, the notion of complex event processing (CEP) came into being. CEP technologies were purpose built to correlate events and detect patterns across event streams in real-time.

4. Lack of Business Context

Despite CEP attempting to solve the staleness problem, it introduced a problem of its own. The events processed by CEP are typically lightweight events and low on context richness. When CEP would attempt to correlate events to detect patterns across disparate streams, it lacked the business context. This lack of context made the solution less effective as no meaningful pattern matching or correlation could be done without the context. The business context needs to be captured at the source when the events are being captured in order to make the correlation relevant.

5. Premature Technology Stack

The last and main blocker was that the underlying technology stacks and the surrounding applications were not mature enough to support the vision. For instance, the relational databases that couldn’t capture unstructured events, or the inability to crunch massive volumes of data in real-time at low latencies while applying the context severely restricted the vendors from delivering on the promise. The packaged applications were very monolithic with very little extensibility, which resulted into implementing intrusive code changes to raise business events.

To be fair, the vision for BAM and CEP was spot on then and is spot on today. The vision was clearly ahead of its time and the problems it promised to solve very real. Who doesn’t want visibility into their business operations?! In fact BAM and CEP become all the more important is today’s day and age of software-defined businesses.

In my next blog post, I will discuss a new approach to BAM. I will also review as to how the technology and application stacks have evolved, and why you should consider revisiting your opinions about BAM and CEP use cases.