Taming the Data Problem and Accelerating AIOps Implementations With RDA

What is Robotic Data Automation (RDA)

Robotic Data Automation (RDA) is a new paradigm to help automate data integration and data preparation activities involved in dealing with machine data for Analytics and AI/Machine Learning applications. RDA is not just a framework, but also includes a set of technologies and product capabilities that help implement the data automation.

RDA enables enterprises to operationalize machine data at scale to drive AI and analytics driven decisions.

RDA has broad applicability within the enterprise realm, and to begin with, CloudFabrix took the RDA framework and applied it to solve AIOps problems — to help simplify and accelerate AIOps implementations and make them more open and extensible.

RDA automates repetitive data integration, cleansing, verification, shaping, enrichment, and transformation activities using data bots that are invoked to work in succession in “low-code” data workflows or pipelines. RDA helps to move data in and out of AIOps systems easily, thereby simplifying and accelerating AIOps implementations that otherwise would depend on numerous manual data integrations and professional services activities.

Why RDA is Needed?

Artificial Intelligence for IT Operations (AIOps) requires processing vast amounts of data obtained from various hybrid IT data sources, that are spread across on-premise, cloud, and edge environments. This data comes in various formats and delivery modes. Additionally, results and outcomes of such data processing need to be also exchanged with other tools in the IT ecosystem (Ex: ITSM/Closed-loop automation/Collaboration Tools and BI/Reporting tools).

All of this requires integrating, ingesting, preparing, verifying, cleaning, transforming, shaping, analyzing, and moving data in and out of AIOps systems in an efficient, reusable, and scalable manner. These essential tasks are most often overlooked in AIOps implementations and cause significant delays and increase the costs of AIOps projects.

Challenges

Let us understand what are some of the key challenges in data preparation & data integration activities when implementing AIOps projects.

• Different data formats (text/binary/json/XML/CSV), data delivery modes (streaming, batch, bulk, notifications), programmatic interfaces (APIs/Webhooks/Queries/CLIs)
• Complex data preparation activities involving integrity checks, cleaning, transforming, and shaping the data (aggregating/filtering/sorting)
• Raw data often lacks application or service context, requiring real-time data enrichment bringing in context from external systems.
• Implementing data workflows require a specialized programming/data science skillset
• Changes in source or destination systems require rewriting/updating connectors

Traditional Approach of Data Handling in AIOps:

In the traditional approach, AIOps vendors provide a set of out-of-the-box integrations and once you connect the AIOps solution to your data sources, you are now pretty much at the mercy of how your data gets utilized, processed for producing results and outcomes:

• The black-box approach of data acquisition, processing, and integration.
• Use cases and scenarios limited to what the platform supports.
• Integrations mostly predefined/hardcoded limiting reuse.
• Complex scripting modules or cookbooks requiring specialized/programming skills (Javascript, Python, etc.).
• Difficult to bring in external integrations for intermittent data processing (ex: enrichment).
• Difficult to access data in a programmatic way for complementary functions (ex: data access for scripting, reporting, dashboarding, automation, etc.).

These are all inhibitors to effective AIOps implementations by way of adding delays & costs (manual data prep/handling activities)

Need of the Hour: Robotic Data Automation for AIOps

Robotic Data Automation (RDA), A Key Enabler For AIOps 2.0

RDA automates DataOps, similar to what RPA did to automate business processes. RDA is an integral part of the AIOps platform that provides augmented data preparation and integration capabilities. RDA is both a data automation framework and a toolkit to accelerate and simplify all data handling in AIOps implementations.

Highlights

• Implement Low-code Data Pipelines using Data bots.
• Native AI/ML bots.
• CFXQL — Uniform Query Language.
• Inline Data Mapping.
• Data Integrity Checks.
• Data masking, redaction, and encryption.
• Data Shaping: Aggregation/Filtering/Sorting.
• Data Extraction/Metrics Harvesting.
• Synthetic Data generation.

Benefits

• Simplify and Accelerate implementation of AIOps use cases.
• Reduces time/effort/costs tied to data prep and integrations.
• Suitable for DevOps/ProdOps personnel (no need for data scientist skills).

Example Use Cases and Scenarios

• Log Clustering: Ingest app logs from cloud and on-prem, run ML models to cluster logs, push results to Kibana/CFX dashboards.
• CMDB Synchronization: Take the latest asset inventory from CFX and push it to CMDB.
• E-Bonding of tickets from partner/subsidiary ITSM to customer’s ITSM (Ex: BMC incidents to ServiceNow).
• Incident NLP Classification: Ingest tickets from ServiceNow, do NLP classification with OpenAI (GPT-3) and enrich tickets back in ServiceNow.
• Anomaly Detection: From Prometheus (or any monitoring tool), get historical CPU usage data for a node (Hourly). Apply regression and send a message on Slack with a list of anomalies as an attachment.
• Ticket Clustering: Take last 24-hrs incidents from ServiceNow, apply clustering on tickets, and push results to a new dataset for visualization in the CFX dashboard.
• Change Detection: Take baseline inventory of AWS EC2 VMs and compare against the current state to highlight unplanned changes.