Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Matching Modern Databases With ML and AI

DZone's Guide to

Matching Modern Databases With ML and AI

You can't do AI without machine learning. You also can't do machine learning without analytics, and you can't do analytics without data infrastructure.

· Database Zone ·
Free Resource

Compliant Database DevOps and the role of DevSecOps DevOps is becoming the new normal in application development, and DevSecOps is now entering the picture. By balancing the desire to release code faster with the need for the same code to be secure, it addresses increasing demands for data privacy. But what about the database? How can databases be included in both DevOps and DevSecOps? What additional measures should be considered to achieve truly compliant database DevOps? This whitepaper provides a valuable insight. Get the whitepaper

Machine learning (ML) and artificial intelligence (AI) have stirred the technology sector into a flurry of activity over the past couple of years.

However, it is important to remember that it all comes back to data. As Hilary Mason, a prominent data scientist, noted in Harvard Business Review,

You can't do AI without machine learning. You also can't do machine learning without analytics, and you can't do analytics without data infrastructure.

Over the last year, we assembled a number of blogs, videos, and presentations on using ML and AI with MemSQL.

MemSQL With ML and AI

As a foundational datastore, MemSQL incorporates machine learning functions in one of three ways:

  1. Calculations outside the database
  2. Calculations on ingest
  3. Calculations within the database

Read an overall summary in our post Machine Learning and MemSQL from Rick Negrin.

Outside the Database

For integrating ML and AI outside the database, two popular methods are integrating with Spark and TensorFlow.

For Spark, MemSQL offers an open-source MemSQL Spark Connector, which delivers high-throughput, bidirectional, and highly parallel operations from partition to partition. This connector opens up unlimited ML and AI possibilities that can be combined with a scalable, durable datastore from MemSQL.

One example of this integration is real-time machine learning scoring. A stereotypical pipeline might be:

IIoT Collection > Kafka > Spark > MemSQL > Queries

For example, some popular statistical software packages allow export as PMML, the predictive machine learning markup language. These or similar models can be exported into Spark, or even the database itself, to score incoming data in real-time. From there, the datapoint and the score of the model on the datapoint can be persisted together for easy analysis.

MemSQL PowerStream is a specific example of combining machine learning with a database to score data in real-time and predict the likelihood of an equipment failure.

An interactive demonstration using MemSQL PowerStream simulates 200,000 wind turbines sending sensor information at a rate of approximately two million inserts per second to MemSQL. From there, the user interface shows live status on real-time information, and ML scoring predicts the likelihood of turbine failures.

Read more in our article IoT at Global Scale: PowerStream Wind Farm Analytics With Spark.

TensorFlow and the results of applying machine learning models in TensorFlow are shown in the the video Scoring Machine Learning Models at Scale From Strata New York.

TensorFlow is often used in real-time image recognition. This presentation and video from Spark Summit Dublin 2017 highlight the popular use case.

ML on Ingest

Another area to run machine learning is as the data arrives in the database. MemSQL enables this with native pipelines from Kafka, including exactly-once semantics, a critical capability for deduplication in event-driven pipelines. Further, the pipeline ingest capability includes the option for executing a custom transformation.

A typical real-time data pipeline in this scenario might be:

Kafka > MemSQL > Query/Visualization

MemSQL UI for custom transformations:

Within the Database

The third area for ML integration with databases is within the database itself. This occurs with a couple of secret ingredients.

The first enabler for machine learning within a database is extensibility or, more specifically, the inclusion of stored procedures, user-defined functions, and user-defined aggregates. These functions allow for customizations that can quickly and easily execute elements of an overall machine learning pipeline.

Another enabler is the inclusion of popular algebraic operations such as DOT_PRODUCT, which can take two vectors and quickly compare them for similarity. DOT_PRODUCT can also easily compare one vector against hundreds of millions of vectors to deliver a degree of similarity.

Thorn, an organization dedicated to ending child sex trafficking and the sexual exploitation of children, relies on image recognition to help fulfill its mission.

Through the use of DOT_PRODUCT within MemSQL, Thorn was able to dramatically improve its real-time image recognition capabilities. Read more on
An Engineering View on Real-Time Machine Learning, or watch the video from AWS re:Invent 2017, Business and Life-Altering Solutions Through AI and Image Recognition.

Beyond image recognition, DOT_PRODUCT has applicability in a wide range of use cases.

For example, we demoed k-means clustering on YouTube tags within a presentation from Gartner Catalyst 2017, The Data Warehouse Blueprint for ML, AI, and Hybrid Cloud. Skip to Slide 23 for the demonstration section.

Another example comes in this presentation from Gartner Data and Analytics 2018 in Texas, Building a Machine Learning Recommendation Engine in SQL. Skip ahead to Slide 48 for the demonstration.

Additional Resources

If you'd like to read more, feel free to check out MemSQL 6 Product Pillars and Machine Learning Approach. This post includes more on:

  • Built-in machine learning functions

  • Real-time machine learning scoring

  • Machine learning in SQL with extensibility

And if you would like to see where 1,600 IT professionals think the industry is headed with machine learning and artificial intelligence, see the 2018 Outlook: Machine Learning and Artificial Intelligence Survey. Developed in conjunction with O'Reilly Media, this survey helps shape the general sentiment and industry trends around ML and AI.

Compliant Database DevOps and the role of DevSecOps DevOps is becoming the new normal in application development, and DevSecOps is now entering the picture. By balancing the desire to release code faster with the need for the same code to be secure, it addresses increasing demands for data privacy. But what about the database? How can databases be included in both DevOps and DevSecOps? What additional measures should be considered to achieve truly compliant database DevOps? This whitepaper provides a valuable insight. Get the whitepaper

Topics:
ai ,database ,machine learning ,data science ,memsql

Published at DZone with permission of

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}