Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Why Big Data and Sentiment Analysis Are a Match Made in Heaven

DZone's Guide to

Why Big Data and Sentiment Analysis Are a Match Made in Heaven

A high-level look at how sentiment analysis and big data sets can be brought together to better inform decisions. Read on to learn more!

· Big Data Zone ·
Free Resource

The open source HPCC Systems platform is a proven, easy to use solution for managing data at scale. Visit our Easy Guide to learn more about this completely free platform, test drive some code in the online Playground, and get started today.

Making Sense of Opinions at a Massive Scale

Since the dawn of modern business, brands have searched for ways to truly understand how the public feels about them. Focus groups, opinion pieces, surveys — all these things only help to understand the feelings of those who choose to participate, not the entire population. But we can do this for everyone, and we can do it in a scalable way.

Enter sentiment analysis. Delivered as a service or through an API, this cognitive service can gauge the emotion of massive sets of data, then reflect the outcome in useful, and sometimes deeply entertaining, ways.

Let’s take a closer look at the two ways to do sentiment analysis: real-time and static.

Real-Time Sentiment Analysis

Social media giants, most notably Twitter, give people a soapbox to state their impassioned support or hatred (and sometimes, if we’re lucky, something in between) and opinions on anything and everything. What this provides for us is a giant set of ever-evolving data on how people feel.

To analyze and derive insights from each individual Tweet, you’d need hundreds of thousands of people monitoring every Tweet, and consistently rating the sentiment of each one. That’s obviously labor-intensive and subject to the implicit and explicit biases of individually rating tweets. Obviously, this is an unlikely strategy for delivering a clean read.

The trick is to combine big data with the sentiment analysis API. This is achieved by hooking into a data firehose, which is basically an open spigot of data flowing in real-time. Working off our example above, the Twitter Firehose streams every Tweet sent over the platform as soon as it’s published. In order to create a usable dataset, you simply filter that firehose based on keywords or topics, and from there, you can run the sentiment analysis on it.

PubNub recently did this for the 2018 Midterms. We were able to filter Tweets related to the upcoming election and gauge the emotion of that Tweet on a scale. To visualize the analysis, we published it in real-time to a live dashboard, making it easy for anyone to see up-to-the-moment updates, as well as trends over time.

Static Sentiment Analysis

When it comes to big data sentiment analysis, it doesn’t always have to be real-time. If the up-to-the-millisecond analysis isn’t important, you can run sentiment analysis on data that’s stored in a database. This gives you the opportunity to better understand historical sentiment trends, and use that analysis for better decision making.

For example, a customer support team could analyze their entire database of transcripts to get an idea of how their customer base is feeling about support. They could derive if the customer finished the conversation happier than when they began. They can even slice up the analysis based on demographic, customer type, age, gender, and more.

Hotels.com is undertaking just such a project and is now able to provide more locally relevant personalized travel experiences by analyzing the sentiment of customer reviews. This makes reviews more accurate, not relying solely on the 5-star ratings, but rather on how the user really felt, as revealed by the language they use. This also allows Hotels.com to discover new insights and create unique experiences by understanding what’s available, and valued, at each property.

Technologies Available Today

The big dogs in the cloud game offer sentiment analysis functionality as a part of their artificial intelligence and machine learning suite of services. There are also some independent players. All of them are able to consume large sets of data. A few examples include:

Looking Forward

Sentiment analysis brings functionality that will be massively beneficial to brands and businesses going forward. The services are incredibly accessible through APIs, and any development team can quickly get a scalable solution up and running. Combined with big data, we have a new, accurate way to gauge the sentiment of massive sets of data. And all this together will answer the (sometimes scary) question — how do they actually feel about us?

Managing data at scale doesn’t have to be hard. Find out how the completely free, open source HPCC Systems platform makes it easier to update, easier to program, easier to integrate data, and easier to manage clusters. Download and get started today.

Topics:
sentiment analysis ,cognitive computing ,machine learning ,big data

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}