Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Let's Talk About Natural Language Processing

DZone's Guide to

Let's Talk About Natural Language Processing

Read on in order to gain insight into natural language processing and what it means as well as look at NLP libraries and NLP services.

· AI Zone ·
Free Resource

Insight for I&O leaders on deploying AIOps platforms to enhance performance monitoring today. Read the Guide.

Image title

"Handle them carefully, for words have more power than atom bombs." — Pearl Strachan Hurd

I am fairly sure that when Pearl Strachan said those words, she did not have text mining & analytics in mind, but nevertheless, her words still ring true, especially in today's world of conversational AI & NLP (Natural Language Processing)!

The interest in conversational bots (chatbots) and digital virtual assistants is growing and growing fast. Enterprises are increasingly getting eager to explore bot-enabling their business processes. Though, as with all fields of AI, conversational AI is evolving at a rapid pace, and the focus is shifting from purely NLP led chatbots to more reliance on rich messages (cards, and buttons in the chat interface — offering higher accuracy etc.) — a trend underlined by Wit.ai, acquired by Facebook in 2015, sunsetting their Bot Engine in early 2016.

But nevertheless, Natural Language Processing still remains a very pertinent and key component of any conversational AI setup, so let's talk about:

  • What is Natural Language Processing and its potential uses
  • Some NLP Libraries for Developers
  • Services/platforms/tools to setup & use NLP for chatbots

What Is Natural Language Processing

Natural Language Processing, as the name suggests, is essentially a field of computer science (more pertinently artificial intelligence) that deals with processing human language to be able to analyze, understand, and derive information from it (so that subsequent actions can be taken).

NLP Libraries - Open Source Libraries

There are many libraries available that make it easier for developers to build the NLP applications. Here is a list of a few that we have used and some of the prominent ones:

NLP Services & Tools for Chatbots

Circling back to our original point about the use of NLP for Chatbots, if you don't want to delve into libraries & code, there are many NLP Tools and Services available.

While we have mostly used Microsoft LUIS ( Language Understanding), some of the other popular ones are Dialogflow ( Api.ai post-acquisition by Google), Amazon Lex, Wit.ai (Facebook), etc.

Some of them might be more suited towards building chatbot systems — Amazon Lex for instance — which provides an integrated setup for multi-turn conversations with provision for prompts for filling the relevant 'slots' towards fulfilling the 'intent'. Microsoft LUIS, on the other hand, focuses on the classical NLP aspects of extracting the 'intent' and 'entity' from the utterance. Others like Wit.ai, Dialogflow etc. provide a combination if you will.

Every tool/service has its own terminology but essentially in terms of base NLP:

  • Utterance is the spoken or the typed phrase that the system receives from the user that invokes the intent. For instance 'Book a flight from London to New Delhi for tomorrow'.
  • Intent - The intent over here is booking a flight.
  • Entities are the specifics of the intent included in the utterance that help us take a particular action. For instance 'London' and 'New Delhi' are Locations (and could be made hierarchical as $Location -> $FromLocation and $ToLocation, for instance) and 'tomorrow' indicates $datetime.

Depending on what the particular focus is for your solution & the relevant use cases, and also maybe what are the other tools you are already using, you can choose the relevant tool or service. I am just jotting down some observations that you might find useful:

  • Modular or tightly coupled: The degree of modularity varies across these tools. For instance, with Microsoft, its a fairly modular system where Microsoft LUIS assists with NLP. All the chatbot related aspects (fulfillment calls, channel integration, etc.) sit with the Bot Framework, and the Bing Speech API converts spoken language requests into text. Amazon Lex, on the other hand, has a speech recognition engine in it that has been trained on telephony audio (8Khz sampling rate). And while Amazon Lex provides access to Enterprises Applications through Amazon Lambda, Dialogflow, on the other hand, allows for webhooks for fulfillment (with or without slot filling).
  • Analytics & Monitoring: Dialogflow provides a ready dashboard, which provides usage data and the NLU data in terms of intent sessions, count, exit percentage etc. Microsoft Bot Framework that integrates with Azure App Insights (requires configuration) for Bot and Usage analytics, and in case of Amazon Lex, it can be integrated with Amazon Cloudwatch (monitoring dashboard pre-integrated with Amazon Lex) for metrics and monitoring. Wit.ai, like all others, can be integrated with third-party analytics platforms.
  • Channels Supported: While both Dialogflow & Microsoft (Bot Framework) integrate with 14-15 odd channels including your webpage, Wit.ai allows for integration with facebook messenger, your webpage, and your app, and for others, one would need to use third-party API integration platforms like APIANT etc. Amazon Lex supports only 4 — Facebook, Kik, Twilio SMS, and Slack.
  • Languages Supported: Most of the platforms support quite a number of languages — LUIS supports 13 languages, Dialogflow supports 15 root languages, Wit.ai supports I think 5+ key languages and Amazon Lex currently supports only US English — which might just be all that you need.
  • Ecosystem: Microsoft LUIS, Amazon Lex, and Dialogflow hugely benefit from the developer and product ecosystems of their parent companies. For instance, if you are already using Bot Framework, maybe LUIS is the obvious choice, or if you are on Amazon Connect for your call center, then Amazon Lex is probably a no-brainer.

Each tool has its own set of unique features as well. For instance, Dialogflow has the option of SmallTalk to provide responses to casual conversations, and Wit.ai provides an element of sentiment analysis (though rather crude and heavily dependent on the number of examples you provide), etc.

But in terms of base NLP, all of them, more or less, are similar, allowing for features like pre-built and custom defined entities. Whilst one has a 'list' of entities and a 'phrase list' (similar but not synonyms) in Microsoft LUIS, we have essentially the same things in 'keywords' and 'free-text' for entities in Wit.ai. Dialogflow has prebuilt agents, while Microsoft LUIS also has pre-built domain models.

At some point, I will try and have parallel implementations and see how they fair on the F1 scores (geometric mean of precision and recall), but in the meantime, I do hope this has helped you gain a slightly better understanding of NLP. Do leave your comments & thoughts, including any aspects that I might have missed. I will be happy to incorporate them.

Disclaimers: The above post in no way claims any copyright to any of the images or literature presented.

TrueSight is an AIOps platform, powered by machine learning and analytics, that elevates IT operations to address multi-cloud complexity and the speed of digital transformation.

Topics:
nlp ,natural language processing ,artificial intelligence ,chatbot ,enterprise ai ,enterprise ,machine learning

Published at DZone with permission of

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}