Let's Talk About Natural Language Processing

"Handle them carefully, for words have more power than atom bombs." — Pearl Strachan Hurd

I am fairly sure that when Pearl Strachan said those words, she did not have text mining & analytics in mind, but nevertheless, her words still ring true, especially in today's world of conversational AI & NLP (Natural Language Processing)!

The interest in conversational bots (chatbots) and digital virtual assistants is growing and growing fast. Enterprises are increasingly getting eager to explore bot-enabling their business processes. Though, as with all fields of AI, conversational AI is evolving at a rapid pace, and the focus is shifting from purely NLP led chatbots to more reliance on rich messages (cards, and buttons in the chat interface — offering higher accuracy etc.) — a trend underlined by Wit.ai, acquired by Facebook in 2015, sunsetting their Bot Engine in early 2016.

But nevertheless, Natural Language Processing still remains a very pertinent and key component of any conversational AI setup, so let's talk about:

• What is Natural Language Processing and its potential uses
• Some NLP Libraries for Developers
• Services/platforms/tools to setup & use NLP for chatbots

What Is Natural Language Processing

Natural Language Processing, as the name suggests, is essentially a field of computer science (more pertinently artificial intelligence) that deals with processing human language to be able to analyze, understand, and derive information from it (so that subsequent actions can be taken).

• Since the corpus of human-generated text in the form of dialogues, text, books etc. is humungous, NLP algorithms are typically based on machine learning algorithms. Although, chatbots are probably what brought it into the mainstream media, but NLP has many other real-world applications, like Sentiment Analysis for understanding the voice of the customer, Automatic Summarization in post summarization on platforms like Reddit for instance, POS (parts of speech) tagging, Text Mining, and rather cool ones like OCR (Optical character recognition), Machine Translation, Speech Recognition, and many more.

NLP Libraries - Open Source Libraries

There are many libraries available that make it easier for developers to build the NLP applications. Here is a list of a few that we have used and some of the prominent ones:

• NLTK - Natural Language Toolkit: NLTK is a leading platform for python and supports classification, tokenization, stemming, POS tagging, parsing, and semantic reasoning functionalities.
• Stanford CoreNLP : Stanford CoreNLP integrates many of Stanford’s NLP tools, including the part-of-speech (POS) tagger, the named entity recognizer (NER), the parser, the co-reference resolution system, sentiment analysis, bootstrapped pattern learning, and the open information extraction tools.
• MALLET : Machine Learning for Language Toolkit : MALLET is a java based package for NLP that provides document classification, sequence tagging, topic modeling, numerical optimization, graphical models, and more.
• Apache OpenNLP: a machine learning toolkit that provides tokenizers, sentence segmentation, part-of-speech tagging, named entity extraction, chunking, parsing, co-reference resolution, and more.
• spaCy: SpaCy is an open-source software library for advanced Natural Language Processing written in the programming languages Python and Cython. It offers tokenization, named entity recognition, pre-trained word vectors, deep learning integration, one of the fastest syntactic parser, statistical models in multiple languages, and more.

NLP Services & Tools for Chatbots

Circling back to our original point about the use of NLP for Chatbots, if you don't want to delve into libraries & code, there are many NLP Tools and Services available.

While we have mostly used Microsoft LUIS ( Language Understanding), some of the other popular ones are Dialogflow ( Api.ai post-acquisition by Google), Amazon Lex, Wit.ai (Facebook), etc.

Some of them might be more suited towards building chatbot systems — Amazon Lex for instance — which provides an integrated setup for multi-turn conversations with provision for prompts for filling the relevant 'slots' towards fulfilling the 'intent'. Microsoft LUIS, on the other hand, focuses on the classical NLP aspects of extracting the 'intent' and 'entity' from the utterance. Others like Wit.ai, Dialogflow etc. provide a combination if you will.

Every tool/service has its own terminology but essentially in terms of base NLP:

• Utterance is the spoken or the typed phrase that the system receives from the user that invokes the intent. For instance 'Book a flight from London to New Delhi for tomorrow'.
• Intent - The intent over here is booking a flight.
• Entities are the specifics of the intent included in the utterance that help us take a particular action. For instance 'London' and 'New Delhi' are Locations (and could be made hierarchical as $Location -> $FromLocation and $ToLocation, for instance) and 'tomorrow' indicates $datetime.

Depending on what the particular focus is for your solution & the relevant use cases, and also maybe what are the other tools you are already using, you can choose the relevant tool or service. I am just jotting down some observations that you might find useful:

• Modular or tightly coupled: The degree of modularity varies across these tools. For instance, with Microsoft, its a fairly modular system where Microsoft LUIS assists with NLP. All the chatbot related aspects (fulfillment calls, channel integration, etc.) sit with the Bot Framework, and the Bing Speech API converts spoken language requests into text. Amazon Lex, on the other hand, has a speech recognition engine in it that has been trained on telephony audio (8Khz sampling rate). And while Amazon Lex provides access to Enterprises Applications through Amazon Lambda, Dialogflow, on the other hand, allows for webhooks for fulfillment (with or without slot filling).
• Analytics & Monitoring: Dialogflow provides a ready dashboard, which provides usage data and the NLU data in terms of intent sessions, count, exit percentage etc. Microsoft Bot Framework that integrates with Azure App Insights (requires configuration) for Bot and Usage analytics, and in case of Amazon Lex, it can be integrated with Amazon Cloudwatch (monitoring dashboard pre-integrated with Amazon Lex) for metrics and monitoring. Wit.ai, like all others, can be integrated with third-party analytics platforms.
• Channels Supported: While both Dialogflow & Microsoft (Bot Framework) integrate with 14-15 odd channels including your webpage, Wit.ai allows for integration with facebook messenger, your webpage, and your app, and for others, one would need to use third-party API integration platforms like APIANT etc. Amazon Lex supports only 4 — Facebook, Kik, Twilio SMS, and Slack.
• Languages Supported: Most of the platforms support quite a number of languages — LUIS supports 13 languages, Dialogflow supports 15 root languages, Wit.ai supports I think 5+ key languages and Amazon Lex currently supports only US English — which might just be all that you need.
• Ecosystem: Microsoft LUIS, Amazon Lex, and Dialogflow hugely benefit from the developer and product ecosystems of their parent companies. For instance, if you are already using Bot Framework, maybe LUIS is the obvious choice, or if you are on Amazon Connect for your call center, then Amazon Lex is probably a no-brainer.

Each tool has its own set of unique features as well. For instance, Dialogflow has the option of SmallTalk to provide responses to casual conversations, and Wit.ai provides an element of sentiment analysis (though rather crude and heavily dependent on the number of examples you provide), etc.

But in terms of base NLP, all of them, more or less, are similar, allowing for features like pre-built and custom defined entities. Whilst one has a 'list' of entities and a 'phrase list' (similar but not synonyms) in Microsoft LUIS, we have essentially the same things in 'keywords' and 'free-text' for entities in Wit.ai. Dialogflow has prebuilt agents, while Microsoft LUIS also has pre-built domain models.

At some point, I will try and have parallel implementations and see how they fair on the F1 scores (geometric mean of precision and recall), but in the meantime, I do hope this has helped you gain a slightly better understanding of NLP. Do leave your comments & thoughts, including any aspects that I might have missed. I will be happy to incorporate them.

Disclaimers: The above post in no way claims any copyright to any of the images or literature presented.