5 Minute ML: Chatbot (QnA) Demystified

From a Natural Language Processing perspective, a chatbot normally consists of many parts — small talk, QnA (question and answer — included intent prediction and entity extraction), context handling (user, session etc.), question completion, personalization, sentiment analysis, and so on. Not every chatbot needs all the above-mentioned capabilities. You can have just small talk and QnA and create a chatbot or assemble small talk, QnA and personalization and handle most of the user’s queries. But every chatbot needs QnA.

In this article, we focus on only the Q part of QnA. This is the most complex part of any chatbot framework and needs expertise in Machine Learning, Natural Language Processing and in some cases Deep Learning. Intent Prediction and Entity Extraction are 2 major components of the Q part, which helps the system understand the user query in terms of the answer repository. Answer repository is the domain for which we have built the chatbot. The answer repository can be as simple as a set of FAQs or an excel file or as complex as a database, a SAP system, or a knowledge base.

Here is a brief explanation of the components — Intent Prediction is used to figure out the objective (intent) of the user query. Here we are assuming that every user query is interested in knowing about only one type of thing from the answer repository. Trying to know about 2 or more types of things from the answer repository is out of scope. Most chatbot frameworks support answering only one type of thing. Entity Detection is used to figure out any constraint that the user query has. For example, for a question “Who is the captain of Argentina football team,” the Intent is captain and the entity is Argentina football team. Such a question can be asked to a FIFA database.

One thing that must be noted here is that the Q part need not always be a question. It can be a statement like “Order Pizza of Regular size with Mushroom and Tomato toppings.” Here clearly the intent is Ordering Pizza and entities are “Regular” of type size and “Mushroom” and “Tomato” of type toppings. Please keep in mind that intent and entities are always pre-defined based on the answer repository. As the type of data stored in the answer repository (metadata) changes, the intent and entity types will also change accordingly. This pre-determination will be usually done by the admin who will configure the system.

Intent Prediction

This is a Machine Learning module that tries to figure the objective of the user’s question. In some scenarios, intent is also referred to as the expected answer type (EAT). For brevity purposes, we will create a very simple intent prediction module with an example for a telecom company. Let’s say that the chatbot wants to help users out with very basic operations like balance inquiry, prepaid packs, and recharge.

Before we code the intent prediction module, we need to do the following:

• Define a set of intents.
• Create training data for our Machine Learning module. Training data is basically a set of questions or statements similar to what a user will ask the system. We need not cover all the types of questions that a user can ask, a representative set would suffice.

For the telecom chatbot, we will define intents to be the same as operations. So the intents will be “Balance Inquiry,” “Prepaid packs,” and “Recharge Order.”

Now that the intents have been created, we will create the training data as in the table below.

We have included only 4 statements per intent. Ideally, you should have a lot more utterances per intent especially if there is a chance of overlap between utterances of 2 or more intents. Also, you can include commonly used synonyms in the utterances.

The following code can be used for intent predictions. The code is in Python and we will be using the Scikit-learn library for machine learning. Other Python modules will be used for processing the data. The code is mostly about using Random Forest Classifier to do classification on the text data. Details of the code will be discussed in upcoming articles.

Code

from sklearn.ensemble import RandomForestClassifier
from sklearn.feature_extraction.text import TfidfVectorizer           
import pandas as pd
import numpy as np
import re

def preprocessing(userQuery):     
    letters_only = re.sub("[^a-zA-Z\\d]", " ", userQuery)
    words = letters_only.lower().split()                  
    return( " ".join(words ))

#read utterance data from a csv file
train = pd.read_csv('Sheet1.csv')
query_features = train[‘Utterances’]

#create tfidf
tfidf_vectorizer = TfidfVectorizer(, ,ngram_range=(1, 1), )

new_query = [preprocessing(query) for query in query_features]
features = tfidf_vectorizer.fit_transform(new_query).toarray()

#create random forest classification model
model = RandomForestClassifier(, )
model.fit(features, train['INTENT'])

#intent prediction on user query
userQuery = "what are the prepaid packs that are available "
userQueryList=[]
userQueryList.append(preprocessing(userQuery))
utfidf = tfidf_vectorizer.transform(userQueryList)
print(" prediction: ", model.predict(utfidf))

Entity Detection

Entity Detection can be straightforward matching from a list of pre-defined entities or by using Named Entity Recognition (NER) modules. If it is a pre-defined list of entities, we can categorize the entities according to the intents and then use a simple keyword-based search to detect the entities.

If we are not sure about the entities that may be mentioned in the user query, it is good to use one of the NER modules provided by popular NLP libraries. We can use one from the following NLP libraries — SpaCy, Stanford NLP, OpenNLP, ClearNLP, AllenAI, or Cloud Natural Language API by Google.

Following is the code in python for Entity Detection using SpaCy:

Code

import spacy
# use English language model
ner_model = spacy.load('en') 
ent_data = ner_model("Alphabet is a new startup in China")
print(ent_data.ents)

That’s all. Thank you for your time. 

Note: This is the first article in a series of 5 Minute ML. The purpose of this series of articles is to explain the basic concepts of Machine Learning and its applications in various domains in small posts, so small that it can be read and understood in around 5 minutes. Hence, the title.