Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Rainbird and Microsoft Cognitive Services

DZone's Guide to

Rainbird and Microsoft Cognitive Services

Learn how we used Microsoft Cognitive Services and Rainbird to produce bots in Skype and Slack to enable Bank Account recommendation.

· AI Zone
Free Resource

During a three-day hack-a-thon in April 2017, we experimented with integrating a range of Microsoft Cognitive Services with the Rainbird Platform. Using a demo Knowledge Map built in Rainbird referred to as the Hamilton demo, we used Microsoft Cognitive Services to produce bots in Skype and Slack. The objective of the Hamilton demo is to enable Bank Account recommendation through rules generated by banking experts. More details on the Rainbird platform and the Hamilton demo are available through the links under the Resources section further down.

In addition, we deployed a Rainbird environment on the Azure cloud-hosted platform. A Rainbird environment consists of database instances, an inferencing engine, and the expert modeling tool. Each required element is available as a docker image and therefore simple to deploy.

Tools Used During Hack-a-Thon

  • Microsoft Bot Framework.
  • Microsoft Cognitive Services:
    • LUIS.
    • QnA Maker.
  • Rainbird Platform.

Challenge

Rainbird’s power is best demonstrated by taking a complex decision-making routine generally performed by teams of people and automating that decision-making process in a tool for others to use. Very often, a customer’s requirement encompasses this scenario along with a need to answer basic questions. Microsoft’s QnA Maker extracts frequently asked questions and answers, making them accessible through its API. Additionally, customers frequently ask about Rainbird bot integration; therefore, we have used Microsoft’s Bot Framework to provide a front end in both Slack and Skype. Finally, we integrated LUIS, Microsoft’s Natural Language Processing (NLP) tool, to identify the user’s free-text input intent from which the application routes requests through QnA Maker or Rainbird.

Tech Purpose
QnA Maker Solves simple question answer scenarios.
Rainbird Platform Complex decision making through extended consultation (multiple questions and answers).
Bot Framework Distribute implementation to various bot platforms.
LUIS

Identify user intent from free text entry.

Solution

See how we did it.

Overview

In summary, the user’s initial request is passed to LUIS to identify whether the request for information should be directed to QnA Maker or Rainbird. When QnA Maker is identified as the route, the API is contacted and the answer communicated back to the user. When Rainbird is identified as the route, an iterative Q&A consultation is conducted to solve the user’s original query. The consultation is directed to aid the interaction using the attributes of the median (Skype, Slack, etc.) as illustrated below:

The solution’s code is publicly accessible here.

Testing the Solution

See how we tested the solution with QnA Maker and Rainbird.

QnA Maker

In QnA Maker, we defined a handful of questions and associated answers as shown below:

Question Answer
Do you have business advisors? We have a team of qualified business advisors, ready to work with you on new or existing business ventures.
Do you have any cash machines? We have three 24 hour cash machines – ready to expel money at a moments notice.
How many counters do you have?

We have 5 general customer service counters that are all open during opening hours.

Rainbird

In Rainbird, we defined a single complex decision task to recommend a suitable bank account. You can ask “Please recommend me a bank account” or a variance of this request as LUIS will determine the intent. Our solution will then interact with Rainbird asking a series of questions before making a recommendation. The final recommendation is supplemented with a graphical representation of the path taken, the Rainbird Evidence Tree.

Try It Out

The Microsoft Bot Framework by default generates Web Chat and Skype implementations.
Click here to try out the final solution in the web chat.

Step-by-Step Guide

Here’s a quick explanation to assist reproducing the solution. To use the Microsoft services, you will need to sign up to each of them:

The solution requires a number of environment variables associated with the above services and Rainbird. When deploying your bot in the Azure Infrastructure, use the Azure portal to enter the required environment variables.

Variable Name Description
MICROSOFT_APP_ID Created during Bot registration at https://dev.botframework.com/bots/new.
MICROSOFT_APP_PASSWORD Created during Bot registration at https://dev.botframework.com/bots/new
RAINBIRD_EVIDENCE_TREE_HOST Rainbird host used to in Evidence Tree links
RAINBIRD_API_URL Rainbird API URL
RAINBIRD_API_KEY Rainbird API key
RAINBIRD_KMID Rainbird knowledge map ID
RAINBIRD_QUERY_SUBJECT Subject value of Rainbird query
RAINBIRD_QUERY_RELATIONSHIP Relationship used in the Rainbird query
LUIS_APP_ID LUIS account ID
LUIS_APP_KEY LUIS application key
QNA_APP_ID QnA Maker account ID
QNA_APP_KEY

QnA Maker application key

The application’s code is primarily in app.js. We’ve additionally developed Metaintent.js, a wrapper to handle the user’s input so we can determine the best tool to solve their question.

Firstly, in app.js, we create a server using restify.js as a means of easily capturing the REST API requests in our application and configure this to listen on the desired port:

var server = restify.createServer();
server.listen(process.env.port || process.env.PORT || 3978, function () {
    console.log('%s listening to %s', server.name, server.url);
}); 

Our setup continues with the Microsoft Bot Framework dependency, botbuilder, from which we can retrieve a universal bot instance used to navigate our conversation through dialogs.

var connector = new builder.ChatConnector({
    appId: process.env.MICROSOFT_APP_ID,
    appPassword: process.env.MICROSOFT_APP_PASSWORD
});

var bot = new builder.UniversalBot(connector);

Our attention then turns to configuring our connection to Rainbird. Below, we have informed the Rainbird API (URL) of our API key and the ID of the Knowledge Map, thus identifying ourselves and the knowledge map we wish to query.

var yolandaSession = new api.session(process.env.RAINBIRD_API_URL,
    process.env.RAINBIRD_API_KEY,
    process.env.RAINBIRD_KMID
);

Our Rainbird knowledge map is able to run a single query, which is ‘recommend a bank account’. Therefore, we have preconfigured the query JSON to this effect using environment variables.

var yolandaQuery = { subject: process.env.RAINBIRD_QUERY_SUBJECT, relationship: process.env.RAINBIRD_QUERY_RELATIONSHIP,
    object: null }; 

Our application’s initial setup is completed through the use of Microsoft’s universal bot connector, which instructs our application to listen for messages.

server.post('/api/messages', connector.listen()); 

Click here to learn more about the default message handler.

Besides a number of helper functions designed to handle the Rainbird interactions, there are seven bot.dialog()  handlers. The default message handler uses dialog to control the conversation flow. See here for more details.

The root dialog handler below decides on one of three courses of action based on the result of calling metaIntent.process(). If you recall we mentioned that the metaIntent.js was used to determine the path to take. Here we pass the user’s input through to our helper in metaIntent.js where the content is used to identify if we are asking a simple Question to be handled by Microsoft’s QnA Maker or if we are making a Rainbird request to recommend a bank account based on a series of questions and answers. As a fail safe, we check for neither action and pass back a ‘Sorry didn’t understand…’ message.

bot.dialog('/', function (session) {
    if (session.message.type === 'message') {
        var text = session.message.text;
        session.sendTyping();
        metaIntent.process(text, function(err, result) {
            if (!result) {
                session.send('Sorry, I didn\'t understand that.  How can I help you?');
            } else if (result.qnaResponse) {
                session.send(result.qnaResponse);
            } else if (result.intent === 'AccountComparison') {
                session.replaceDialog('/prestart');
            }
        });
    }
});

Before we run the code to determine the path to take, we execute the line session.sendTyping(). This line tells the bot to inform our user that we are typing.

To help capture variations to text indicating the need to run Rainbird’s ‘Recommend me a bank account’ request, we filter the user’s input through a trained LUIS implementation. To understand this, let's turn our attentions to the metaintent.js content. Beginning with the processTextLuis() function, we can see that LUIS is informed of the user’s input. If the response from LUIS exceeds a high enough score, we return the first element in the array received from LUIS (remember we only have one intent).

function processTextLuis(text, callback) {

    var url = 'https://westus.api.cognitive.microsoft.com/luis/v2.0/apps/' + L_APPID +
        '?subscription-key=' + L_APPKEY  + '&timezoneOffset=0.0&verbose=true&q=' +
        querystring.escape(text);

    superagent
        .get(url)
        .end(function(err, result) {
            if (err) {
                return callback(err);
            }
            var data = result.body;
            var result = null;

            if (data.intents && data.intents.length > 1) {
                var bestIntent = data.intents[0];
                if (bestIntent.score > 0.8 && bestIntent.intent !== 'None' && data.intents[1].score < 0.3) {
                    result = bestIntent;
                }
            }
            callback(null, result);
        });

}

The function exposed to our main app.js code attempts the LUIS intent match using the above function before evaluating the correct path to take. Assuming no error is returned from processTextLuis(), a result object informs our calling code to take the Rainbird route; otherwise, we attempt the QnA maker route.

function processText(text, callback) {

    processTextLuis(text, function(err, result) {
        if (err) {
            callback(err);
        } else if (result) {
            callback(null, result);
        } else {
            processTextQnA(text, callback);
        }
    });
}

Our processTextQnA() function contacts Microsoft’s QnA Maker with a formed URL, JSON body containing the user’s text, and our QnA Maker credentials (defined in environment variables). Upon the success of a QnA Maker match, the associated answer forms our result object and is passed back to our processText() function and subsequently our calling function in app.js. Again note we check for a high certainty before confirming a match response.body.score > 80.

function processTextQnA(text, callback) {
    var kbID = Q_APPID;
    var url = 'https://westus.api.cognitive.microsoft.com/qnamaker/v1.0' +
        '/knowledgebases/' + kbID + '/generateAnswer';

    var jsonBody = {'question': text};
    superagent
        .post(url)
        .set('Ocp-Apim-Subscription-Key', Q_APPKEY)
        .send(jsonBody)
        .end(function(err, response) {
            if (err) {
                return callback(err);
            }
            var result = null;
            if (response.body && response.body.score && response.body.score > 80) {
                result = {
                    'qnaResponse': response.body.answer
                };
            }

            callback(null, result);
        });
}

Returning to our calling code in app.js, we can see where we begin the more complex task of processing the Rainbird query by redirecting to our /prestart dialog.

metaIntent.process(text, function(err, result) {
    if (!result) {
        session.send('Sorry, I didn\'t understand that.  How can I help you?');
    } else if (result.qnaResponse) {
        session.send(result.qnaResponse);
    } else if (result.intent === 'AccountComparison') {
        session.replaceDialog('/prestart');
    }
});

The /prestart dialog checks for the existence of a running Rainbird conversation and redirects to our main rainbird loop dialog when one exists (/rbloop). Otherwise, we start a conversation by redirecting to the /start dialog.

bot.dialog('/prestart', function (session, args, next) {
    if (!session.privateConversationData.yolandaSession) {
        session.replaceDialog('/start');
    } else {
        session.replaceDialog('/rbloop');
    }
});

The /start dialog as the name suggests starts our Rainbird conversation. This is achieved through two steps. The first calls the Rainbird API start endpoint where we use our Rainbird credentials (API key and Knowledge Map ID) to authenticate. Next, we call the query endpoint to instruct Rainbird to start the ‘recommend me a bank account’ predefined query.

function startYolandaSession(cb){
    yolandaSession.start(function (err){
        if (err){
            return cb('Error calling start..' + err);
        }

        yolandaSession.query(yolandaQuery, function(err, response) {
            if (err) {
                return cb('Error running query..' + err);
            }

            cb (null, response);
        });
    });
}

Note that we’ve used a publicly available dependency to help abstract away much of the Rainbird Rest API syntax. See GitHub for details.

Once complete, our /start dialog redirects to the /rbloop dialog. Essentially, all Rainbird paths lead to the /rbloop dialog where we iterate through questions until Rainbird has sufficient knowledge to return result(s) relating to the original query — that is, unless Rainbird exhausts all possibilities and concludes it is unable to answer the user’s query.

bot.dialog('/start', function (session) {
    session.sendTyping();
    startYolandaSession(function (err, response) {
        if (err) {
            return session.send('Sorry there has been a problem starting a Rainbird session.');
        }

        session.privateConversationData.yolandaSession = yolandaSession.id;
        session.privateConversationData.yolandaResponse = response;
        session.replaceDialog('/rbloop');
    });
});

Finally, let’s examine the /rbloop dialog that controls much of the Rainbird discussion flow.

bot.dialog('/rbloop', [
    function (session) {
        if (session.privateConversationData.yolandaResponse.question){
            sendRBQuestion(session, session.privateConversationData.yolandaResponse.question);
        } else {
            sendRBResult(session, session.privateConversationData.yolandaResponse.result);
            delete session.privateConversationData.yolandaSession;
            session.endDialog();
        }
    },
    function (session, results) {
        var userAnswer;
        if (results.promptType === 1) { // number
            userAnswer = results.response;
        } else {
            userAnswer = results.response.entity;
        }

        session.sendTyping();
        yolandaResponse(session, userAnswer, function (err, response){
            session.privateConversationData.yolandaResponse = response;
            if (session.privateConversationData.yolandaResponse.question){
                session.replaceDialog('/rbloop');
            } else {
                sendRBResult(session, session.privateConversationData.yolandaResponse.result);
                delete session.privateConversationData.yolandaSession;
                session.endDialog();
            }
        });
    }
]).cancelAction('restart', 'No problem, how else can I help you?', {
    matches: /restart/i,
    onSelectAction: function (session, args, next){
        delete session.privateConversationData.yolandaSession;
        next();
    }
});

Before we look into the two functions in our array, let’s take a look at the last function, cancelAction(). Once a user has entered into a Rainbird conversation their input will always get redirected to the /rbloop dialog. With this function in place, if the user wishes to exit this line of conversation, they can do so by typing ‘restart’. The function removes the conversation data from the session and returns the message ‘No problem, how else can I help you?’. Now the user is free to enter into a new line of conversation.

Now let’s look at the functions in the array. In our first function, we check the conversation data to see if Rainbird has a question for the user. If so, then we call sendRBQuestion().

Our sendRBQuestion() function examines the question content received from Rainbird to help form a suitably illustrated question, i.e. a question with multiple choice options. We’ve used builder.Prompts() to present the question to our bot user.

function sendRBQuestion(session, rbQuestion) {
    if (rbQuestion.concepts && rbQuestion.concepts.length > 0) {
        var choices = rbQuestion.concepts.map(function(item) {return item.name});
        builder.Prompts.choice(session, rbQuestion.prompt, choices, { 'listStyle': builder.ListStyle['button'] });
    } else if (rbQuestion.dataType === 'number') {
        builder.Prompts.number(session, rbQuestion.prompt);
    } else if (rbQuestion.dataType === 'date') {
        builder.Prompts.time(session, rbQuestion.prompt);
    }
}

Our second function handles the bot user’s response to a question. Taking this response, we contact Rainbird again, this time via the yolandaResponse() function, which calls the response Rainbird API endpoint. Similarly to when we called the query, we consider the result of calling the endpoint to determine if we need to ask another question by redirecting to/rbloop  dialog or if we have a result to our original request.

function (session, results) {
    var userAnswer;
    if (results.promptType === 1) { // number
        userAnswer = results.response;
    } else {
        userAnswer = results.response.entity;
    }

    session.sendTyping();
    yolandaResponse(session, userAnswer, function (err, response){
        session.privateConversationData.yolandaResponse = response;
        if (session.privateConversationData.yolandaResponse.question){
            session.replaceDialog('/rbloop');
        } else {
            sendRBResult(session, session.privateConversationData.yolandaResponse.result);
            delete session.privateConversationData.yolandaSession;
            session.endDialog();
        }
    });
}

When the Rainbird response is not a question, we process the response as an answer to our query. First, we call sendRBResult() , which constructs our message to inform the user of the query results. We then clear out our conversation to allow future Rainbird queries to run without knowledge of the answers given in this conversation, and finally, we complete our clean up by instructing our bot to endDialog().

function sendRBResult(session, rbResult) {
    if (rbResult && rbResult.length > 0) {
        var message = '';
        rbResult.forEach(function(result) {
            message += result.subject + ' ' +result.relationship + ' ' + result.object + ' \t' +
                getEvidenceTreeLink(result.factID) + '\n\n';
        });
        session.send(message);
    } else {
        session.send('Could not find any answers.');
    }
}

In the above sendRBResults(), we construct a link to Rainbird’s Evidence Tree, a visual representation of the path Rainbird took to derive the associated answer. An optional display which in some industries is essential for compliance.

To complete our analysis of the solution, let’s look at the code used by Skype and Web Chat to present a welcome message. Note this feature is not available in Slack.

// Event to send a welcome message (works with Web Chat)
bot.on('conversationUpdate', function (message) {
    if (message.membersAdded) {
        message.membersAdded.forEach(function (identity) {
            if (identity.id === message.address.bot.id) {
                bot.beginDialog(message.address, '/welcome');
            }
        });
    }
});

// Skype event used to send a welcome message
bot.on('contactRelationUpdate', function (message) {
    if (message.action === 'add') {
        var name = message.user ? message.user.name : null;
        var reply = new builder.Message()
            .address(message.address)
            .text("Hello %s.  Thanks for adding me, how can I help you?", name || 'there');
        bot.send(reply);
    }
});  

As you can see, it’s possible to handle bot specific events. Here, we have demonstrated handling both Web Chat and Skype specific events to present the user with a ‘welcome’ message when they first enter the bot.

Our final step of the hack was to configure and deploy our bot implementation in Azure. We followed this blog to guide us through the deploy process.

Summary

This demo produced in the three-day hack represents a starting point that other implementations can quite easily build on inline with specific requirements. You could go on to add additional Rainbird queries identified through further training in LUIS. Expand the range of QnA Maker questions and even configure multiple Rainbird knowledge maps. A solution that is scalable yet flexible.

Technical limitations such as the handling of the restart command could be improved on perhaps by supporting multiple conversations simultaneously.

The bot framework through Azure makes it easy to roll out to other bots (referred to as channels in Azure), by default Skype and Web Chat are selected, but adding others is as easy as signing up, selecting the channel and configuring with your credentials. This is all achieved in the bot framework portal. The array of available channels is extensive; Bing, Cortana, Facebook Messenger, Kik, and Slack, as well as several others.

Our only stumbling block came from the bots inability to handle plural answers when presenting a multiple choice question. When the bot presented options the functionality followed radio button behavior, whereas our use case would have favored checkbox behavior. Having spoken with Microsoft, they assure us this is being worked on and we look forward to seeing this feature in a future release.

Resources

Topics:
ai ,automated decision making ,microsoft bot framework ,recommendation engine ,cognitive computing ,tutorial ,rainbird

Published at DZone with permission of Nathan Roberts. See the original article here.

Opinions expressed by DZone contributors are their own.

THE DZONE NEWSLETTER

Dev Resources & Solutions Straight to Your Inbox

Thanks for subscribing!

Awesome! Check your inbox to verify your email so you can start receiving the latest in tech news and resources.

X

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}