An Introduction to LUIS (Language Understanding Intelligent Service)

Language Understanding Intelligent Service (LUIS) enables developers to build smart applications that can understand human language and respond accordingly to user requests.

Let's first try to understand why we need LUIS.

How Do You Communicate With a Web Application?

In a web application, you search for a functionality in the menu. The menu, menu items, screen layout, and navigation vary for each application. Before you use any application, you need to familiarize yourself with the menu items and the navigation. The same functionality may have different names in different applications. The UI will be updated from time to time as part of enhancements. Thus, learning to use a web application is a continuous process.

For example, see the following screenshots of a few online pizza sites. They provide 90% similar functionality but have only 10% similar user interfaces.

"How do you communicate with humans in any business?" you ask. It is the same everywhere, whether it is a bank, retail, or a hospital. There is no learning or training involved.

What if you could achieve the same with software? Imagine the usability and comfort levels this would bring. It takes the user experience to a different level.

Rather than human, the software will be trained to understand the intent and provide the requested data. It will keep learning from user interactions and provide better service over time.

This need for software that can understand human language has been around for a long time.

How Do You Communicate With a Bot?

You speak or type. It is the same way you communicate with any kind of business. The bot understands your language and gets the intent. Based on the intent, it asks for the required inputs and provides the required service. This is possible only if the software has the ability to understand human language.

Introduction

Read the following sentences.

• What is your name?
• Your name.
• Your good name, please.
• Name.
• I am Murali. You?
• How do I call you?
• Sorry. I don’t remember your name.
• Sorry. I don’t remember you.
• My bad. Not able to recollect your name.
• May I know your name?
• Myself Ambily. You?
• Your name, please.

If you observe, there is so much variation in the above sentences but we can still understand the intent, which is to know the name of the person. This kind of human intelligence like understanding language, identifying a picture, predicting — among other mental capabilities — is cognitive intelligence.

Microsoft provides the following cognitive services.

LUIS is a service that provides a language understanding cognitive intelligence to bots or any other application.

LUIS uses the power of machine learning to solve the difficult problem of extracting meaning from natural language input so that your application does not have to. Any client application that converses with users, like a dialog system or a chatbot, can pass user input to a LUIS app and receive results that provide natural language understanding.

LUIS is a service developed by Microsoft that has algorithms to understand human language and come up with intents that can be used to execute specific commands/code snippets in the client application.

LUIS App

A LUIS app or LUIS model is defined by a developer for a specific application/domain. The output of a LUIS app is a web service with an HTTP endpoint that you reference from your client application to add natural language understanding to. The user utterance is the input. Intents and entities are returned to the client application. The client application can then take appropriate action based on the user intentions that LUIS recognizes.

Intent

Intents are similar to verbs in a sentence. An intent represents actions the user wants to perform. It is a purpose or goal expressed in a user's input, such as booking a flight, paying a bill, or finding a news article. You can define a set of named intents that correspond to actions users want to perform in your application. A travel app may define an intent named "BookFlight,” which LUIS extracts from the utterance "Book me a ticket to Paris." A maximum of 80 intents is allowed in a LUIS model.

For example, all the following questions (utterances), which do not appear similar in any way, have only one intent: to know whether the current user licenses quota has exceeded the upper limit.

• Were there any issues with user quota exceeding the limit?
• How was user quota utilization?
• User quota exceeded?
• Any instants when user quota exceeded?
• How many times user quota exceeded?
• Exceeded user quota count?
• How many times did the user quota get exceeded?
• License limit exceeded events.
• Get the quota exceeded events.

Utterance

An utterance is a textual input from the user that the app needs to interpret. It may be a sentence, like "Book me a ticket to Vizag" or a fragment of a sentence, like "Booking" or “Vizag flight." Utterances are not always well-formed and there can be many utterance variations for a particular intent. The maximum length of an utterance is 500 characters.

For example, again, all of the above questions are the utterances for a single intent: user quota utilization.

Entity

If intents are verbs, then entities are nouns. An entity represents an instance of a class of object that is relevant to a user’s intent. Entities are parameters required to process the user request. In the utterance, "Book me a ticket to Vizag," “Vizag" is an entity of type location. By recognizing the entities that are mentioned in the user’s input, LUIS helps you choose the specific actions to be executed to fulfill an intent. A maximum of 30 entities is allowed in a LUIS model.

For example, your intent is to find service tickets raised for each company. The company will be a required parameter for you and will be defined as an entity in LUIS Model.

The utterance, "How many service tickets were raised for [Company]?" could identify all the following queries from the user. Adequate training is required to achieve this.

• How many service tickets were raised for HP company?
• How many service tickets were raised for Sony company?
• How many service tickets were raised for TCS company?
• How many service tickets were raised for ABC company?

Types of Entities

Entities may be prebuilt or custom.

Prebuilt

These are built-in types that represent common concepts like dates, time, and geography. These do not count towards the maximum number of entities you may use in your LUIS app.

Example: ordinal, datetimeV2, age, and so on.

Custom

There are many types of custom entities.

Simple

A simple entity is a generic entity that describes a single concept. These count towards the maximum number of entities you may use in your LUIS app.

Example: company, travel agency, and so on.

• Book my travel through Gobibo travel agency.

• Please book my travel with MakeMyTrip agency.

• Arrange my travel with Yatra travel agency.

• Prefer to book my travel through ClearTrip.

List

List entities represent a fixed set of synonyms or related words in your system. Each list entity may have one or more synonyms. They are not machine-learned and are best used for a known set of variations on ways to represent the same concept. List entities do not have to be labelled in utterances or trained by the system. A list entity is an explicitly specified list of values. Unlike other entity types, LUIS does not discover additional values for list entities during training. Therefore, each list entity forms a closed set.

Your app may use up to 50 list entities, and they do not count toward the maximum number of list entities you may use, which is 30. Each list can contain up to 20,000 items.

While adding the list items, LUIS will display some recommendations from its own learning through machine language algorithms. You can also import a list in JSON format.

For example, a list entity TravelClass can be defined with items like sleeper, first, second, and third.    

• Book a train ticket in Sleeper class.

• Book a train ticket in First class.

Metros is a list of cities — Hyderabad, Chennai, and so on. Hyderabad has synonyms like hyd, capital of Telangana, and others.

Hierarchal

A hierarchical entity defines a category and its members. It is made up of child entities that form the members of the category.

A hierarchical entity can consist of up to ten child entities. These count towards the maximum number of entities you may use.

For example, in a travel agent app, you can add hierarchical entities like $Location, including $FromLocation and $ToLocation, as child entities that represent origin and destination locations.

For example, a hierarchal entity Location could be defined with child entities and ToLocation.

• Book a cab from Hyderabad to Secunderabad
• Book a cab from my home to Gachibowli

Composite

A composite entity is made up of other entities that form parts of a whole. A composite entity can consist of up to 20 child entities. These count towards the maximum number of entities you may use.

For example, to book a flight, the inputs required can be travel date, travel class, location, number of travellers, and so on.

A composite entity “FlightBookingInputs” can be defined composing all the entities for number of travellers, travel class, travel date, from location, to location, and so on.

• Book a flight ticket for 3 members in economy on 01 - sep from  hometown kakinada to ongole

• Book a flight for two in executive from america to england next monday

Features

You can add features to your LUIS app to improve its performance. Features help LUIS recognize both intents and entities by providing hints to LUIS that certain words and phrases are part of a category or follow a pattern. When your LUIS app has difficulty identifying an entity, adding a feature and re-training the LUIS app can often help improve the detection of related intents and entities.

Phrase list features contain some or all of an entity's potential values. If LUIS learns how to recognize one member of the category, it can treat the others similarly. The maximum length of a phrase list is 5,000 items. You can have a maximum of ten phrase lists per LUIS app.

Example: Cities {London, New York and Paris}; Count {count, how many, number of}.

Pattern features help your LUIS app easily recognize regular patterns that are frequently used in your application's domain, such as the pattern of flight numbers in a travel app or product codes in a shopping app.

Example: Flight numbers are normally in the format JA213, AI162, and so on. The format can be captured by having the pattern feature [A-Za-z]{2}[0-9]{3} as the value of the flight number pattern.

Training and Testing a LUIS Model

Training is the process of teaching your LUIS app by example to improve its language understanding. If you make updates by adding, editing, or deleting entities, intents, or utterances to your LUIS app, you need to train your app before testing and publishing. When you train a LUIS app, LUIS generalizes from the examples you have labeled and learns to recognize the relevant intents and entities in the future, which improves its classification accuracy.

Training and testing is an iterative process. After you train your LUIS app, you test it with sample utterances to see if the intents and entities are recognized correctly. If not, make updates to the LUIS app, train, and test again.

The testing could be done in two ways:

1. Interactive testing
2. Batch testing

Interactive testing enables you to test both the current and published versions of your app and compare their results on one screen. Interactive testing runs by default on the current trained model only. For a published model, interactive testing is disabled and needs your action to enable it because it is counted in hits and will be deducted from your key balance.

The Interactive Testing tab is divided into two sections:

1. The test view on the left side of the screen, where you can type the test utterance in the text box and press Enter to submit it to your app.

2. The result view on the right side of the screen, where your LUIS app returns the test result, which is the predicted interpretation of the utterance.

Publish LUIS Model

Once you finish building, training, and testing your LUIS app, you can publish it.

You can either publish your app directly to the production slot where end users can access and use your model or you can publish to a staging slot for working through trials and tests to validate changes before publishing to the production slot.

If you want to use a key other than the starter key, assign the new key.

Test your published endpoint in a browser.

Dashboard

The dashboard is a tabular and graphical report that will help you monitor your LUIS model.

Summary

This article gave an introduction to LUIS and the contents of a LUIS model. To summarize...

The communication with software is going to change from being menu-driven to involving more human-like conversations with the capabilities brought by LUIS.

Utterances are the textual inputs/user queries that your application needs to interpret.

An intent represents actions the user wants to perform. It is a purpose or goal expressed in a user's input.

An entity represents an instance of a class of object that is relevant to a user’s intent. Entities are parameters required to process the user request.

Training is the process of teaching your LUIS app by example to improve its language understanding. Training and testing is an iterative process. After you train your LUIS app, you test it with sample utterances to see if the intents and entities are recognized correctly. If they're not, make updates to the LUIS app, train, and test again.

Once you finish building, training, and testing your LUIS App, you can publish it.

The dashboard is a tabular and graphical report that will help you monitor your LUIS model.

Conclusion

LUIS brings in artificial intelligence (AI) to applications so that computers and humans can speak with each other seamlessly. It is built on machine learning and complex algorithms. It will continue to gain momentum and transform business in the upcoming years.