{{announcement.body}}
{{announcement.title}}

DevOps Your Skill: VUI (Voice User Interface) Tests

DZone 's Guide to

DevOps Your Skill: VUI (Voice User Interface) Tests

In this article, we discuss how to test the voice user interface in Alexa in your pipeline and its importance in the day-to-day Alexa Skills development.

· DevOps Zone ·
Free Resource

According to the Wikipedia definition, "a Voice User Interface (VUI) enables human interaction with computers through a voice/speech platform to initiate automated processes or services. VUI is the interface of any speech application."

Thanks to machine learning, big data, cloud, and artificial intelligence, we have managed to communicate with "computers" through the most natural way of communication of the human being: speech.

One of the most important steps in our pipeline is to test the VUI because it is the frontend of our Alexa Skill. These tests are automated in the continuous integration system (CircleCI) and are executed in each new version of the software.

Prerequisites

Here you have the technologies used in this project

  1. ASK CLI - Install and configure ASK CLI.
  2. CircleCI Account - Sign up here.
  3. Node.js v10.x.
  4. Visual Studio Code

ASK CLI (Alexa Skill Kit CLI)

The Alexa Skills Kit Command Line Interface (ASK CLI) is a tool for you to manage your Alexa skills and related resources, such as AWS Lambda functions. With ASK CLI, you have access to the Skill Management API, which allows you to manage Alexa skills programmatically from the command line. We will use this powerful tool to test our Voice User Interface. Let's start!

Installation

The ASK CLI is included in the Docker image we are using, so it is not necessary to install anything else.

Writing Tests

In this step of the pipeline we are going to develop some tests written in bash using the ASK CLI.

These tests are the following:

1. Detect Utterance conflicts

Once we have uploaded the Alexa Skill in the deploy job, it is time to know if the interaction model we have uploaded has conflicts.

To know the conflicts, we will use the ASK CLI command:

  1. For ask cli v1:
Shell
 




xxxxxxxxxx
1


1
ask api get-conflicts -s ${skill_id} -l ${locale}



  1. For ask cli v2:
Shell
 




xxxxxxxxxx
1


 
1
ask smapi get-conflicts-for-interaction-model -s ${skill_id} -l ${locale} -g development --vers ~current



Those commands are integrated in the bash script file test/vui-test/interaction_model_checker.sh.

Here you can find the full bash script:

Shell
 




x


 
1
    #!/bin/bash
2
    skill_id=$1
3
 
             
4
    cli_version=$2
5
 
             
6
    echo "######### Checking Conflicts #########"
7
 
             
8
    if [[ ${cli_version} == *"v1"* ]]
9
    then
10
        folder="../models/*"
11
    else
12
        folder="../skill-package/interactionModels/*"
13
    fi
14
 
             
15
    for d in ${folder}; do
16
 
             
17
        file_name="${d##*/}"
18
        locale="${file_name%.*}"
19
 
             
20
        echo "Checking conflicts for locale: ${locale}"
21
        echo "###############################"
22
 
             
23
        if [[ ${cli_version} == *"v1"* ]]
24
        then
25
            conflicts=$(ask api get-conflicts -s ${skill_id} -l ${locale})
26
        else
27
            conflicts=$(ask smapi get-conflicts-for-interaction-model -s ${skill_id} -l ${locale} -g development --vers ~current)
28
        fi
29
 
             
30
        number_conflicts=$(jq ".paginationContext.totalCount" <<< ${conflicts})
31
 
             
32
        if [[ -z ${number_conflicts} || ${number_conflicts} == "null" ]]
33
        then
34
            echo "No Conflicts detected"
35
            exit 0
36
        else
37
            echo "Number of conflicts detected: ${number_conflicts}"
38
            echo "Conflicts: ${conflicts}"
39
            exit 1
40
        fi
41
 
             
42
    done



The test automatically detects the different interaction models of the skill and it checks their conflicts. This script has two parameters:

  1. The id of the skill.
  2. The version of the ASK CLI you are running (v1 or v2).

2. Test utterance resolutions

Now it is time to check the utterance resolution of our Voice User Interface. Test your utterance resolutions with the utterance profiler as you build your interaction model. You can enter utterances and see how they resolve to the intents and slots. When an utterance does not invoke the right intent, you can update your sample utterances and retest, all before writing any code for your skill.

To run utterance resolutions, we will use the ASK CLI command:

  1. For ask cli v1:
Shell
 




xxxxxxxxxx
1


1
ask api nlu-profile -s ${skill_id} -l ${locale} --utterance "${utterance}"



  1. For ask cli v2:
Shell
 




xxxxxxxxxx
1


1
ask smapi profile-nlu -s ${skill_id} -l ${locale} --utterance "${utterance}" -g development



Those commands are integrated in the bash script file test/vui-test/utterance_resolution_checker.sh.

Here you can find the full bash script:

Shell
 




x


 
1
    #!/bin/bash
2
    skill_id=$1
3
 
             
4
    cli_version=$2
5
 
             
6
    echo "######### Checking Utterance Resolutions #########"
7
 
             
8
    if [[ ${cli_version} == *"v1"* ]]
9
    then
10
        folder="../models/*"
11
    else
12
        folder="../skill-package/interactionModels/*"
13
    fi
14
 
             
15
    for d in  ${folder}; do
16
 
             
17
        file_name="${d##*/}"
18
        locale="${file_name%.*}"
19
 
             
20
        echo "Checking Utterance resolution for locale: ${locale}"
21
        echo "###############################"
22
 
             
23
        while IFS="" read -r  utterance_to_test || [ -n "${utterance_to_test}" ]; do
24
 
             
25
            IFS=$'|' read -r -a utterance_to_test <<< "${utterance_to_test}"
26
            utterance=${utterance_to_test[0]}
27
            echo "Utterance to test: ${utterance}"
28
            expected_intent=${utterance_to_test[1]}
29
            #clean end of lines
30
            expected_intent=$(echo ${expected_intent} | sed -e 's/\r//g')
31
 
             
32
            echo "Expected intent: ${expected_intent}"
33
 
             
34
            if [[ ${cli_version} == *"v1"* ]]
35
            then
36
                resolution=$(ask api nlu-profile -s ${skill_id} -l ${locale} --utterance "${utterance}")
37
            else
38
                resolution=$(ask smapi profile-nlu -s ${skill_id} -l ${locale} --utterance "${utterance}" -g development)
39
            fi
40
 
             
41
            intent_resolved=$(jq ".selectedIntent.name" <<< ${resolution})
42
 
             
43
            echo "Intent resolved: ${intent_resolved}"
44
 
             
45
            if [[ ${intent_resolved} == *"${expected_intent}"* ]]
46
            then
47
                echo "No Utterance resolutions errors"
48
            else
49
                echo "Utterance resolution error"
50
                echo "Resolution: ${resolution}"
51
                exit 1
52
            fi
53
 
             
54
        done < "utterance_resolution/${locale}"
55
 
             
56
    done



Additionally, we have a set of utterances and their expected intents depending on the locale. These set of utterances to tests are available in test\utterance_resolution. In our case, this skill is only available in Spanish, so you can find in that folder the file es-ES:

Properties files
 




xxxxxxxxxx
1


1
    hola|HelloWorldIntent
2
    ayuda|AMAZON.HelpIntent 



As you can see, the format of this file is Utterance|ExpectedIntent. You can check the slot resolution, but I did not do it in this example.

The test automatically detects the different interaction models of the skill and it checks the resolution of the utterances. This script has two parameters:

  1. The id of the skill.
  2. The version of the ASK CLI you are running (v1 or v2).

3. Evaluate and Test our Interaction

To evaluate your model, you define a set of utterances mapped to the intents and slots you expect to be sent to your skill. This is called an annotation set. Then, you start an NLU evaluation with the annotation set to determine how well your skill's model performs against your expectations. The tool can help you measure the accuracy of your NLU model, and run regression testing to ensure that changes to your model don't degrade the customer experience.

This test will check the same that we have tested in the one described above but in a different way. In this test, we are going to test the utterance resolution using annotations.

First of all, we have to create annotations in all locales that we have available our skill.

To know how to create an annotation check this link from the official documentation.

When we have the annotations created, now we can check the utterance resolution using these annotations with the ASK CLI utterance evaluation commands. This is an asynchronous process. so we have to start the evaluation with one command and then get the result with another when the evaluation is finished:

  1. For ask cli v1:
Shell
 




x


1
    #start the evaluation
2
    id=ask api evaluate-nlu -a ${annotation} -s ${skill_id} -l ${locale}
3
    #get the results of the evaluation
4
    ask api get-nlu-evaluation -e ${id} -s ${skill_id}


  1. For ask cli v2:
Shell
 




x


1
    #start the evaluation
2
    id=ask smapi create-nlu-evaluations --source-annotation-id ${annotation} -s ${skill_id} -l ${locale} -g development
3
    #get the results of the evaluation
4
    ask smapi get-nlu-evaluation --evaluation-id ${id} -s ${skill_id}



Those commands are integrated in the bash script file test/vui-test/utterance_evaluation_checker.sh.

Here you can find the full bash script:

Shell
 




x


1
    #!/bin/bash
2
    skill_id=$1
3
 
             
4
    cli_version=$2
5
 
             
6
    echo "######### Checking Utterance Evaluation #########"
7
 
             
8
    if [[ ${cli_version} == *"v1"* ]]
9
    then
10
        folder="../models/*"
11
    else
12
        folder="../skill-package/interactionModels/*"
13
    fi
14
 
             
15
    for d in ${folder}; do
16
 
             
17
        file_name="${d##*/}"
18
        locale="${file_name%.*}"
19
 
             
20
        echo "Checking Utterance evaluation for locale: ${locale}"
21
        echo "###############################"
22
 
             
23
        while IFS="" read -r  annotation || [ -n "${annotation}" ]; do
24
 
             
25
            #clean end of lines
26
            annotation=$(echo ${annotation} | sed -e 's/\r//g')
27
            echo "Annotation to test: ${annotation}"
28
            if [[ ${cli_version} == *"v1"* ]]
29
            then
30
                evaluation=$(ask api evaluate-nlu -a ${annotation} -s ${skill_id} -l ${locale})
31
            else
32
                evaluation=$(ask smapi create-nlu-evaluations --source-annotation-id ${annotation} -s ${skill_id} -l ${locale} -g development)
33
            fi
34
 
             
35
            id=$(jq ".id" <<< ${evaluation})
36
            #Remove quotes
37
            id=$(echo "${id}" | sed 's/"//g')
38
            echo "Id of evaluation: ${id}"
39
            status="IN_PROGRESS"
40
 
             
41
            while [[ ${status} == *"IN_PROGRESS"* ]]; do
42
 
             
43
                if [[ ${cli_version} == *"v1"* ]]
44
                then
45
                    status_raw=$(ask api get-nlu-evaluation -e ${id} -s ${skill_id})
46
                else
47
                    status_raw=$(ask smapi get-nlu-evaluation --evaluation-id ${id} -s ${skill_id})
48
                fi
49
 
             
50
                status=$(jq ".status" <<< ${status_raw})
51
                echo "Current status: ${status}"
52
                
53
                if [[ ${status} == *"IN_PROGRESS"* ]]
54
                then
55
                    echo "Waiting for finishing the evaluation..."
56
                    sleep 15
57
                fi
58
 
             
59
            done
60
 
             
61
            echo "Utterance evaluation finished"
62
 
             
63
            if [[ ${status} == *"PASSED"* ]]
64
            then
65
                echo "No Utterance evaluation errors"
66
            else
67
                echo "Utterance evaluation error"
68
                echo "Evaluation: ${status_raw}"
69
                exit 1
70
            fi
71
 
             
72
        done < "utterance_evaluation/${locale}"
73
 
             
74
    done



Additionally, we have a set of annotations depending on the locale. These set of annotations to tests are available in test\utterance_evaluation. In our case, this skill is only available in Spanish so you can find in that folder the file es-ES:

Shell
 




xxxxxxxxxx
1


 
1
bcdcd3d8-ed74-4751-bb9f-5d1a4d02259c



As you can see, this is the id of the annotation we have created in the Alexa Developer Console. If you have more than one, just add it to a new line.

The test automatically detects the different interaction models of the skill and it runs the evaluation for the annotations given.

This script has two parameters:

  1. The id of the skill
  2. The version of the ASK CLI you are running (v1 or v2).

Reports

There are not reports defined in this job.

Integration

It is not necessary to integrate it in package.json file.

Pipeline Jobs

Everything is ready to run and test our VUI, let's add it to our pipeline!

These 3 tests described above are defined in three different jobs that will run in parallel:

1. Check-Utterance-Conflicts

This job will execute the following tasks:

  1. Restore the code that we have downloaded in the previous step in /home/node/project folder
  2. Run the interaction_model_checker script.
YAML
 




x


 
1
  check-utterance-conflicts:
2
    executor: ask-executor
3
    steps:
4
      - attach_workspace:
5
          at: /home/node/
6
      - run: cd test/vui-test/ && ./interaction_model_checker.sh $SKILL_ID v1



2. Check-Utterance-Resolution

This job will execute the following tasks:

  1. Restore the code that we have downloaded in the previous step in /home/node/project folder
  2. Run the utterance_resolution_checker script.
YAML
 




x


1
  check-utterance-resolution:
2
    executor: ask-executor
3
    steps:
4
      - attach_workspace:
5
          at: /home/node/
6
      - run: cd test/vui-test/ && ./utterance_resolution_checker.sh $SKILL_ID v1



3. Check-Utterance-Evaluation

This job will execute the following tasks:

  1. Restore the code that we have downloaded in the previous step in /home/node/project folder
  2. Run the utterance_evaluation_checker script.
  3. Persist again the code that we will reuse in the next job
YAML
 




x
10


1
  check-utterance-evaluation:
2
    executor: ask-executor
3
    steps:
4
      - attach_workspace:
5
          at: /home/node/
6
      - run: cd test/vui-test/ && ./utterance_evaluation_checker.sh $SKILL_ID v1
7
      - persist_to_workspace:
8
          root: /home/node/
9
          paths:
10
            - project



Note: To perform these tests in CircleCI, you have to set the environment variable SKILL_ID with the id of your Alexa Skill.

Resources

Conclusion

The VUI is our frontend and one of the most important things of our Alexa Skill. This is why these tests are very relevant in our pipeline. Thanks to the ASK CLI we can perform these complex tests.

That’s all, folks!  

You can find the code in my GitHub.

I hope it will be useful! If you have any doubts or questions, do not hesitate to contact me or put a comment below.

Happy coding!

Topics:
alexa, alexa app development, alexa skill, alexa skill development, alexa skills, alexa skills developer, alexa skills development, devops, test, vui

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}