How to Properly Test Your Dialogflow CX Agents

Previous Requisites

Here are the technologies used in this project:

1. Google Cloud Account
2. Dialogflow API enabled
3. Dialogflow CX CLI 

Tests, Tests, and Tests!

Testing your conversations ensures that the components of an agent are running properly at a level that includes the auxiliary infrastructure of the agent, such as the NLU, the webhook, and the integration with external systems.

A test can evaluate the components of an application at a high level. We can use tests to check that two or more components of an agent work together and generate an expected result, possibly including all the components necessary to fully process a request.

These tests can be executed manually or automated in a continuous integration system and are executed in each new version of the agent.

Manual

Simulator

Dialogflow Console is a web interface where you can design your conversations by creating agents and within an agent, creating flows, intents, entity types, etc. On the Dialogflow Console, you can create and interact easily with your intent. To do that you just need to go to the Dialogflow CX Console. This is what it looks like:

The console includes a really useful tool to manually test your agent. It is a simulator where you can interact with your agent to test that the conversation is flowing as expected. To start testing your agent, you have to click on the Test Agent button located in the top-right corner of the canvas. When you start the simulator, you will have to choose the environment, flow, and page. To interact with the simulator, you can just type text and send it to the agent, but you can also set parameters, send events, etc. You can undo the last conversation turn every time you want.

Once you have finished with your manual interaction you can:

1. Save the entire conversation as a test case.
2. Replay the conversation automatically.
3. Reset the conversation in case you want to start from the beginning.

Automated

CI/CD

In software, it is a common pattern (and a best practice) to have different environments where developers can deploy different versions of their software. Each environment has its configurations.

In Dialogflow CX we have the same concept, you can create a version of your agent and then, deploy it to an environment. It is the same with the webhook: you can deploy a webhook version and use that version in an environment.

When you saved a conversation that you have done in the simulator as a test case, you can add it to a continuous integration pipeline of a specific environment. You will find your CI/CD pipelines in the Manage tab and the clicking in the CI/CD section:

The Dialoglfow CX CLI or cxcli is a command line interface tool that you can use to interact with your Dialogflow CX projects in a terminal. It is an open-source project I created. With the cxcli you can interact easily with your Dialogflow CX intents.

With the cxcli you can interact easily with the CI/CD pipelines of your environments of your Dialogflow CX agents.

You can find the CI/CD command usage down the cxcli environment execute-cicd command. 

cxcli environment execute-cicd [environment] [parameters]

This a simple example of the cxcli environment execute-cicd command:

cxcli environment execute-cicd cicd-env --project-id test-cx-346408 --location-id us-central1 --agent-name test-agent

The command above will give you an output like this one:

$ cxcli environment execute-cicd cicd-env --project-id test-cx-346408 --location-id us-central1 --agent-name test-agent
INFO Executing cicd for environment cicd-env      
INFO PASSED                     

NLU Profiling Using Dialogflow CX CLI

Use the NLU Profiler to test user utterances and improve your agent’s interaction model.

With the NLU Profiler, you can see how utterances resolve to intents and slots in your interaction model. When an utterance doesn’t resolve to the right intent or slot, you can update the interaction model and try again. With the cxcli, you can see what intents Dialogflow CX considered and discarded. Then, you can determine how to use additional samples to train your model to resolve utterances to their intended intents and slots.

Every suite is executed in the Dialogflow CX session, so you can test not only your NLU, but also a conversation itself.

All of the commands that you have available in the cxcli to execute the NLU profiler are located down the cxcli profile-nlu command.

This command will execute a suite that includes a set of tests. It is important to know which suites and tests you can build. Suites and tests are defined as yaml files. You can run these suites from your terminal or your CI pipelines using cxcli.

To execute a suite, you have to run the cxcli profile-nlu execute command. For usage, please refer to this page.

cxcli profile-nlu execute [suite-file] [parameters]

Suites

A suite is a YAML file with the following structure:

# suite.yaml

# Name of the suite.
name: Example Suite
# Brief description of the suite.
description: Suite used as an example
# Project ID on Google Cloud where is located your Dialogflow CX agent.
projectId: test-cx-346408
# Location where your Dialogflow CX agent is running. 
# More info here: https://cloud.google.com/dialogflow/cx/docs/concept/region
locationId: us-central1
# Agent name of your Dialogflow CX agent.
# Notice: it is the agent name, not the agent ID.
agentName: test-agent
# You can have multiple tests defined in separated files
tests:
  # ID of the test.
  - id: test_id
    # File where the test specification is located
    file: ./test.yaml

You can find the full reference here.

Tests

A test is a YAML file with the following structure:

# test.yaml

# Name of the test.
name: Example test
# Brief description of the test.
description: These are some tests
# Locale of the interaction model that is gonna be tested. 
# You can find the locales here: https://cloud.google.com/dialogflow/cx/docs/reference/language
localeId: en
# A check is a test itself: given an input, you will validate the intents and the parameters/entities detected by Dialogflow CX
# You can have multiple checks defined
checks:
  # The ID of the check
  - id: test
    input:
      # the input type
      # it could be text or audio
      type: text
      # The input itself in text format. For type: audio, you have to specify the audio tag.
      text: I want 3 pizzas
    validate:
      # Intent that is supposed to be detected
      intent: order_intent
      # You can have multiple parameters/intents
      # Notice: this could be empty if your intent does not have any entities/parameters.
      parameters:
        # Entity name that is supposed to be detected
        - parameter: number
          # Value that is supposed to be detected
          value: 3

You can find the full reference here.

Examples

• A simple example that demonstrates the NLU Profiler in action
• An example that shows how to validate Dialogflow CX built-in entities
• Full example with multiple user-defined entities and Dialogflow CX built-in entities
• Example using an audio file as an input

You can find more examples on our GitHub repo and the Examples page.

This a simple example of the cxcli profile-nlu execute command:

cxcli profile-nlu execute examples/suite.yaml

The command above will give you an output similar to this one:

$ cxcli profile-nlu execute suite.yaml
INFO Suite Information: test-agent                
INFO Test ID: test_1                              
INFO Input: type: text, value: hi                 
INFO Intent Detected: hi_intent                   
INFO Input: type: text, value: hello              
INFO Intent Detected: hi_intent                   
INFO Input: type: audio, value: ./audio/hi.mp3    
INFO Intent Detected: hi_intent                   
INFO Test ID: test_2                              
INFO Input: type: text, value: I want 3 pizzas    
INFO Intent Detected: order_intent                
INFO Param order_type: pizza                      
INFO Param number: 3                              
INFO Input: type: text, value: I want 2 cokes     
INFO Intent Detected: order_intent                
INFO Param number: 2                              
INFO Param order_type: coke                        

Resources

Check out the full usage of the cxcli profile-nlu command and the cxcli environment command.

If you want to learn more about Dialogflow CX testing, check the official documentation.

Conclusion

This was a basic tutorial to learn how to properly set your Dialogflow CX Agents. As you have seen in this example, creating tests and setting CI/CD pipelines either with the console or the cxcli is very easy!

I hope this tutorial will be useful to you.

That’s all folks! Happy coding!