DevOps Your Skill: End-to-End Tests
In this article, we discuss how to make end-to-end tests with Bespoken in Alexa in your pipeline and its importance in the day-to-day Alexa Skills development.
Join the DZone community and get the full member experience.Join For Free
This type of test allows us to verify if the interaction of the software components in our Alexa Skill, such as, VUI, lambda, or if a database works as expected. In summary, end-to-end testing tests the application's ability to satisfy all the requests that the end-user can make.
In terms of voice it is not easy to achieve this kind of tests because of the interaction of the voice as an input of the end-to-end test. We will use Bespoken to write our end-to-end tests. Bespoken allow us to make this kind of tests in an easy way!
These tests are automated in the continuous integration system (CircleCI) and are executed in each new version of the software.
Here you have the technologies used in this project
- ASK CLI - Install and configure ASK CLI.
- CircleCI Account - Sign up here.
- Bespoken Account - Register here.
- Node.js v10.x.
- Visual Studio Code.
Bespoken is the best tool to test our Alexa Skill, as it has us covered from unit tests to end-to-end tests. If we take a look of its official documentation: "Bespoken makes it easy to do end-to-end testing for voice apps."
They use Virtual Devices to do it. A virtual device works like a physical device, such as Amazon Echo, but can be interacted with via Bespoken simple test scripting syntax (as well as programmatically via Bespoken API).
As Bespoken says, there is a lot going on under the covers, but the tests themselves are very easy to write and maintain — in fact, they believe it's as easy to type the tests as it is to manually say them. And once they have been written, you have an automated test suite that can be run at any time and incorporated into other automated processes (such as continuous integration/continuous delivery).
We have written, tested, and run tests using Bespoken, and we can conclude that is the best test tool for voice apps you can use!
The Bespoken tool is included in the Docker image we are using so it is not necessary to install anything else.
The configuration in Bespoken is really easy. We only need to have a Bespoken Account and create a virtual device in its easy-to-use website:
As our Docker image has the
bespoken-tools npm package installed, mentioned in the previous section, we can run the command that will set up our test suite:
This command will prompt the following step-by-step guide:
Where we will have to enter the following information:
- The type of tests we you creating. In our case, end-to-end tests.
- The name of your voice app. In our case, hola mundo.
- The available locales. In our case es-ES.
- The platform we are using, In our case Alexa.
- And finally, the token of the virtual device we have created above.
After running that command, Bespoken will create a JSON file
testing.json and a
e2e folder with an example of an end-to-end test.
First, we will take a look to our
Really easy, right? In that file, we can see the input we entered in the
bst init command.
In order to have the repo well structured, all the bespoken files are located in
Now we are ready to write our end-to-end test suite.
In this step of the pipeline we are going to develop some end-to-end tests written using Bespoken.
Once we have tested our Voice User Interface, our backend and those systems together using text as input and we check that everything is correct. It is time to test all software components related in an Alexa Skill request. But now, using the voice as input.
In the folder
test/e2e-bespoken-test/e2e, you can find the test suite file
index.e2e.yml. This is how this file looks like:
As you can see, we have 3 tests in this suite:
- The first one will test a launch request and the HelloWorld intent.
- The second one will test a launch request, the AMAZON.HelpIntent, and will end the session using the AMAZON.StopIntent.
- The third one will test a launch request, the AMAZON.HelpIntent, and will end the session using the AMAZON.CancelIntent.
This end-to-end tests can be performed with the
bespoken-tools npm package.We will use the following
When we execute this suite, this is what happens in Bespoken:
- Bespoken turns the text input of the test into audio using text-to-speech from Amazon Polly.
- Bespoken sends the speech audio to Alexa.
- Alexa invokes the skill.
- The skill provides a reply — a combination of audio (the vocal response from Alexa) and metadata (for card information).
- We turn the response back into text using speech-to-text.
- We compare it to the expected output.
Once we have executed the suite explained above, the
bespoken-tools npm package will create a fabulous HTML report located in
In this report we can check how that execution went.
This is how this report looks like:
This report will be stored as an artifact of this job in CircleCI.
It is not necessary to integrate it in
Everything is ready to run our end-to-end tests, so let's add it to our pipeline!
This job will execute the following tasks:
- Restore the code that we have downloaded in the previous step in
- Run the Bespoken end-to-end tests using the command
- Store the HTML report generated by the previous execution as an artifact of this job.
- Persist again the code that we will reuse in the next job.
- DevOps Wikipedia - Wikipedia reference
- Official Bespoken Documentation - Bespoken Documentation
- Official CircleCI Documentation - Official CircleCI Documentation
The main reason for carrying out this testing is to determine various dependencies of an application as well as ensuring that accurate information is communicated between various system components. Thanks to Bespoken we can perform this complex tests.
That’s all, folks!
You can find the code in my GitHub.
I hope it will be useful! If you have any doubts or questions, do not hesitate to contact me or put a comment below.
Opinions expressed by DZone contributors are their own.