Running UAT and Integration Tests During a VSTS Build

there are lots of small suggestions i’ve learned from experience when it is time to create a suite of integration / uat test for your project. a uat or integration test is a test that exercises the entire application, sometimes composed of several services that are collaborating to create the final result. the difference from uat tests and integration test, in my personal terminology, is that the uat uses direct automation of user interface, while integration tests can skip the ui and exercise the system directly from public api (rest, msmq commands, etc).

the typical problem

when it is time to create a solid set of such tests, having them run in an automated build is a must, because it is really difficult for a developer to run them constantly as you do for standard unit tests.

such tests are usually slow: developers cannot waste time waiting for them to run before each commit, we need to have an automated server to execute them constantly while the developer continues to work.

those tests are ui invasive: while the uat test runs for a web project, browser windows continues to open and close, making virtually impossible for a developer to run an entire uat suite while continue working.

integration tests are resource intensive: when you have 5 services, mongodb, elasticsearch, and a test engine that fully exercises the system, there are little more resources available for doing other work, even if you have a really powerful machine.

large sets of integration/uat tests are usually really difficult to run for a developer; we need to find a way to automate everything.

creating a build to run everything in a remote machine can be done with vsts / tfs, and here is an example.

figure 1: a complete build to run integration and uat tests.

the build is simple; the purpose is having a dedicated virtual machine with everything needed to run the software already in place. then, the build will copy the new version of the software in that machine, with all the integration test assemblies, and finally, run the tests on the remote machine.

running integration and uat tests is a task that is usually done in a different machine from that one running the build. this happens because that machine should be carefully prepared to run the tests and simplify deployment of the new version.

phase 1 – building everything

first of all, i have phase 1, where i compile everything. in this example, we have a solution that contains all .net code, then a project with angular 2. we first build the solution, then npm restore all the packages and compile the application with ng cli. finally, i publish a couple of web sites. those two web sites are part of the original solution, but i publish them separately with an msbuild command to have full control of publish parameters.

in this phase, i need to build every component, every service, every piece of the ui needed to run the tests. also, i need to build all the test assemblies.

phase 2 – pack release

in the second phase, i need to pack the release, a task usually accomplished by a dedicated powershell script included in source code. that script knows where to look for dll, configuration files, modify configuration files etc, copying everything into a couple of folders: masters and configuration. in the masters directory, we have everything needed to run everything.

to simplify everything, the remote machine that will run the tests is prepared to accept an xcopy deployment. this means that the remote machine is already configured to run the software in a specific folder. every prerequisite, everything needed by every service, is prepared to run everything from a specific folder.

this phase finishes with a couple of windows file machine copies, to copy this version on the target computer.

phase 3 – pack uat testing

this is really similar to phase 2, but in this phase, the pack powershell scripts create a folder with all the dll of the uat and integration tests, then copy all test adapters (we use nunit for unit testing). once the pack script is finished, another windows file machine copy task will copy all integration tests on the remote machine used for testing.

phase 4 – running the tests

this is a simple phase where you use deploy test agent on the test machine, followed by run functional test tasks. please be sure to always place a deploy test agent task before each run functional test task, as described in this post that explains how to deploy test agent and run functional tests .

conclusions

for complex software, creating an automated build that runs uat and integration tests is not always an easy task, and in my experience, the most annoying problem is setting up winrm to allow remote communication between agent machine and test machine. if you are in a domain, everything is usually simpler, but if for some reason the test machine is not in the domain, prepare yourself to have some problems before you can make the two machines talk together.

in the next post, i’ll show you how to automate the run of uat and integration tests in a more robust and more productive way.