I have this web project. A pretty complex one. The code coverage is over 90%. Awesome! I even have a Continuous Integration set up. My code is built, and my tests make sure that functionally works as expected. I even have a well defined suite of UI tests using an internal renderer that is super fast. But, ohhh… today a Customer filled in a complaint saying that the site is not working well. Buttons are not responding, images are missing and the like. Catastrophe! This may be familiar to you? What am I missing here? I need external, not controlled-by-us browser behavior testing! How do I solve this problem? Or at least have an early warning that Sauron is about to be awake?
There are many available tools for web testing, all of them have advantages and disadvantages. The key concepts to take into account are re-usability, maintainability, and ability to reproduce as close as possible the actual user experience in a programmatic way (happy user → happy Customer → Christmas bonus!) .
You have a screen, you define some actions based on coordinates on the screen and you develop your tests accordingly. Even better, the tool provides you with a UI to make your life simpler. Everyone is happy until the web designer has this crazy idea about improving usability and change the order of Save – Cancel buttons, to Cancel – Save. Tests start behaving in a strange way. The problem is not easy to debug. And we are not talking of a major UI design change.
There is a big improvement to this approach: Instead of coordinates, small screenshots are used to identify a certain element within a page (eg Project Sikuli). It is definitively a major strategy improvement, but you will have the same problems as when using coordinates if changes in the UI are not trivial ones.
The Bad: Not very scalable, low re-usability, poor maintenance.
The Good: Quick solution. Good for proof of concept.
You have access to the internals of the applications under testing (data items, controls, widgets, etc.) . Of the tool that I used that take this approach (i.e. TestComplete ), I can say that it is a paid solution per workstation; it was not portable to other OS (making difficult the integration with an existing Continuous Build machine) and has many re-usability problems. Last, but not least, element identification was done mostly using the actual text displayed on-screen. Which is bad… Think for one second about tests defined that way and internationalization… Ufff!
The Bad: Not very scalable, low re-usability, poor maintenance, possible i18n issues, possible poor integration with CI solutions, portability issues, and you are not using the actual browser to test.
The Good: Access to the internals of the application. Easy to do assertions over third party artifacts (eg. contracts generated in PDF format by the system)
In my opinion, test recorders such as Selenium IDE are really a bad solution for medium/large projects. Let’s say that you record 100 tests. Then, something changes in your web project. If it is a core change you will have to work on most of your hundred tests to make them run again. On the other side you do have Page Models (they are developed not automatically generated), where you make a change in one or two classes and automatically all your hundred tests start working like a charm. I’ll talk about this next.
This is the approach we finally decided to use. But which tool to use? What about Selenium RC? What about Windmill? This last one looks pretty good at a first glance… And at a second one too… But we finally choose Selenium. Why?
- With Selenium you can use the same IDE in which you develop your code to debug your tests too, increasing developer productivity.
- Selenium can be integrated with existing testing frameworks, for example JUnit and TestNG, while Windmill has its own test runner. So in existing projects you cannot take advantage of existing configurations, for example in your CI machine.
- Selenium is working hard on Selenium 2.0, which will solve the problems with IE among others.
The Good: Selenium is open source with a most active community behind it. It can be easily integrated into Java code. Using Java and Maven, you can start up a Selenium Server, a web app server, deploy the application in that server doing a clean deploy, test it using one or more browsers and, if you are in the mood, making a screenshot if a problem is detected. Your solution will be cross-platform too. What about re-usability and maintainability? Here it comes one of the most interesting things about this approach.
Because Selenium can be used in a programmatic way (with Java, for example), you can implement the Page Model Pattern that will save you of several developer hours fixing tests. I will follow up with a post about the Page Model Pattern next.