Testing web applications with Selenium

There is a common problem between many testing harnesses: they are different from the real client (in the case of web applications a browser). Zend_Test, HttpUnit and similar tools perform fake HTTP requests (that may go or not go over the TCP/IP stack), and check an HTTP response in HTML or XML to make sure your application is producing the correct results.

Another problem is that usually JavaScript is not executed by these tools (and if it's executed, like in HttpUnit's case, it is a subset of the real JavaScript available in browsers and may lack some objects or methods in the DOM.)

Selenium uses real browsers

Let's talk about Selenium instead. Selenium instances manage real browsers, like Firefox, and send commands telling them how to crawl web applications in their behalf. This way, Selenium is easily able to execute JavaScript in every form (there is no Rhino or similar alternate implementation which supports only some features.). There is no replication of code between common HTTP clients and the testing harness.

Still, you need to run your app in a sandbox, so that you can recreate an initial state that the tests start from, and that you can throw away after having checked with assertions the results embedded in the generated pages.

As a Firefox extension

There are different distributions of Selenium. Selenium IDE is a record and replay tool: basically you install it as a Firefox extension, visit an host that contains the site or application to test, and save what you do in the browser as HTML files (much like with FitNesse acceptance tests).

You can then replay a test suite in the browser or via an Api (discussed later), on any host (specifying at that time localhost, stagingserver, etc.)

An issue that came to my mind is that Selenium IDE is not adapt to Test-Driven Development: there must be an existing user interface before you can record a test case. Therefore you lose the benefits of testability as a design tool (not graphic design but application design.)

That said, I'm still learning how to test-drive an AJAX interface, and maybe Test-Driven Development is not that useful for GUIs. However, Acceptance Test-Driven Development with HTML and CSS was a charm (in tests I checked that XPath or CSS expressions selected an element which contained a particular text, or that a table or a list had N elements...)

As a server to refer to in your builds

By the way, Selenium IDE is not the only way to produce a test for Selenium, nor the only way to run it. Let's move towards some kind of build automation, that is run via command line or automatically instead of at the push of a browser button.

Selenium RC (Remote Control) is a server that launches browsers as requested by its clients. Clients can be xUnit test cases in your language of choice (JUnit, PHPUNit), that access the Selenium Api provided as a library in the very Selenium RC package. This is radically better in case you're test-driving the user interface.

You can launch Selenium RC at every test run, passing it the test suite in HTML format, or simply leave it in execution on a server (possibly in Continuous Integration). I run it as a daemon on the Ubuntu machine that performs the build.

When using Selenium RC, you'll actually see Firefox launched and terminated for test runs on the machine that is hosting it, and a walktrough of the application will be executed for you at any time. You can even film it, manually or programmatically, and in case of a test regression rewatch it to see what went wrong. Selenium RC is obviously heavier than HttpUnit and other lower-level tools, but actually seeing a browser access the application is much more powerful and revealing.

Java Api

In case you're using Java for your tests, SeleneseTestCase is the class you're going to refer to (you'll find the Jar in the package that contains Selenium RC.) You don't need to always extend it, you can just subclass it one time, add a getSelenium() method and compose it in your acceptance tests.

Here is my SeleniumIntegrationTest (checks that the Api and Selenium RC are installed and working, and my wrapper of SeleneseTestCase to use it as a collaborator in your test instead of as a superclass.

//SeleniumIntegrationTest.java
package it.polimi.chansonnier.test;

import java.io.File;

import junit.framework.TestCase;

import com.thoughtworks.selenium.*;

public class SeleniumIntegrationTest extends TestCase {
	WrappableSeleneseTestCase wrapped;
	Selenium selenium;
	
	
	public void setUp() throws Exception {
		wrapped = new WrappableSeleneseTestCase();
		wrapped.setUp("http://www.google.com/", "*chrome");
		selenium = wrapped.getSelenium();
	}
	
	public void tearDown() throws Exception {
		wrapped.tearDown();
	}
	
	public void testSeleniumRCIsStartedAndWorks() throws Exception {
		selenium.open("/");
		wrapped.verifyTrue(selenium.isTextPresent("Google"));
	}	
}

//WrappableSeleneseTestCase.java
package it.polimi.chansonnier.test;

import com.thoughtworks.selenium.SeleneseTestCase;
import com.thoughtworks.selenium.Selenium;

public class WrappableSeleneseTestCase extends SeleneseTestCase {
	public Selenium getSelenium() {
		return selenium;
	}
}

In build servers

Selenium RC normally requires a graphical environment, since modern browsers won't launch without a display to render themselves in. But it is still useful to run Selenium RC in build automation, maybe in server without a GUI.

My situation was exactly that: I run my builds and test suites on an Ubuntu Server. But at least on Unix, servers haven't graphical interfaces. Fortunately, this is Linux and you can plug in software like Lego bricks: meet X virtual framebuffer, essentially a Fake testing pattern applied to XServer:

In the X Window System, Xvfb or X virtual framebuffer  is an X11 server that performs all graphical operations in memory, not showing any screen output. From the point of view of the client, it acts exactly like any other server, serving requests and sending events and errors as appropriate. However, no output is shown. This virtual server does not require the computer it is running on to even have a screen or any input device. Only a network layer is necessary.

Of course every server has a network layer, or you won't be able to access it by ssh.

Thus, I installed the xvfm and firefox packages on Ubuntu Server, and instead of running selenium RC server like this:

sudo java -jar selenium-server.jar -log /home/giorgio/log/selenium &

in my /etc/rc.local, I now run it like this:

sudo xvfb-run -e /home/giorgio/log/xvfb-run java -jar selenium-server.jar -log /home/giorgio/log/selenium &

If you install also the package x11-apps, you can make some nice photographs with xwd while Selenium RC accomplishes its job just fine: