What Assumptions Do Your Tests Make?

Does your test suite launch your application or expect it to be running already? This question came up while working on a client project; I launched the Selenium-based test suite and everything failed ... no requests got processed and I spent some time tracking down why the test suite was failing to launch the application.

I chatted about this with my client ... and found out that the Selenium test suite expects the application to already be running. It starts up Selenium Server, sure, but the application-under-test should already be running.

And suddenly, I realized I had a Big Blind Spot, dating back to when I first started using Selenium. For years, my testing pattern was that the test suite would start up the application, run the Selenium tests, then shut it down ... and this makes perfect sense for testing the Tapestry framework, where the test suite starts and stops a number of different mini-applications (and has to run headless on a continuous integration server).

But an application is different, and there's a lot of advantages to testing against the running application ... including being able to quickly and easily reproduce any failures in the running application. Also, without the time to start and stop the application, the test suite runs a bit faster.

Certainly, the command line build needs to be able to start and stop the application for headless and hands-off testing. However, the test suite as run from inside your IDE ... well, maybe not.

So ... how do you handle this on your projects?

From http://tapestryjava.blogspot.com/2012/02/tests-auto-launch-application-or-expect.html