Earlier this week, I attended the inaugural Selenium User Group meetup at Google's Campus in Mountain View. It was an excellent event, with many of the core committers on hand to present and answer questions. Each presenter had about 5 minutes to speak and we learned many things about the Selenium Project itself, what's coming in the future and how Google has standardized on Selenium as their integration testing tool of choice.
Patrick Lightbody started the meeting by going over a number of project statistics with pretty graphs and such. There were simply too many numbers to write down, so hopefully his slides will be published soon. I was pleased to see that Google did videotape the entire event, so it should be available online soon. I'll update this post when it it. Below are my notes from the event.
Jason Huggins, Test Engineer at Google
Selenium is a test automation framework. Some folks abuse it as a macro tool. There's two reasons Selenium became so popular: it was able to test Ajax before any other testing tool and it allows end-to-end workflow testing. Selenium works on any platform, with any browser and allows many, many languages. It's possible that other frameworks that are more focused are better. There are 4 Selenium products:
- Selenium Core (TestRunner)
- Selenium IDE (for Firefox)
- Selenium Remote Control
- Selenium Grid
Paul Hammant, Open Source Manager at Thoughtworks
Selenium 1.0 - beta this week. RC, Core, Grid and IDE together. 1.0 will be shipped in a few weeks. Compared to today, lots of bugs killed, documentation improved and a greatly improved Selenium IDE.
Selenium 1.1 will be shipped in a couple months. Selenium IDE will be enhanced to obey RC instruction ... becoming the best mode of operation when it ships.
Some experimental iPhone-Safari capability (dependent on SDK for iPhone).
Selenium 2.0 will hopefully be released this year. Roll in of WebDriver functionality and code. New model-based API. IE plugin and Safari plugin (not really a plugin, most likely uses AppleScript).
Philippe Hanrigou (http://ph7spot.com)
Testing is good! Selenium is awesome! However, end-to-end web testing is slow. It is and always will be slow. How do you solve this problem? You solve it the same way we've solved slow traffic in the past - by building more lanes. Rather than one browser testing everything, Selenium Grid allows multiple lanes and 20 browsers. Features include:
- Faster Builds, Faster Feedback!
- Easy Install and Everyday Use
- No Need to Change Your Tests
- Leverage Your Existing Computing Infrastructure
One of the best parts about Selenium Grid is you can download and have it running in 10 minutes or less. Selenium Grid comes out-of-the-box with Amazon EC2 support.
Jennifer Bevan, Lead from Selenium Farm Project at Google
As of Friday, Google has over 50 teams running over 51K tests per day on internal Selenium Farm. 96% of these tests are handled by Selenium RC and the Farm machines correctly. The other 4% are partly due to RC bugs, partly to test errors, but isolating the cause can be difficult. Selenium has been adopted as the primary technology for functional testing of web applications within Google. That's the good news.
The bad news is Google is pushing the limits of Selenium. Using Selenium (RC + Core) at this scale exposed certain issues not originally anticipated as high-impact. IE/XPath slowness has a high impaction given that can only run one test at a time. Tests can cause many conditions from which RC cannot easily or automatically recover (for example, tests that don't call stop() in every exit path). Unexpected browser dialogs, popups, etc. eventually cause timeout exceptions.
Googlers expect that RC will work for the most part, but they want it to be more reliable, with better performance. So they have:
- Created utility methods to improve performance when examinging large tables, overlaying domain-specific languages, etc.
- Deployed retry policies based on failure reason.
Looking to the future - Google has not yet reached their expected usage of Selenium RC. Some projects cannot use the Farm until RC supports session-level configuration (not server-level). Many just want RC to be more reliable. So Google will:
- Continue to contribute to RC, Core and to user-created helper libraries.
- Keep doing so until all failed tests are not Selenium's fault.
Dave Astels, Google (Driving Selenium with RSpec)
Using RSpec, you can create a very easy-to-read Story with Scenarios that can be read (and likely written) by practically anyone. Dave then uses a small script to load up the stories and run them in Selenium. When he runs the script, the scenario is spit out and test pass/fail information. Learn more at rspec.info.
Alex Chaffee, Mad Scientist at Pivotal Labs (The Selenium/Ruby Project that Must Not Be Named)
What is Polonium? It's also known as Selenium RC Fu or Selenium On Rails 2 or Funkytown. It has simple extensions to Selenium RC:
- element assertions
- launching/managing servers locally
Blackbox testing you're sitting out of the box and send in stuff. In whitebox testing, you get to open up the box and look at stuff. With Selenium, you can do Graybox testing, where you are doing blackbox testing (against the UI) and querying your database (or other resources) at the same time.
Dan Faulich, Sr. QA Engineer at Redfin Corporation (How Not to Run a Successful Open Source Project)
Shinya Kasatani, Developer of Selenium IDE
Selenium IDE is a Firefox extension that can record and play back tests in your browser. It can translate the recorded tests to many languages.
Selenium IDE 1.0 adds support for TestSuites. Another new feature is better recording features - it detects when the DOM is modified. Shinya has a demo where he uses the new IDE to test the Dojo Dijit Theme Test Page. Apparently, this doesn't work in the current version.
The Goal of Selenium IDE is to get more people interested in test automation of web applications and to help their projects to be successful.
Haw-bin Chai, Developer at CommerceHub
XPath is a powerful selection took, but it'd be great if we could use something like "article 5" instead of the cryptic //table/tbody/... syntax.
getLocator(), then you add a page set and add an element - all in JSON Format.
Simon Stewart on Web Driver
Soon there won't be a WebDriver project ... because it will be part of Selenium!
I had a great time learning more about Selenium and how most of its problems will be solved in the near future. Google has standardized on Selenium for it's Web Integrating Testing, do you think your company will follow suit?
Update: Videos of this event have been posted.