Automation for the People (because Everybody Doing It Manually Hurts)

Today I'm kicking off a new emphasis here at Agile Zone on software project automation. The astute reader will recognize the double reference to R.E.M.'s 1992 4X platinum album "Automatic for the People." That album contained their hit single "Everybody Hurts." And that's precisely what I want to talk about in this kickoff article - the pain that all of us suffer from "doing things manually" on software projects. In the coming days we'll be looking at practical ways that you can eliminate this pain from your development efforts.

To start off, I'd like to address the immediate objection that this type of article just isn't needed. I mean, anybody that's a competent software developer has this down already, right? Wrong. While I have absolutely no data to back this up (Data? We don't need no stinkin' data!), it's my firm belief that nearly every developer out there is today doing something manually that they could automate. The fact of the matter is, they just haven't taken the time to do it.

In his book The Productive Programmer, Neal Ford recounts [1] the story of needing to parse a large legacy SQL file. In order to ease their task, Neal and his coworker decided to break the file into 1,000 line chunks. They quickly decided to automate the task rather than doing it by hand. The automation task took them roughly one hour, and they estimated that completing the task manually would have taken about ten minutes. Mathematically this looks like a loss until you find out that they eventually had to complete the same task five more times, almost reclaiming all of the time spent in automation. But according to Neal, that's not the important point. Instead, he asserts:

Performing simple, repetitive tasks by hand makes you dumber, and it steals part of your concentration, which is your most productive asset.

And:

Figuring out a clever way to automate the task makes you smarter because you learn something along the way.

I've realized the value of these points many times over. Recently I was faced with the necessity of loading multiple large spreadsheets of client data into one of our applications. For various reasons, the straightforward route of loading directly into the database tables was going to be a painful one. It quite likely would have taken me more than a week to get right, and I needed to get these data into the application within twenty-four hours. Shortly before biting the bullet and tackling the task via brute force (becoming "dumber" along the way), it hit me that I had a couple of tools in my toolbox that might just work.

The Selenium framework was developed at ThoughtWorks out of the need to test an internally developed Time and Expenses application. It has rapidly grown into one of the most heavily used tools for automated cross-browser testing of modern web applications. Part of what makes Selenium so powerful is its ability to record user interaction scripts via its Selenium IDE plugin for Firefox and then export those scripts to your programming language of choice (at this writing Java, Groovy, C#, Python, Ruby, Perl, and PHP are explicitly supported). A developer can then leverage the Selenium Remote Control (RC) server to drive most modern web browsers from these exported scripts. But that isn't the best part.

Because these scripts are now written in real programming languages, you can bring all of your best development techniques and third-party libraries to bear on your problem!

My thought process was as follows:

1. Record a Selenium script of me loading the first row of data into our application.
2. Export that script as a GroovySeleneseTestCase (a subclass of groovy.util.GroovyTestCase).
3. Leverage Apache POI's support (http://poi.apache.org/spreadsheet/index.html) for parsing Microsoft Excel spreadsheets to write a parser for the client data in Excel.
4. Wrap the originally exported Selenium logic in a loop over the spreadsheet's rows, substituting the parsed values for the manually typed data.

It took me a few hours to grasp some of the eccentricities of Selenium with which I wasn't familiar, and I had to dive head first into some rather ugly XPath expressions, but eventually I had a working solution. About twenty minutes after my success, all of the data were loaded. But as in Neal's case, the story didn't end there. I've been called upon to repeat the said data load several times. Unfortunately my problem shifted around a bit, as none of the spreadsheets were identical. However, every time I was only a few minutes away from a new working solution. I could then go back to real knowledge work while my computer happily performed the mundane work.

Not only had I built a reusable automated solution that saved me vast amounts of otherwise unproductive time, I also rapidly ramped up my skill level with Selenium. Double win!

So today we've looked at an isolated, small success story that saved one developer (me) a great deal of time and made him smarter along the way. Is it possible to scale that kind of success up to the entire project team? Absolutely. And that's just what we'll be doing in the coming weeks. I'm hopeful that you'll pick up some nuggets along the way and incorporate them into your own projects. Trust me, once you automate the mundane, sweetness follows.

References

[1] Ford, Neal. The Productive Programmer. O'Reilly, July 2008.