New Programming Techniques and the Productivity Curve

Though I love learning new programming techniques and technologies, I often struggle to make them a part of my normal development processes. For example, it took years before I finally started usingregular expressions on a normal basis. The reason? The productivity curve:

You may have seen a chart like this before; the productivity curve is used to show one of thechallenges of software delivery. The general idea is that when you first use a new product (or technology), your productivity always plummets in the short term. No matter how much better the new stuff is, you will be less productive using it at first, because you were more familiar with the old stuff. However, as you become more proficient with the new technology, your productivity gradually increases, and eventually your productivity will be greater than it was with the old technology (assuming the new technology is actually better).

So the key is to push through that initial dip, and eventually you get to the increased productivity that comes with mastering the new technology. Easy, right? Well, if your company just replaced your HR system, you have little choice in the matter; you’ll get through that productivity dip because you have no other choice.

However, if you’ve just decided you want to pick up a new programming technique, it’s a little more difficult. Before I got the hang of regular expressions, it was simply far faster for me to whip together something ugly using things like StringTokenizers and substring. Nobody cared if I was using regular expressions, but they did care if I took 10 times as long to complete a task.

On the rare occasions where I needed to make a change to existing regular expressions, I’d end up slogging through references and online tutorials to try and make sense out of it. Each time, after finally figuring it out, I’d tell myself that I was going to remember how these worked next time. But since I hadn’t made regular expressions a part of my standard toolchain, I’d forget everything I learned, and I would be lost again next time I needed to use them.

I eventually got the hang of regular expressions through forced practice and forced application.

Forced practice was straightforward enough: I dedicated time to reading references, running throughtutorials, and solving regex challenges. This practice was an important step in building the skills, but I had tried this to a certain degree before. I’d spend a few hours practicing regular expressions, and feel like I was getting the hang of it. But time would pass, and the next time I needed to fix a regex, I’d have forgotten most of it again.

The key for me was forced application of regular expressions. Like the poor HR staff who learn a new HR system because they have no other choice, I forced myself to use regular expressions any time they could be useful, even though it would take me longer than a quick hack with stuff I was already familiar with. Good unit tests helped a lot; though I was not confident in my ability to write bug-free regular expressions, I was confident in my ability to write thorough tests.

As the productivity curve predicted, I was definitely less efficient at first. But as I forced myself to use regular expressions instead of hacking something together more quickly, I gradually improved. And before long, I was past the productivity dip, having finally gotten the hang of regular expressions. Now they are an essential part of my development toolchain, and I am a better programmer.

I know this isn’t really earth shattering advice; my five year old son has figured out that you get better at something the more you do it. But when you find some new technology or technique you want to make part of your development process, I think it’s important to go into it recognizing that not only will it take time to master it, but you will almost certainly be less productive in the short-term than you were before. And I think that if you are prepared for the productivity dip and are willing to accept it, you can push through the difficult times and master it. In the long run, you will be a better developer for it.