We had a problem: our continuous integration environment had turned into a continuous breakage environment.  Around half the time, our builds would fail, either due to compilation issues or failed unit tests.  Each broken build generated an email to the team, and due to their frequency and verbosity, we began to tune these emails out.  That was even worse, as it meant multiple broken builds could pile onto each other before anyone noticed.  No longer did we have to fix a single bad change, we had to unwind multiple bad changes.  And this wasn't just a build issue: Each engineer would end up pulling down this broken code and breaking their own workspace.  It was like Voldemort himself designed this process to drive us all crazy. 

In all fairness, it was very, very complicated.  We had dozens of engineers contributing to 20 packages, with lots of explicit and transitive dependencies across these packages.  Many times, the build would work in person A's workspace and fail in CI because a dependency had changed or because they had changed some implicit behavior that another package relied upon.  It wasn't easy, in short.

Initially, several of us tried to solve these broken builds the same way: nagging emails.  I sent plenty of these on the sanctity of the build, laying out high-minded build tenets and describing what a big cost it was to the team when our builds failed.  I put all of this onto a wiki that I made all new hires read.  I was seriously considering buying a sandwich board, writing "THE BUILD APOCALYPSE IS NIGH!  RUN YOUR UNIT TESTS!" on it and walking up and down the halls.  None of this accomplished very much.

Then, one night, I got desperate.  After a particularly horrendous streak of failures, I sent a team email saying, "Guys, the build is all screwed up and we need to fix it immediately.  If you fix the build, I'll buy you doughnuts."  Yep, I resorted to bribery.  I'm sure there were other motivational tactics available, but man, I work hard already and there's a donut shop right down the street.

You know what happened next?  Someone fixed the build within the hour.  Once I saw that, I made it a new build policy: if you fix a build that you didn't actually break, you get a dozen doughnuts the next morning.

Now, my doughnut incentive plan hasn't fixed all broken builds forever, but it has led to immediate improvement.  Roughly every week, people stay late to fix a broken build specifically so they can claim their doughnuts.  Contrast that to the previous scenario, where people theoretically would've stayed late to fix a build in order to save team productivity.  That happened once a year, maybe.  Science shows that the doughnut incentive plan is 5,200 percent more effective in preventing broken builds.

I work with talented people who are paid well: They don't need me to buy them doughnuts.  And yet, once that tiny, tangible incentive is there, behavior changed dramatically.  What does all of this mean?  Why did this work better than lots of emails about how important successful builds are to team productivity?     I ... don't know.  Maybe it's the fact that it's a small sign of appreciation, something that can be shared with teammates.  Maybe it's the fact that the doughnut incentive is visible, kinda funny, and has since become baked into team culture.  Then again, maybe people just really like doughnuts.

There are bigger solutions here, like simplifying the build process, and we're working on all of that.  In the meantime though, our existing process needs to continue to work.  If you find your team in a similar situation, look to the doughnut shop.