3 Way Ops Can Help Devs: A Developer Perspective

In a lot of companies, it seems, software developers are from Mars, and operations folks are from Venus. Or perhaps you see the two departments as vampires and werewolves. Whatever analogy floats your boat, dev and ops each has its own goals, metrics for success, and team culture. Often the biggest issue is that ops is in the business of making things predictable and unchanging, while developers are in the business of changing everything.

Those opposing priorities can create friction, and make it harder for dev and ops to communicate freely. Despite the industry’s ongoing efforts to bring the communities together, developers continue to grumble about ops, who simultaneously grumble about devs. That’s particularly true in traditional organizations, while DevOps organizations are already making progress to bridge that divide.

In the meantime, though, grumbling doesn’t help to resolve the tension. Understanding does. So I asked several developers the question at the top of this post (in some cases we’re using only their first names to enable them to speak freely). I also asked ops folks what they’d change about developers; you’ll read about that in an upcoming post. Three themes clearly emerged from their answers:

1. Mutual respect is required.
2. Serving the end user is paramount.
3. Ops code is important, too.

1. It’s All About Mutual R-E-S-P-E-C-T

It helps to remind one another (and ourselves) that ops and devs have different powers. They are complementary, not adversarial. So even if you hate the idea of more meetings, it’s important to talk with (not just at) each other.

Maximum efficiency is obtained when both dev and ops understand all the components required to run applications in production and the security and uptime implications of each one, says a developer named Adam. “The best way to achieve this is with very high bandwidth communications. So, ops should embed on the dev team.” That may take a few workarounds, Adam admits, since some organizations require the teams to remain separate. “So, you have a small team to fulfill that gatekeeper function and they interface with the ops people embedded on the dev teams,” he says.

Without regular communication, developers and operations interact only when there’s a problem. That can result in a power struggle in which the sole purpose seems to be identifying who’s at fault.

2. You Gotta Serve Somebody

It’s important to remember that the system is there to serve users, not to serve sysadmins. “Too often, development and operation teams forget they are working for the customer,” says Jeff Huckaby, CEO at rackAID, a managed service provider focusing on Linux-powered operations. “They blame each other for outages, poor website performance, deployment obstacles, or a myriad of other issues. Neither silo wants to accept responsibility.”

Create a culture of responsibility, says Huckaby, and enforce it using objective metrics. “Rather than placing blame, challenge the responsible team for maintaining the customer experience,” he suggests. That’s important, because “when the finger-pointing starts, the customer loses.”

I’ve listened to developers complain about operations staff who seem to think that keeping the machines running is the point of the exercise. In the frustrated dev view, ops forgets that what matters is to create and deliver software that serves the users (and by extension benefits the company). And to those developers, anything that makes it harder to achieve that goal is sinful. Anything.

Developers may recognize that ops have good intentions—often around making the system more secure—but they resent it when the implementation gets in the way of getting their work done. Or, worse, when the process becomes an arbitrary barrier to the user.

As a developer named John writes, “Software development sometimes needs to be allowed to bend the rules/regulations in order to operate efficiently/quickly. Too many times, the rules (e.g., who has access, when, what can be installed, etc.) cause ridiculous delays in cycle time for development or support.”

Honest developers often confide that they see arbitrary barriers as nothing more than challenges to climb over. Sure, the developers’ workarounds could be potentially dangerous; but if ops doesn’t know about it, then it’s not a problem, right?

The better approach, John said, is for ops to bring devs into the fold, saying: “Here’s the problem we need to solve. We’re currently planning to [implement this security policy] to address it. How would that affect your work? Do you see a better way?” Then both teams have the opportunity to help one another—or at least understand each others’ priorities.

3. Ops Code is Software, Too

Ironically, devs also complain that ops too often treat the systems under their purview in a more slapdash manner than dev does.

Both developers and operations write software, they both worry about performance and stability, and they both worry about security. When ops is managing VPNs or writing scripts, they are managing a software process. But developers sometimes run into situations where ops doesn’t follow a methodology—including creating a process to, say, test changes on staging servers.

“As a developer, I want to put out bug-free code,” says Dominic Acito atDomsTechBlog.com. “However, I am also tasked with making new features work.”

The changes that ops make need to go through a quality control process, just as the devs’ original code does. As Acito mentions, it doesn’t help when ops makes an otherwise reasonable change at the last moment, without regard to dev’s production schedules: “Hey, let’s upgrade the server the night before with the new version without asking the developers first.”

It gets worse. I’ve personally seen sysadmins make changes to production without testing to prevent a system crash. (Wanna guess how I know about them?)

Put it all together and there are countless reasons for the two groups to talk with each other about what they’re doing and what they’re planning. Who knows? Perhaps they can even help each other!

These are just a subset of the frustrations developers feel about their ops teams. Here’s a call for developers to let us know on Twitter (@newrelic) about what spells you would cast upon the ops team with your own magic wand.