[This article was written by Marten Terpstra.]
Triggered by a discussion with a customer yesterday, it occurred to me (again?) that network engineers are creatures of habit and control. We have strong beliefs of how networks should be architected, designed and build. We have done so for long times and understand it well. We have tweaked our methods, our tools, our configuration templates. We understand our networks inside out. We have a very clear mental view of how they behave and how packets get forwarded, how they should be forwarded. It’s comfort, it’s habit, we feel (mostly) in control of the network because we have a clear model in our head.
I don’t believe this is a network engineering trait per se. Software engineers want to understand algorithms inside out, they want to understand the data modeling, types structures and relationships.
Many of us know the feeling. Something new comes around and it’s hard to put your head around it. It challenges the status quo, it changes how we do things, it changes what we (think we) know. When we are giving responsibility of something new, there is a desire to understand “it” inside out, as a mechanism to be able to control “it”.
I will bet you almost all good network engineers have a complete view of the network in their head, or at least the portion they are responsible for. They know exactly what it looks like, understand how things are connected, understand how traffic is supposed to flow. It makes it easy to detect subtle changes, they usually point to something not being right. To that purpose we have gone through great lengths to understand all the protocols that govern connectivity.
Challenging the Status Quo
When solutions come around that do not easily fit into the model we know, we prod and poke. Initially it for engineering curiosity, but ultimately it comes back to wanting or needing to understand how it works in great detail. We want to be able to create a model of the network in our mind. We want to be able to grab a piece of paper and explain how things are expected to work and why.
Totally understandable, but it is extremely limiting. Our desire to reproduce in our head what our network devices do in the real world will limit our solution space. Like it or not, except for those extremely smart few, modern day algorithms and the compute power available can create more optimal solutions than we can put together in our head. Or that piece of paper.
Most of the rules that govern our networks are singular in their metric space. We have a single metric that describe resources in our network and create behavior based on that. Except perhaps BGP which has a multitude of metrics, but even there we nicely ordered them in a way that we can simulate the decision process in our minds.
As Mat mentioned yesterday in his blog, the solutions we create have to evolve to much more abstract and descriptive requirements that are based in business policy. These are by definition multi dimensional, and in many cases conflicting, or at least interacting with each other. Fairly complex policy and rule engines are being developed to interpret these business policies and ultimately bring them back to the basic building blocks our datacenter resources can understand.
That is no longer something you can simulate in your head. For one rule, perhaps. Maybe even 2 or 3, or 10. But once you get to 1000s of resources and 100s of business rules that govern them, our heads are simply not big enough to compute the needed or expected result.
I am not suggesting you should simply trust whatever magic answer the policy engines (and the controllers as receivers of the resulting resource assignments) come up with. But we should realize that those results may not always be simple (or possible) to validate in our head considering the complexity of the problem that has been solved. And that is something we need to get used to. As vendors we need to provide the tools so you can carefully examine the results we created. We need to make sure you can look at the results before you or your controller implements them onto your network.
Of course the results need to look reasonable, but we may have to let go of the notion that we can recreate every choice made by hand in normal time. This is hard. It will take getting used to. But like so many other things that we as users cannot necessarily recreate, we can get there. We have to.
[Today's fun fact: Leonardo da Vinci could draw with one hand while writing with the other. Most kids today have a similar skill: play video games while texting friends and ignoring parents.]