Technology has moved into almost every part of our lives. The Internet of Things surrounds us with knick-knacks that rely on silicon — our apparel even contains microcontrollers. We are customers for sellers clamoring for market supremacy in a sea of feature-richness. We are faced with billboards lit up to display dizzying commercial color in endless motion. This spectacle is the essence of western industrial life.
How did a hairless ape make all of this happen? Is it Moore’s law? Is it consumer demand? Is it large industrial corporations? Is it simply clever marketing? It’s all of that, plus something so embedded that we can’t see it unless we take a step back: abstraction.
It’s what we all do every day without thinking. It’s how we make sense of the world, how we generalize and see the essence of things. This gives us the ability to manage complexity and novelty. It has been with us since we were foraging for food on the Serengeti. The thing we call abstraction is used to mentally collect like things and treat them as a group of entities. We classify, encapsulate and draw generalizations, and it’s incredibly powerful. When people say "simplify," what they mean is “manage complexity effectively.” Our systems aren’t getting any more simple, but we can manage the complexity so our systems don’t cause chaos. This is our discipline.
Now let’s take a look at abstraction and consider how it lets us make sense of all the complexity.
Abstraction gives us a creative landscape derived from the most mechanistic edifice imaginable: a zero and a one. Our digital world is built upon ridiculously simple mechanistic rules, yet the emergent properties of those rules can be transformed into something organic. When they are well designed, rules are a catalyst for creative capabilities, and they should serve to build more layers of robust, well-managed thought levers.
Every enabling function is based on well-conceived abstractions. Amazing things and unexpected capabilities emerge when abstracted patterns are overlaid on suitable domains. The transformation is often elegant to the point where it resembles art.
But it all has to be built on a solid foundation. The elemental rules must be remembered, no matter how high we climb. When you’re building a complex system, you’ll need to ask yourself these questions:
- What is good design and how can I tell if I have it?
- How do I manage complexity without adding chaos?
- How do I write solid predictable code?
- How might I deliver emergent properties from my engineering disciplines?
Abstraction and the Orchestra
Let’s take a look at an example of the principles of abstraction coming together to produce something grand. We all know about the orchestra, but have we looked at what is happening here with our abstraction hats on?
In the example above, the orchestra is an instance of hierarchy and abstraction. How does that relate to a business or a computer?
The orchestra is analogous to so many things we see every day, including IT systems, games, factories, cars or cities. They all have a common thread — collections and layers of rules, specifications, archetypes, and abstractions.
In an orchestra, the conductor is leading the system. He’s not playing an instrument, but he is directing the events and acting as quality control. He’s the highest level in a hierarchy of abstractions. If we look at the musical score, we see embedded instructions. We see notes written, but they are not the sounds. They encapsulate the rules of the music — the sequence, pitch, volume, and timing. But nothing happens until we get to the executive layer: the individual musicians and their instruments.
The score contains high-level sequential instructions necessary to execute in specific ways the lower-level instructions required to get the intended result. The musicians do that job, and are performing a role analogous to the Puppet agent on a target server.
Orchestrating with Puppet
The musician (Puppet agent) will execute parts (automation) within the score (high-level commands) on a particular instrument (operating system/end points). The musician translates the encoded information into sounds (instrument) necessary to have the audience appreciate the outcome (business function), all managed by the top layer (orchestration), and intended by the composer (IT service owner) to please the audience (business owner).
These abstractions enable useful hierarchies. Abstractions use interfaces between layers. Abstractions are built on trusted behaviours.
The system works because each layer is aware of the rules agreed between the adjacent layers. There is a common language between those layers: music. A transformation happens within each layer when something more specific is added. If the conductor had to play all the instruments himself, the outcome would be impossible. Here we see specialization and abstraction work to form a useful hierarchy, from general case to specific. There are technical elements, encoding elements, and transformational elements. Without these structures, we would not have an orchestra. Abstract layers link together via a common language, and internally take care of translation.
While we are all aware of this example on the surface, it is worth examining it in the light of our new awareness. This idea is as natural to us as breathing, but the sophistication of the abstractions would not have happened without recognizing the incremental disciplines involved. Humans built this musical result after years of evolved thinking. It is an example where far more has been realized because of imposed structural rules. It has yielded far greater returns than if we simply sat the entire group of musicians in a room and shouted, “Play!”
Instead, we sat the musicians down and had them agree to a set of operating principles over an agreed common target. Then we filled the agreed structure with the rules of musical notation (software) that we enabled and tested a priori, and we executed the sequence within them. The result is robust, recognizable and repeatable, and can deliver far more impact than random sound, or even one man trying to play each instrument individually.
Here’s the lesson. Good abstractions:
- Are robust.
- Depend on well-formed rules.
- Manage complexity.
- Need evolution to refine them.
- Facilitate useful hierarchies.
- Take care of translation between layers.
- Use common interfaces between layers.
- May yield or empower results that exceed expectation.
Be sure to look out for Part II where we look at bits, bytes, code, and Puppet.