Next Generation Observability: An Architectural Introduction

In my past life, I spent many hours researching, creating, explaining, and publishing portfolio architectures across a collection of application development, domain verticals, infrastructure solutions, and hybrid cloud domains. Most of these concentrated on the application layers and their usage of the infrastructure.

Then, I transitioned into the cloud native observability space, and observability became the guiding light on my learning path. This quickly led to the realization that the same solution mapping that the previous portfolio architectures brought to organizations struggling with solving hard problems also applied to the observability world. It's just a matter of a different angle from which we look at those solutions.

Early in 2023, I introduced the first changes to an existing Financial Payments architecture, publishing a short series where observability was included in the solution diagrams. As my learnings in the observability domain continued, this architectural need for a generic view of how to design a cloud-native observability solution for any organization remained on my mind.

For the last few months, a small group has been digging into how to define the next-generation observability architecture. This article is the first introduction to our findings.

This next-generation observability architecture covers the highest-level use cases that are the foundations needed for such an architecture, one that encompasses all the functionality and approaches to observability. It’s tackling the challenges that cloud-native observability at scale presents with all elements present to provide organizations with control over their telemetry data and associated cloud-native costs.

The Process

You might be asking, what's in these architectures?

It's always a challenge when creating architectural content, trying to capture real-world stories into a generic enough format to be useful without revealing any organization's confidential implementation details. We are basing these architectures on common customer adoption patterns. That's very different from most of the traditional marketing activities usually associated with generating content for the sole purpose of positioning products for solutions. When you're basing the content on actual execution in solution delivery, you're cutting out the marketing chuff. 

This observability architecture provides us with a way to map a solution using open-source technologies focusing on the integrations, structures, and interactions that have proven to work at scale. Where those might fail us at scale, we will provide other options. What's not included are vendor stories, which are normal in most marketing content. Those stories that, when it gets down to implementation crunch time, might not fully deliver on their promises.

Let's look at the next-generation observability architecture and explore its value in helping our solution designs. The first step is always to clearly define what we are focusing on when we talk about the next-generation observability architecture. The following statement is our guide for this architecture:

"The next generation observability architecture is an approach for organizations tackling the challenges of telemetry data and their associated cloud native costs." 

This will be the guiding principle for our research into how organizations are looking to create their own cloud-native observability solutions.

The approach is taking existing solutions in this space, collecting that content, sorting through implementations, reading available documentation, and possibly speaking with domain experts.

Available Assets

This series is about sharing the results of our research into a next-generation observability architecture, but there are project assets and diagram collections that can also be shared. 

The first place to watch is the Observability Architectures collection repository, where all project examples are hosted. Each project is a collection of individual images and diagrams with links to open them directly in the free online diagram tooling.

For example, if we select the Next Generation Observability project in this collection, it provides an overview of all the files, images, and diagrams on the main page. We can browse from high-level marketectures to logical diagrams to schematic diagrams for each use case, and images are provided for your use of each asset. 

Finally, at the bottom, there will be a Download Diagrams section with a link to automatically open the available architecture in a new browser window in the hosted diagram tooling project. Give it a try, and feel free to explore the collection. This should allow you to get started much quicker than from scratch if you can kick-start a project with existing diagrams. Should you design your own diagrams, please contribute to the project file by raising an issue with the file attached. 

Exploring More

Get started exploring the next-generation observability architecture as we cover everything from the introduction, common architectural elements, and individual use cases within the architecture:

1. An architectural introduction (this article)
2. Common architectural elements
3. Automation
4. Automatic instrumentation
5. Configuration management
6. Container management 
7. Cost management
8. Digital experience monitoring (DEM)
9. Incident response
10. Profiling
11. Real user monitoring (RUM)
12. Synthetics

Catch up on any of these articles by following the links provided in the above list. Next, we will be exploring the generic common architectural elements that are part of the next-generation observability architecture.