OpenTelemetry's OpAMP Potential Far Beyond Supporting Collectors

It is good to hear that the CNCF OpenTelemetry project has become a graduated project (7 years after Fluentd, which encapsulates Fluent Bit, which is also OTLP-compliant). Admittedly, OpenTelemetry is a far larger project, as it provides tooling for tasks such as code auto-instrumentation for applications and many other considerations, such as:

• Open Agent Management Protocol (OpAMP)
• Profiles
• Semantic Conventions (descriptions of how to carry data for different contexts from Generative AI to Functions as a Service).

While the Semantic Conventions will help organizations implement Observability around AI, given its non-deterministic nature, it makes observability essential for understanding more about what has happened. The Open Agent Management Protocol is less notable today than the others, but it has the potential to impact solutions far beyond the OpenTelemetry space.

For the rest of this article, I'm going to explore why I think this is the case, and to do that, we need to also understand the key capabilities that OpAMP enables.

What Actually Is OpAMP?

Let's start with the approach being adopted with OpAMP. Just as the OpenTelemetry project focused on getting an explicit definition and broad support for the OpenTelemetry Protocol (OTLP) before the OTel Collector development got real momentum. Today, OpAMP is essentially a protocol definition, with configuration managed gRPC definitions that can be generated to code stubs for nearly any language through Protobuf.

The spec addresses the issue that OTel deployments typically have — lots of Collectors that are very widely distributed, and there is a need to centrally understand and manage the full picture. For example, handle scenarios such as:

• Being able to examine the operational state in depth by being able to retrieve information about the current configuration being used.
• Ability to control and command the agent to do things from a central location.
  • Common actions such as updating the config
  • Custom commands for specific. Use cases
• Manage security considerations such as:
  • alive and healthy down to a granular component level
  • Manage the way agents  can be identified and accepted dynamically
  • Certificate management, e.g. cert rotation
• Orchestrate the deployment of plugin features, patches, and related assets.

This could easily become monolithically large, but the clever decision is that the protocol includes a handshake in which the client and server can declare what they can and can't do, and allow custom commands to be exchanged. The minimum capabilities expected by OpAMP are small — enough to know what is being watched and that it is alive.

This flexibility is where the key value for going beyond OpenTelemetry lies. We can add custom commands and use the same framework to control OTel Collectors just as easily as having a custom command handler that could command a database to change its memory allocation to cache, for example. 

Flexibility can often lead to complexity in message exchanges. While the payloads provide a lot of freedom to describe custom commands, for example, the exchanges aren't complex. The following illustrates one of the most complex exchanges, where authentication between the agent and server is done through certificates, and we need to refresh the certificates that the agent is using.

Configuration deployment is not limited to OTLP, so you can dispatch configuration as you wish. This may require you to introduce a custom message so the client/agent understands what to do with the configuration file.

Given the OpAMP name, it would be easy to assume that this does mean we have to deploy a new process. The documentation presents two models in which the agent logic can be embedded into the process: either as the OTel Collector or as a Supervisor, which, visually at least, points to the process that launches the thing being managed. Both of these have varying degrees of invasiveness. There is a variation of the Supervisor, which we've called Observer, where the agent knows how to figure out which process they need to monitor through interacting with the host; as a result, it doesn't impact (even to change how the process is launched) the component being observed.

In our ChatOps model, we include a set of flows in Fluent Bit (or Fluentd) that perform housekeeping activities and provide feedback through observability channels. To make OpAMP easier for us, we have server logic that can receive instructions directed to a specific agent. The agent then triggers Fluent Bit, for example, to have an observability flow run that can include housekeeping activities. While we could simply give the agent a custom action to perform tasks directly, this separation of concerns reduces risk by isolating permissions.

There Are Many Ways to Do That Already

It is true that most of these things can be done within a Kubernetes ecosystem. BUT how many companies have their entire software estate purely running on K8S? How many pods do you really want to blow away just to tweak the log level of one package while you look into a possible app error, particularly if your pods are a first-stage transition to a Microservice ecosystem and are still very monolithic? What is interesting is that the OTel blog offers a way to combine OpAMP with the K8S control plane (although today there is a more direct K8S Operator).

Yes, we can do a lot of this with GitOps, but not all of it. Not to mention, just because you've pulled the latest configuration doesn't mean the recipient has successfully adopted it.

Again, we can go through the argument about how you could solve the problems with Chef or Puppet, etc.

What all these arguments miss is that the control plane protocol is unique to the tool concerned. Here, OpAMP has flipped it on its head and said, "This is how you communicate; how you implement is your choice." Currently, the OTel Git repo contains the basic skeleton code. But I can imagine that over time, more flesh will be added to this, just as we saw with the Collector.

Essentially, we're talking about the potential for a Universal Control Plane, but where the agents can be from any provider, and the control plane could be from another.

This is perfect, as we can apply the same mechanisms in a K8S environment, on bare-metal HPC, on VM-based platforms that host monolithic apps, and for handling devices that may drop in and out of contact.

Coming back to the Observability space again, this means that we can start to think about a single control plane that could manage OTel collectors, Fluentd, Fluent Bit, Elastic Stack and Beats components and manage it from a central plane - so now as our signals pass through possibly multiple points before reaching the Log Analytics, metrics dashboards etc we can see that each stage and each part of an end to end is operating.

One Concern

The biggest concern is possible adoption. Unlike OTLP, which focuses on standardizing how signal sources generate data and route it to the backend, where the core value comes from commercial vendors able to manage their fleets of collectors and perform clever analysis. OpAMP can be seen as commodifying the mid-tier. If a vendor makes their collector OpAMP-compliant, I could come along and start managing that agent (with some caveats around custom commands). If a vendor is charging per agent, they may well see them 'disappear' as the provided control plane no longer manages them.

But for Open Source solutions such as Fluentd, Fluent Bit, Beats, and the Elastic Stack, we can now consider simplifying our enterprise.

A Protocol Is Not a Full Implementation 

This is true, and that is where my project comes in. We have started implementing the OpAMP capabilities (and taking advantage of its custom commands), as it provides an ideal vehicle for our ChatOps idea. We've stayed within the Observability space (largely to make it easy for audiences to connect to the implementation). We're implementing the internals of the client and server in an extensible way, so it should be possible to use OpAMP with our initial targets of Fluent Bit and Fluentd, but we intend to expand to support the Elastic Stack/Beats and the OTel Collector.

Using the protocol, we've addressed the basic needs: what is the agent type, and what configuration is it operating under? That extends to the idea that on the server, we should be able to view the configuration file, ideally maintain it, and deploy it. The latter suggests the need for a configuration editor. So we have built something to support that, which can be entirely configuration-driven and can work with StreamingSQL and Lua scripts.

To learn more about the project, try it out, and understand what ChatOps is, go here.

A feature or two to help the Fluent stack

To deliver some bonus value, we need to understand and potentially amend the Observability agent's configuration. So we have a configuration-driven editor which can be run standalone or as part of the control plane.

Conclusion

Some parts of the OpAMP protocol have the payload marked as 'Beta' or 'Development'. This may be off-putting until you realise that the OTLP definition was in a 'Beta' state for a very long time before being finally signed off. So don't let this discourage you. The Development message objects are more at risk of change, but this is driven by practical insights, and I don't anticipate any major changes.

As for our project implementing OpAMP, it is still young and would benefit from people trying it out and feeding back. There are a lot of possible features we're considering, and we'd welcome feedback and to hear if people have tried to leverage the points of extension.

Resources

• OpAMP official spec
• Fluent-OpAMP in GitHub and a landing page (we may have called the project Fluent (as in Fluentd, Fluent Bit), but we're starting to think beyond those two 'collectors').
• ChatOps
• My blogging on OpAMP
• Fluent Bit
• Fluentd