Observability Agent Architecture

Observability agents are essential components in modern software development and operations. These software entities act as data collectors, processors, and transmitters, gathering critical telemetry data from applications, infrastructure, and network devices. This data is then sent to centralized observability platforms where it can be analyzed to gain valuable insights into system performance, identify issues, and optimize operations.

By efficiently capturing, processing, and transmitting logs, metrics, and traces, observability agents provide a comprehensive view of system health and behavior. This enables organizations to make informed decisions, improve application reliability, and ensure compliance with relevant regulations.

Architecture

From an architectural perspective, observability agents provide a more efficient and maintainable solution for instrumenting applications compared to manual code injection. By automating the collection and processing of telemetry data for common use cases, these agents reduce the overhead on application developers, promote code modularity, and simplify the management of monitoring infrastructure.

Auto-instrumentation techniques vary based on language runtime. Two primary approaches are weaving and monkey patching, often used in conjunction.

1. Weaving: Typically employed in statically typed languages like Java, weaving involves modifying bytecode at compile time, load time, or runtime. Tools like AspectJ or the Java Instrumentation API facilitate this process, allowing for granular control over injected behavior.
2. Monkey patching: Common in dynamically typed languages like Python and JavaScript, monkey patching dynamically modifies or extends existing methods or functions at runtime. This technique is often used for quick fixes, adding new behavior, or collecting telemetry data without altering the original codebase.

Observability agents can significantly impact both the functionality and performance of an application. Therefore, it's crucial to adhere to sound architectural practices when developing and deploying them.

• Prioritize security: Protect sensitive data by implementing measures like data sanitization, obfuscation, and user-controlled data collection options. 
• Minimize overhead: Avoid introducing performance bottlenecks or stability issues. Optimize data collection and processing for efficiency. 
• Handle data efficiently: Manage large volumes of telemetry data effectively using techniques like batching, compression, and asynchronous processing. Set limits on the agent's resource usage to prevent excessive impact on the application. 
• Sample data judiciously: If necessary, sample data to control resource consumption. Carefully select the data to collect or retain based on priorities like performance monitoring, error tracking, or user behavior analysis.

To earn customer trust, focus on these areas.

• Comprehensive coverage: Collect logs, metrics, and traces from all relevant components of the system.
• Standardization: Use standardized formats and protocols (e.g., OpenTelemetry) for data collection to ensure compatibility and ease of integration.
• Simple integration: Provide easy integration methods, such as a single command or configuration file, to reduce the effort required to deploy the agent.
• Configuration management: Support dynamic configuration changes without requiring application restarts, allowing for flexible and responsive observability.
• Self-monitoring: Implement self-monitoring capabilities to track the performance and health of the observability agent.
• Error handling: Ensure robust error handling and logging within the agent to facilitate troubleshooting and maintenance.

Let's explore examples of weaving and monkey patching in action.

Weaving

Consider a Java observability agent using OpenTelemetry. This agent typically employs bytecode instrumentation to dynamically modify your application's classes at runtime. Here's a breakdown of the process:

Agent Attachment

Attach the agent to your Java application using the -javaagent flag:

java -javaagent:/path/to/opentelemetry-javaagent.jar -jar myapp.jar

Bytecode Manipulation

The agent intercepts the loading of your application's classes and modifies their bytecode. This allows the agent to insert additional code into existing methods.

Java’s premain method is a special entry point used by Java agents for bytecode manipulation. This method allows agents to modify the bytecode of classes before they are loaded into the JVM. Here’s a detailed look at how it works:

Premain Method

The premain method is the entry point for a Java agent. It is called before the main application starts. The method signature looks like this:

public static void premain(String agentArgs, Instrumentation inst);

• agentArgs: Arguments passed to the agent
• inst: An instance of the Instrumentation interface, which provides methods to transform classes

Example Using ASM for Bytecode Manipulation

• Define the Agent class:

import java.lang.instrument.Instrumentation;
import java.lang.instrument.ClassFileTransformer;
import java.security.ProtectionDomain;
import org.objectweb.asm.*;

public class MyAgent {
    public static void premain(String agentArgs, Instrumentation inst) {
        inst.addTransformer(new MyClassFileTransformer());
    }
}

• Implement the ClassFileTransformer:

class MyClassFileTransformer implements ClassFileTransformer {
    @Override
    public byte[] transform(ClassLoader loader, String className, Class<?> classBeingRedefined,
                            ProtectionDomain protectionDomain, byte[] classfileBuffer) {
        if (className.equals("com/example/MyClass")) {
            ClassReader cr = new ClassReader(classfileBuffer);
            ClassWriter cw = new ClassWriter(cr, 0);
            ClassVisitor cv = new MyClassVisitor(Opcodes.ASM9, cw);
            cr.accept(cv, 0);
            return cw.toByteArray();
        }
        return classfileBuffer;
    }
}

• Modify the Class with ASM:

class MyClassVisitor extends ClassVisitor {
    public MyClassVisitor(int api, ClassVisitor classVisitor) {
        super(api, classVisitor);
    }
    @Override
    public MethodVisitor visitMethod(int access, String name, String descriptor, String signature, String[] exceptions) {
        MethodVisitor mv = super.visitMethod(access, name, descriptor, signature, exceptions);
        if (name.equals("myMethod")) {
            return new MyMethodVisitor(Opcodes.ASM9, mv);
        }
        return mv;
    }
}

class MyMethodVisitor extends MethodVisitor {
    public MyMethodVisitor(int api, MethodVisitor methodVisitor) {
        super(api, methodVisitor);
    }

    @Override
    public void visitCode() {
        super.visitCode();
        // Insert custom bytecode here
        mv.visitFieldInsn(Opcodes.GETSTATIC, "java/lang/System", "out", "Ljava/io/PrintStream;");
        mv.visitLdcInsn("Hello from ASM!");
        mv.visitMethodInsn(Opcodes.INVOKEVIRTUAL, "java/io/PrintStream", "println", "(Ljava/lang/String;)V", false);
    }
}

• Packaging and running the Agent:
  • To run the agent, package it into a JAR file with a MANIFEST.MF file that specifies the Premain-Class: Premain-Class: MyAgent
  • Then, run your application with the agent:

java -javaagent:path/to/agent.jar -jar your-application.jar

Monkey Patching Pattern

Library Detection

The agent detects commonly used libraries and frameworks within the application, such as Express, HTTP, and database clients. This is done by scanning the application’s dependencies and runtime environment1.

Hooking Into Libraries

Once the relevant libraries are identified, the agent hooks into their internal methods. This is achieved by wrapping or monkey-patching functions to insert instrumentation code. For example, it can wrap the HTTP request handler to start and stop traces around each request.

class MyClass:
    def original_method(self):
        print("Original method")

def new_method(self):
    print("New method")

    MyClass.original_method = new_method
obj = MyClass()
obj.original_method()  # Outputs: New method

Conclusion

Observability agents play a pivotal role in modern software development, providing invaluable insights into application performance and behavior. By adhering to architectural best practices, developers can ensure that these agents are efficient, secure, and maintainable. 

Additionally, exploring patterns and technologies like OpenTelemetry and eBPF can further enhance the capabilities and efficiency of observability agents. In future posts, we will delve deeper into these topics, exploring their benefits and practical applications.