Mastering Observability in 10 Minutes Using OpenSearch

Observability has become a key component in software development as it enables the best customer experience by ensuring system health and performance and detecting systemic issues proactively. However, getting started can often feel overwhelming. OpenSearch simplifies this by providing an open-source, scalable solution for logging, metrics, and visualization.

In this article, we’ll walk through setting up observability in 10 minutes using OpenSearch Observability. No complex jargon, just simple steps to get you started with real-world examples.

What Is Observability?

Observability is the ability to understand a system's internal state by analyzing its external outputs, such as logs, metrics, and traces. Below are the three key telemetry signals:

• Logs: What happened?
• Metrics: What is the system’s performance?
• Traces: How did events flow through the system?

For a deeper dive into observability basics, check out What is Observability? A Practical Overview on DZone.

Why OpenSearch and Data Prepper for Observability?

Here’s why this combo is ideal:

1. Open-source flexibility: Scale with your requirements.
   
2. Simple setup: No complex configurations.
3. Real-time insights: Visualize logs instantly.
4. Streamlined data pipelines: Ingest and prepare data with ease.

For comparisons with other tools, check out Choosing the Right Observability Platform.

Step 1: Quick Setup of OpenSearch

Follow the steps to get OpenSearch running quickly using Docker. Open your terminal and run:

docker run -d --name opensearch-node \
  -p 9200:9200 -p 9600:9600 \
  -e "discovery.type=single-node" \
  -e "plugins.security.disabled=true" \
  opensearchproject/opensearch:latest

This single command sets up your environment. You can now access OpenSearch at http://localhost:9200. 

If you’re new to Docker, you might find Getting Started With Docker on DZone helpful.

Step 2: Set Up Data Prepper for Data Ingestion

Data Prepper is an open-source tool for ingesting logs, metrics, and traces to OpenSearch. It’s lightweight and easy to configure for observability pipelines.

Install Data Prepper

Download and start Data Prepper using Docker:

docker run --name data-prepper \
  --network=host \
  opensearchproject/data-prepper:latest

This command spins up a Data Prepper instance listening for log data.

Configure Data Prepper

Create a data-prepper-config.yml file with the following content to process logs:

pipelines:
  my-pipeline:
    source:
      file:
        path: /var/logs/application.log
    sink:
      opensearch:
        hosts: ["http://localhost:9200"]
        index: "application-logs"

Place this configuration file in the same directory where you run Data Prepper.

Restart the container with the config file mounted:

docker run --name data-prepper \
  -v $(pwd)/data-prepper-config.yml:/usr/share/data-prepper/data-prepper-config.yml \
  --network=host \
  opensearchproject/data-prepper:latest

Step 3: Generate Logs

Create a Python script (app.py) to simulate application logs:

import logging
import time

logging.basicConfig(
    filename='application.log',
    level=logging.INFO,
    format='%(asctime)s - %(levelname)s - %(message)s'
)

actions = ["User logged in", "Page viewed", "Error occurred"]
while True:
    for action in actions:
        logging.info(action)
        time.sleep(1)

Run the script to generate logs. Data Prepper will ingest them and send them to OpenSearch.

Step 4: Visualize Logs in OpenSearch Dashboards

1. Open OpenSearch Dashboards at http://localhost:5601.
2. Navigate to Discover and create an index pattern for application-logs.
3. Explore the logs and create visualizations.

Step 5: Real-World Use Case: E-Commerce Monitoring

Simulate an e-commerce use case by extending the Python script to log user actions like login, add-to-cart, and checkout:

import random

users = range(1, 10)
actions = ["login", "add_to_cart", "checkout"]

while True:
    user = random.choice(users)
    action = random.choice(actions)
    logging.info(f"User {user} performed {action}")
    time.sleep(2)

Visualize:

• Track frequent actions using bar charts.
• Identify anomalies, like checkout errors, using filters.

Step 6: Alerts and Automation

Set up alerts to proactively monitor system health:

1. Go to Alerting in OpenSearch Dashboards.
2. Create a rule to trigger alerts for specific log patterns (e.g., frequent errors).

Conclusion

You have set up OpenSearch and Data Prepper to ingest and visualize logs in less than 10 minutes. Observability for your projects is now at hand, from small to gigantic. Now that you've got a good handle on logs, add metrics and traces to complete your full-stack observability.

OpenSearch's ecosystem is both novice-friendly and powerful. Start now, and let your systems talk to you.