Chaos Engineering for Microservices

As someone who works closely with distributed systems and microservices, I've seen firsthand how complex things can get once Kubernetes, Istio, and service meshes enter the picture. The shift to a cloud-native world is exciting, but it brings new challenges — especially around resilience.

We can't just hope things won’t fail — because they will. That’s where chaos engineering comes in. It’s a proactive way to build confidence in your system’s ability to handle real-world disruptions by intentionally injecting failure and observing how everything holds up.

As modern applications adopt microservices, Kubernetes, and service meshes like Istio, ensuring resilience becomes a critical challenge. Distributed architectures introduce new failure modes, which demand proactive testing to achieve high availability. Chaos engineering helps organizations identify and fix weaknesses before they impact users by introducing controlled failures to observe and improve system behavior.

For Java (Spring Boot) and Node.js applications, tools like Chaos Toolkit, Chaos Monkey, and Istio-based fault injection offer powerful ways to put chaos engineering into practice. Kubernetes-native chaos experiments — including pod failures, network latency injection, and region-based disruptions — allow teams to stress-test system stability at scale.

In this article, I’ll walk you through how we implemented chaos engineering across our stack using Chaos Toolkit, Chaos Monkey, and Istio — with hands-on examples for Java and Node.js. If you're exploring ways to strengthen system resilience, this guide is packed with practical insights you can apply today.

What Is Chaos Engineering?

Chaos engineering is a discipline designed to proactively identify weaknesses in distributed systems by simulating real-world failures. The goal is to strengthen application resilience by running controlled experiments that help teams:

• Simulate the failure of an entire region or data center.
• Inject latency between services.
• Max out CPU cores to evaluate performance impact.
• Simulate file system I/O faults.
• Test application behavior when dependencies become unavailable.
• Observe the cascading impact of outages on microservices.

By incorporating chaos engineering practices, organizations can detect weaknesses before they impact production, reducing downtime and improving system recovery time.

Chaos Engineering Lifecycle

The process of conducting chaos engineering experiments follows a structured lifecycle:

This lifecycle ensures that failures are introduced methodically and improvements are made continuously.

Chaos Toolkit vs. Chaos Monkey: Key Differences

Chaos Toolkit and Chaos Monkey are powerful tools in chaos engineering, but they have distinct use cases.

When to Use Chaos Toolkit?

• When working with Kubernetes-based deployments.
• When requiring multi-cloud or multi-language chaos testing.
• When defining custom failure scenarios for distributed environments.

When to Use Chaos Monkey?

• When testing Spring Boot applications.
• When needing application-layer failures such as method-level latency and exceptions.
• When preferring a lightweight, built-in solution for Java-based microservices.

Chaos Toolkit: A Versatile Chaos Testing Framework

Installation

For Java and Node.js applications, install the Chaos Toolkit CLI:

pip install chaostoolkit 

To integrate Kubernetes-based chaos testing:

pip install chaostoolkit-kubernetes 

For Istio-based latency injection:

pip install -U chaostoolkit-istio 

To validate application health using Prometheus:

pip install -U chaostoolkit-prometheus 

Chaos Monkey for Spring Boot

The below diagram illustrates how Chaos Monkey for Spring Boot integrates with different components of a Spring Boot application to inject failures and assess resilience. 

On the left, it shows the key layers of a typical Spring Boot application, including @Controller, @Repository, @Service, and @RestController, which represent the web, business logic, and data access layers. These components are continuously monitored by Chaos Monkey Watchers, which include Controller Watcher, Repository Watcher, Service Watcher, and RestController Watcher. These watchers track activity within their respective layers and enable Chaos Monkey to introduce failures dynamically. 

On the right, the diagram depicts different types of chaos assaults that can be triggered, such as Latency Assault, which introduces artificial delays in request processing; Exception Assault, which injects random exceptions into methods; and KillApp Assault, which simulates a complete application crash. By leveraging these chaos experiments, teams can validate how well their Spring Boot applications handle unexpected failures and improve system resilience. This visualization helps in understanding the failure injection points within a Spring Boot application and highlights how Chaos Monkey enables fault tolerance testing in real-world scenarios.

Installation

Add the following dependency to your Spring Boot project (pom.xml):

<dependency>
    <groupId>de.codecentric</groupId>
    <artifactId>chaos-monkey-spring-boot</artifactId>
    <version>2.5.4</version>
</dependency>

Enable Chaos Monkey in application.yml:

spring:
  profiles:
    active: chaos-monkey
chaos:
  monkey:
    enabled: true
    assaults:
      level: 3
      latency-active: true
      latency-range-start: 2000
      latency-range-end: 5000
      exceptions-active: true
    watcher:
      controller: true
      service: true
      repository: true

Running Chaos Monkey in Spring Boot

Start the application with:

mvn spring-boot:run -Dspring.profiles.active=chaos-monkey 

To manually enable Chaos Monkey attacks via Spring Boot Actuator endpoints:

curl -X POST http://localhost:8080/actuator/chaosmonkey/enable

To introduce latency or exceptions, configure assaults dynamically:

curl -X POST http://localhost:8080/actuator/chaosmonkey/assaults \   -H "Content-Type: application/json" \   -d '{ "latencyActive": true, "exceptionsActive": true, "level": 5 }'

Chaos Engineering in Node.js: Implementing Chaos Monkey and Chaos Toolkit

While Chaos Monkey for Spring Boot is widely used for Java applications, Node.js applications can also integrate chaos engineering principles using Chaos Toolkit and Node-specific libraries.

Chaos Monkey for Node.js

For Node.js applications, the Chaos Monkey functionality can be introduced using third-party libraries.

Installation for Node.js

To install the Chaos Monkey library for Node.js:

npm install chaos-monkey --save 

Basic Usage in a Node.js Application

const express = require("express");
const chaosMonkey = require("chaos-monkey");
const app = express();
app.use(chaosMonkey()); // Injects random failures
app.get("/", (req, res) => {
  res.send("Hello, Chaos Monkey!");
});
app.listen(3000, () => {
  console.log("App running on port 3000");
});

What does this do?

• Injects random latency delays.
• Throws random exceptions in endpoints.
• Simulates network failures.

Configuring Chaos Monkey for Controlled Experiments in Node.js

To have more control over chaos injection, you can define specific failure types.

Configuring Failure Injection

chaosMonkey.config.js:

module.exports = {
  latency: {
    enabled: true,
    minMs: 500,
    maxMs: 3000,
  },
  exceptions: {
    enabled: true,
    probability: 0.2, // 20% chance of exception
  },
  killProcess: {
    enabled: false, // Prevents killing the process
  },
};

Now, modify the server.js file to load the configuration:

const express = require("express");
const chaosMonkey = require("chaos-monkey");
const config = require("./chaosMonkey.config");
const app = express();
app.use(chaosMonkey(config)); // Inject failures based on configuration
app.get("/", (req, res) => {
  res.send("Chaos Engineering in Node.js is running!");
});

app.listen(3000, () => {
  console.log("App running on port 3000 with Chaos Monkey");
});

Chaos Toolkit for Node.js Applications

Similar to Kubernetes and Java applications, Chaos Toolkit can be used to inject failures into Node.js services.

Example: Latency Injection for Node.js using Chaos Toolkit

This Chaos Toolkit experiment will introduce latency into a Node.js service.

{
  "title": "Introduce artificial latency in Node.js service",
  "description": "Test how the Node.js API handles slow responses.",
  "method": [
    {
      "type": "action",
      "name": "introduce-latency",
      "provider": {
        "type": "process",
        "path": "curl",
        "arguments": [
          "-X",
          "POST",
          "http://localhost:3000/chaosmonkey/enable-latency"
        ]
      }
    }
  ],
  "rollbacks": [
    {
      "type": "action",
      "name": "remove-latency",
      "provider": {
        "type": "process",
        "path": "curl",
        "arguments": [
          "-X",
          "POST",
          "http://localhost:3000/chaosmonkey/disable-latency"
        ]
      }
    }
  ]
}

To execute and report the experiment:

chaos run node-latency-experiment.json --journal-path=node-latency-journal.json 

chaos report --export-format=json node-latency-journal.json > node-latency-report.json

Chaos Experiments in Multi-Cloud and Kubernetes Environments

For microservices deployed on Kubernetes or multi-cloud platforms, Chaos Toolkit provides a more robust way to perform failover testing.

For microservices deployed on Kubernetes or multi-cloud platforms, Chaos Toolkit provides a more robust way to perform failover testing.

A pod-kill experiment to test application resilience in Kubernetes:

{
  "version": "1.0.0",
  "title": "System Resilience to Pod Failures",
  "description": "Can the system survive a pod failure?",
  "configuration": {
    "app_name": { "type": "env", "key": "APP_NAME" },
    "namespace": { "type": "env", "key": "NAMESPACE" }
  },
  "steady-state-hypothesis": {
    "title": "Application must be up and healthy",
    "probes": [{
      "name": "check-application-health",
      "type": "probe",
      "provider": {
        "type": "http",
        "url": "http://myapp.com/health",
        "method": "GET"
      }
    }]
  },
  "method": [{
    "type": "action",
    "name": "terminate-pod",
    "provider": {
      "type": "python",
      "module": "chaosk8s.pod.actions",
      "func": "terminate_pods",
      "arguments": {
        "label_selector": "app=${app_name}",
        "ns": "${namespace}",
        "rand": true,
        "mode": "fixed",
        "qty": 1
      }
    }
  }]
}

Running the Chaos Experiment

To execute the experiment, run:

chaos run pod-kill-experiment.json --journal-path=pod-kill-experiment-journal.json 

To generate a report after execution:

chaos report --export-format=html pod-kill-experiment-journal.json > pod-kill-experiment-report.html 

Rolling back the experiment (if necessary):

chaos rollback pod-kill-experiment.json 

Example: Region Delay Experiment (Kubernetes and Istio)

This experiment injects network latency into requests by modifying Istio’s virtual service.

version: "1.0.0"
title: "Region Delay Experiment"
description: "Simulating high latency in a specific region"
method:
  - type: action
    name: "inject-fault"
    provider:
      type: python
      module: chaosistio.fault.actions
      func: add_delay_fault
      arguments:
        virtual_service_name: "my-service-vs"
        fixed_delay: "5s"
        percentage: 100
        ns: "default"
  pauses:
    before: 5
    after: 20
rollbacks:
  - type: action
    name: "remove-fault"
    provider:
      type: python
      module: chaosistio.fault.actions
      func: remove_delay_fault
      arguments:
        virtual_service_name: "my-service-vs"
        ns: "default"

To execute:

chaos run region-delay-experiment.yaml --journal-path=region-delay-journal.json 

Generate a detailed report:

chaos report --export-format=html region-delay-journal.json > region-delay-report.html