Injecting Chaos: Easy Techniques for Simulating Network Issues in Redis Clusters

While comprehensive chaos testing tools offer a wide range of features, sometimes you just need a quick and easy solution for a specific scenario. This article focuses on a targeted approach: simulating network issues between Redis client and Redis Cluster in simple steps. These methods are ideal when you don't require a complex setup and want to focus on testing a particular aspect of your Redis cluster's behavior under simulated network issues.

Set-Up

This article assumes that you already have a Redis cluster and the client code for sending traffic to the cluster is set up and ready to use. If not, you can refer to the following steps:

• Install a Redis cluster: You can follow this article to set up a Redis cluster locally before taking it to production.
• There are several Redis clients available for different languages, you can choose what’s most suitable for your use case. 
  • Jedis documentation
  • Lettuce documentation
  • Redisson documentation
  • redigo documentation
  • go-redis/redis documentation
  • redis-py documentation
  • hiredis documentation

Let’s explore a few ways to simulate network issues between Redis clients and the Redis Cluster.

Simulate Slow Redis Server Response

DEBUG SLEEP

DEBUG SLEEP <seconds>

The DEBUG SLEEP command will suspend all operations, including command processing, network handling, and replication, on the specified Redis node for a given time duration effectively making the Redis node unresponsive for the specified duration. 

Once this command is initiated the response is not sent until the specified duration is elapsed. 

In the above screenshot, the response (OK) is received after 5 seconds. 

Use Case

This command can be used to simulate a slow server response, server hang-ups, and heavy load conditions, and observe the system’s reaction to an unresponsive Redis instance.

Simulate Connection Pause for Clients

CLIENT PAUSE 

CLIENT PAUSE <milliseconds>

This command temporarily pauses all the clients and the commands will be delayed for a specified duration however interactions with replica will continue normally. 

Modes: CLIENT PAUSE supports two modes:

ALL (default): Pauses all client commands (write and read).

WRITE: Only blocks write commands (reads still work).

It gives finer control if you want to simulate connection pause only for writes or all client commands.

Once this command is initiated it responds back with “OK” immediately (unlike debug sleep)

Use Case

Useful for scenarios like controlled failover testing, control client behavior, or maintenance tasks where you want to ensure that no new commands are processed temporarily.

Simulate Network Issues Using Custom Interceptors/Listeners

Interceptors or listeners can be valuable tools for injecting high latency or other network issues into the communication between a Redis client and server, facilitating effective testing of how the Redis deployment behaves under adverse network conditions. 

Inject High Latency Using a Listener

Interceptors or Listeners act as a middleman, listening for commands sent to the Redis server. When a command is detected, we can introduce a configurable delay before forwarding it by overriding the methods of the listener. This way you can simulate high latency and it allows you to observe how your client behaves under slow network conditions. 

The following example shows how to create a basic latency injector by implementing the CommandListener class in the Lettuce Java Redis client. 

package com.rc;

import io.lettuce.core.event.command.CommandFailedEvent;
import io.lettuce.core.event.command.CommandListener;
import io.lettuce.core.event.command.CommandStartedEvent;
import io.lettuce.core.event.command.CommandSucceededEvent;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.util.concurrent.TimeUnit;

public class LatencyInjectorListener implements CommandListener {

   private static final Logger logger = LoggerFactory.getLogger(LatencyInjectorListener.class);
   private final long delayInMillis;
   private final boolean enabled;

   public LatencyInjectorListener(long delayInMillis, boolean enabled) {
       this.delayInMillis = delayInMillis;
       this.enabled = enabled;
   }

   @Override
   public void commandStarted(CommandStartedEvent event) {
       if (enabled) {
           try {
               // Introduce latency
               Thread.sleep(delayInMillis);
           } catch (InterruptedException e) {
               // Handle interruption gracefully,
               logger.error("Exception while invoking sleep method");
           }
       }
   }

   @Override
   public void commandSucceeded(CommandSucceededEvent event) {
   }

   @Override
   public void commandFailed(CommandFailedEvent event) {
   }

}

In the above example, we have added a class that implements CommandListener interface provided by the Lettuce Java Redis client. And, in commandStarted method, we have invoked Thead.sleep() that will cause the flow to halt for a specific duration, thereby adding latency to each command that will be executed. You can add latency in other methods also such as commandSucceeded and commandFailed, depending upon the specific behavior you want to test. 

Simulate Intermittent Connection Errors

You can even extend this concept to throw exceptions within the listener, mimicking connection errors or timeouts. This proactive approach using listeners helps you identify and address potential network-related issues in your Redis client before they impact real-world deployments

The following example shows the extension of the commandStarted method implemented in the above section to throw connection exceptions to create intermittent connection failures/errors implementing CommandListener class in Lettuce Java Redis client. 

@Override
public void commandStarted(CommandStartedEvent event) {

   if (enabled && shouldThrowConnectionError()) {
       // introduce connection errors
       throw new RedisConnectionException("Simulated connection error");
   } else if (enabled) {
       try {
           // Introduce latency
           Thread.sleep(delayInMillis);
       } catch (InterruptedException e) {
           // Handle interruption gracefully,
           logger.error("Exception while invoking sleep method");
       }
   }
}

private boolean shouldThrowConnectionError() {
   // adjust or change the logic as needed - this is just for reference.
   return random.nextInt(10) < 3; // 30% chance to throw an error
}

Similarly, Redis clients in other languages also provide hooks/interceptors to extend and simulate network issues such as high latency or connection errors. 

Conclusion

We explored several techniques to simulate network issues for chaos testing specific to network-related scenarios in a Redis Cluster. However, exercise caution and ensure these methods are enabled with a flag and used only in strictly controlled testing environments. Proper safeguards are essential to avoid unintended disruptions. By carefully implementing these strategies, you can gain valuable insights into the resilience and robustness of your Redis infrastructure under adverse network conditions.

References

• Create and Configure a Local Redis Cluster
• Redis Docs
• Lettuce

Other Related Articles

If you enjoyed this article, you might also find these related articles interesting.

• Techniques for Chaos Testing Your Redis Cluster 
• Manage Redis Cluster Topology With Command Line