How to Conduct a Failsafe Redis Setup

Redis is an in-memory NoSQL DB that is fast and persistent. This article will cover performance tests, high availability, and load balanced configuration. Spring has builtin support for Redis as a cache, so you can use Redis as a cache with ease.

I got a chance to learn Redis, its configuration, and usages as Redis cache. Configured the advance production-ready configuration of Redis and ran the benchmark test on it and results were amazing.

I have used the redis-benchmark command line tool to perform benchmarking. 

Here's the first test, where I used 568 bytes of data for GET/SET:

redis-benchmark -q -n 100000 -c 50 -P 12 -r 16 -d 568
PING_INLINE: 337837.84 requests per second
PING_BULK: 331125.84 requests per second
SET: 284090.91 requests per second
GET: 318471.31 requests per second
INCR: 444444.47 requests per second
LPUSH: 349650.34 requests per second
RPUSH: 352112.66 requests per second
LPOP: 392156.88 requests per second
RPOP: 390624.97 requests per second
SADD: 425531.91 requests per second
SPOP: 401606.44 requests per second
LPUSH (needed to benchmark LRANGE): 346020.75 requests per second
LRANGE_100 (first 100 elements): 354609.94 requests per second
LRANGE_300 (first 300 elements): 337837.84 requests per second
LRANGE_500 (first 450 elements): 343642.59 requests per second
LRANGE_600 (first 600 elements): 317460.31 requests per second
MSET (10 keys): 62227.75 requests per second

The second test where I used 1,000 bytes of data for GET/SET. Still, there is no huge decline in the throughput:

redis-benchmark -q -n 100000 -c 50 -P 12 -r 16 -d 1000
PING_INLINE: 369003.69 requests per second
PING_BULK: 416666.69 requests per second
SET: 277777.78 requests per second
GET: 367647.03 requests per second
INCR: 423728.81 requests per second
LPUSH: 277777.78 requests per second
RPUSH: 277777.78 requests per second
LPOP: 462962.94 requests per second
RPOP: 432900.41 requests per second
SADD: 373134.31 requests per second
SPOP: 403225.81 requests per second
LPUSH (needed to benchmark LRANGE): 251889.16 requests per second
LRANGE_100 (first 100 elements): 318471.31 requests per second
LRANGE_300 (first 300 elements): 317460.31 requests per second
LRANGE_500 (first 450 elements): 335570.47 requests per second
LRANGE_600 (first 600 elements): 325732.88 requests per second
MSET (10 keys): 41666.66 requests per second

So I think it's more than enough fast caching provider if you compare it with Memcached. Redis gives you persistence capability, and if you compare it with EHCache, Redis has better supportability for distributed applications. EHCache will have the caching on each instance and if you want to synch it within all servers, there is additional care you need to take — whereas Redis will have separate servers so distributed cache will be out of the box.

There are cloud and AWS solutions available for Redis, which is easier to use and get started with if you have exposure to the cloud.

If you are looking for a simple Redis setup to cache the application data or service data, and if you want to set in-house servers, then please continue reading. Setup complements high availability and fault tolerance.

Let's go through the terms first.

• Redis master: As the name suggests, it's a master server that serves the client — in my case, web application. This is the server which will take write and read requests.

• Redis slave: Hmm... what could this possibly be? Yup, it's the slave to Redis master! It will maintain a copy of data and if the master goes down, then this one can be the next master (if you set up in that way). The slave takes data copy from master and beauty of the implementation is it's asynchronous so there is separate thread/process which connects to the client for data sync.

• Redis sentinel: This is the one I like most. Sentinel is a separate (even you can have a separate machine or VM) process that runs on a different port and monitors the Redis server. It keeps monitoring the Redis masters and slaves. There needs to be a separate sentinel for each slave and master. If the master goes down for a configured time, then all the sentinels do a poll and select the next Master.

Let’s dig into the Redis setup. I have configured one master, four slaves, and five sentinels to monitor the Redis instances. Sentinel is the Redis monitoring program that monitors the Redis instances, and if the master goes down, it selects the next master from one of the slaves.

Redis has a configuration file called Redis.conf, which is well-documented. Here, I will cover the main points to consider while configuring Redis.

Below is the snippet of the Redis.conf file of the slave.

bind 10.0.0.13
protected-mode yes
port 78965
daemonize yes
slaveof 156.45.44.25 33095
masterauth "MyVeryLongMasterPasswordToProtectServerFromBrutForceAttacks"
slave-priority 100
slave-announce-ip "156.45.44.35"
requirepass "MyVeryLongMasterPasswordToProtectServerFromBrutForceAttacks"

1. bind binds the private IP of the machine where the slave is running.
2. Enable the protected mode to yes, as it will be your prod instance.
3. This is the port on which the slave will listen.
4. The daemonize flag tells Slave Redis to run in the background.
5. slaveof indicates that who is the master of this slave.
6. masterauth is the passphrase you need to connect to the master.
7. slave-priority is a required, important configuration for the fail-safe mode. Based on the priority, the sentinel decides which slave will be elected as the master in the case of an existing master going down.
8. slave-announce-ip is the public IP that the Redis slave will announce to be used by clients and sentinels.
9. requirepass is the passphrase used to connect to this slave machine. I have used the same passphrase for all the Redis servers (master and slaves).

You need to repeat this configuration for each slave.

Points to consider while configuring Redis:

• Custom IP address bind configuration uses a private IP address instead of a public one.
• Change the default port to the custom one.
• You need to select the timeout setting carefully. This setting tells the Redis server when to disconnect the client if it’s ideal for n seconds. If you are considering this setup to use with Spring Cache, then align this setting with your Jedis connection pool setting.
• The log level for the PROD instance needs to be kept to lower, as you may face server memory issues if your logs occupy more disk space than what is necessary.
• Limit the maximum database your server wants to handle for maintainability.
• The snapshotting section is the critical one. Here, if you have the number of slaves, then you need to select the snapshot frequency accordingly. Redis does asynchronous sync with the slaves, but you need to consider a good balance between CPU and memory usage and the time for the consistent state. You can have multiple conditions to trigger the snapshot — for example:
  • Save 900 1 (after 900 seconds if there is one record changed).
  • Save 300 10 (after 300 seconds if there are 10 records changed).
  • Save 60 10000 (after 60 seconds if there are 1000 records changed).
• Configure the slave master using slaveof and masterauth settings.
• slave-priority is another key configuration you need to consider if you are planning to use the sentinel. If the master is down, then based on this configuration, the sentinel process identifies the next master. The lower-priority one will be considered as next master.
  • Note: Do not set it to 0. The value indicates that this Redis instance will not be promoted to master, it will be always considered as slave.
• If you have separate IPs (private and public), then — based on the challenges I faced — always use the private IP for binding to Redis server and configure the announce-IP configuration to announce the IP of a Redis server with a public IP address.
• Always set the maxclient configuration.
• We found this very useful when we performed load tests on the Redis server to see how Redis performs with huge data sets. maxmemory tells what is the max memory allocation for the Redis. maxmemory-policy tells Redis what to do when the maxmemory threshold is reached. Redis has provided different strategies you can choose from.

Once this is done you can start all the Redis servers by using src/redis-server redis.conf.

You can ensure whether all your slaves are connected to slaves by following below steps.

Connect to your master with below command, its simple arguments, IP, and port, and then the master passphrase: sudo src/redis-cli -h 156.45.44.25 -p 50668 –a MyLongPassword.

Once you are connected to the Redis master, run the info command, which will give you the details of running instance. Look for the synchronization section. You will see connected_slaves:4.

This way, you can ensure that your configuration is running as expected.

For high availability and fail safe, you really need to understand the sentinel configuration. Sentinel configuration is maintained in the sentinel.conf file. Below are the details of a few lines from the sentinel configuration:

bind 10.0.0.13
logfile "./redis/Sentinel.log"
port 26379
sentinel monitor MyProdMaster 156.45.44.21 78965 2
sentinel auth-pass MyProdMaster MyLongPassword
sentinel config-epoch MyProdMaster 20
sentinel leader-epoch MyProdMaster 20
sentinel announce-ip 156.45.44.25

1. bind binds the sentinel process to the private IP address.
2. logfile provides the logfile location — please do provide the full path. This is required as you can look into the sentinel activity.
3. Line 3 is the port number where the sentinel will listen.
4. At line 4, provide the name of the server to refer to. I provided it as MyProdMaster. Then, specify the master IP (this is the public IP, which you configured through the announced IP configuration in the redis.conf file). The last part of the line is the quorum. In my case, two sentinels need to agree if the server is down.
5. Line 5 specifies the passphrase to connect to a master or slave.
6. leader-epoch is the time in seconds. The sentinel will mark the server as down if the sentinel observes the server down for 20 seconds.
7. According to line 7, if a number of sentinels that you set as a quorum agree that the master is down, the master selecting will start after 20 seconds.

You can start the sentinel server by executing sudo src/redis-sentinel sentinel.conf .

To ensure your sentinel is configured and running as expected, you can connect to the sentinel by executing the following command (use your "announced" IP and the custom port to connect; refer your sentinel.conf file to get these details): sudo src/redis-sentinel -h 156.45.44.28 -p 23988 –a MyLongPassword.

Once you are connected to the sentinel, you can run the INFO command to get the sentinel information. Below is a snippet of the result of the INFO command:

master0:name=MyProd1Master,status=ok,address=156.45.44.21:68,slaves=4,sentinels=5

Now your setup is ready. The next steps to consider are:

1. Test your setup on the lower environment and the non-prod environment. Put down your master and analyze how the sentinel polls and elects the master; analyze the sentinel logs for details.
2. Try to put the load on your setup and then test the configuration. You can use Gatling to test the end-to-end flow; in this case, the REST service call and then Redis Cache.
3. Implement the monitoring process to get insights from your setup.