How to Conduct a Failsafe Redis Setup
How to Conduct a Failsafe Redis Setup
Learn about how to set up Redis to be failsafe by considering doing performance tests, having high availability, and having a load balanced configuration.
Join the DZone community and get the full member experience.Join For Free
Databases are better when they can run themselves. CockroachDB is a SQL database that automates scaling and recovery. Check it out here.
Redis is an in-memory NoSQL DB that is fast and persistent. This article will cover performance tests, high availability, and load balanced configuration. Spring has builtin support for Redis as a cache, so you can use Redis as a cache with ease.
I got a chance to learn Redis, its configuration, and usages as Redis cache. Configured the advance production-ready configuration of Redis and ran the benchmark test on it and results were amazing.
I have used the redis-benchmark command line tool to perform benchmarking.
Here's the first test, where I used 568 bytes of data for
redis-benchmark -q -n 100000 -c 50 -P 12 -r 16 -d 568 PING_INLINE: 337837.84 requests per second PING_BULK: 331125.84 requests per second SET: 284090.91 requests per second GET: 318471.31 requests per second INCR: 444444.47 requests per second LPUSH: 349650.34 requests per second RPUSH: 352112.66 requests per second LPOP: 392156.88 requests per second RPOP: 390624.97 requests per second SADD: 425531.91 requests per second SPOP: 401606.44 requests per second LPUSH (needed to benchmark LRANGE): 346020.75 requests per second LRANGE_100 (first 100 elements): 354609.94 requests per second LRANGE_300 (first 300 elements): 337837.84 requests per second LRANGE_500 (first 450 elements): 343642.59 requests per second LRANGE_600 (first 600 elements): 317460.31 requests per second MSET (10 keys): 62227.75 requests per second
The second test where I used 1,000 bytes of data for
SET. Still, there is no huge decline in the throughput:
redis-benchmark -q -n 100000 -c 50 -P 12 -r 16 -d 1000 PING_INLINE: 369003.69 requests per second PING_BULK: 416666.69 requests per second SET: 277777.78 requests per second GET: 367647.03 requests per second INCR: 423728.81 requests per second LPUSH: 277777.78 requests per second RPUSH: 277777.78 requests per second LPOP: 462962.94 requests per second RPOP: 432900.41 requests per second SADD: 373134.31 requests per second SPOP: 403225.81 requests per second LPUSH (needed to benchmark LRANGE): 251889.16 requests per second LRANGE_100 (first 100 elements): 318471.31 requests per second LRANGE_300 (first 300 elements): 317460.31 requests per second LRANGE_500 (first 450 elements): 335570.47 requests per second LRANGE_600 (first 600 elements): 325732.88 requests per second MSET (10 keys): 41666.66 requests per second
So I think it's more than enough fast caching provider if you compare it with Memcached. Redis gives you persistence capability, and if you compare it with EHCache, Redis has better supportability for distributed applications. EHCache will have the caching on each instance and if you want to synch it within all servers, there is additional care you need to take — whereas Redis will have separate servers so distributed cache will be out of the box.
There are cloud and AWS solutions available for Redis, which is easier to use and get started with if you have exposure to the cloud.
If you are looking for a simple Redis setup to cache the application data or service data, and if you want to set in-house servers, then please continue reading. Setup complements high availability and fault tolerance.
Let's go through the terms first.
Redis master: As the name suggests, it's a master server that serves the client — in my case, web application. This is the server which will take write and read requests.
Redis slave: Hmm... what could this possibly be? Yup, it's the slave to Redis master! It will maintain a copy of data and if the master goes down, then this one can be the next master (if you set up in that way). The slave takes data copy from master and beauty of the implementation is it's asynchronous so there is separate thread/process which connects to the client for data sync.
Redis sentinel: This is the one I like most. Sentinel is a separate (even you can have a separate machine or VM) process that runs on a different port and monitors the Redis server. It keeps monitoring the Redis masters and slaves. There needs to be a separate sentinel for each slave and master. If the master goes down for a configured time, then all the sentinels do a poll and select the next Master.
Let’s dig into the Redis setup. I have configured one master, four slaves, and five sentinels to monitor the Redis instances. Sentinel is the Redis monitoring program that monitors the Redis instances, and if the master goes down, it selects the next master from one of the slaves.
Redis has a configuration file called
Redis.conf, which is well-documented. Here, I will cover the main points to consider while configuring Redis.
Below is the snippet of the
Redis.conf file of the slave.
bind 10.0.0.13 protected-mode yes port 78965 daemonize yes slaveof 220.127.116.11 33095 masterauth "MyVeryLongMasterPasswordToProtectServerFromBrutForceAttacks" slave-priority 100 slave-announce-ip "18.104.22.168" requirepass "MyVeryLongMasterPasswordToProtectServerFromBrutForceAttacks"
bindbinds the private IP of the machine where the slave is running.
- Enable the protected mode to
yes, as it will be your prod instance.
- This is the
porton which the slave will listen.
daemonizeflag tells Slave Redis to run in the background.
slaveofindicates that who is the master of this slave.
masterauthis the passphrase you need to connect to the master.
slave-priorityis a required, important configuration for the fail-safe mode. Based on the priority, the sentinel decides which slave will be elected as the master in the case of an existing master going down.
slave-announce-ipis the public IP that the Redis slave will announce to be used by clients and sentinels.
requirepassis the passphrase used to connect to this slave machine. I have used the same passphrase for all the Redis servers (master and slaves).
You need to repeat this configuration for each slave.
Points to consider while configuring Redis:
- Custom IP address bind configuration uses a private IP address instead of a public one.
- Change the default port to the custom one.
- You need to select the timeout setting carefully. This setting tells the Redis server when to disconnect the client if it’s ideal for n seconds. If you are considering this setup to use with Spring Cache, then align this setting with your Jedis connection pool setting.
- The log level for the PROD instance needs to be kept to lower, as you may face server memory issues if your logs occupy more disk space than what is necessary.
- Limit the maximum database your server wants to handle for maintainability.
- The snapshotting section is the critical one. Here, if you have the number of slaves, then you need to select the snapshot frequency accordingly. Redis does asynchronous sync with the slaves, but you need to consider a good balance between CPU and memory usage and the time for the consistent state. You can have multiple conditions to trigger the snapshot — for example:
- Save 900 1 (after 900 seconds if there is one record changed).
- Save 300 10 (after 300 seconds if there are 10 records changed).
- Save 60 10000 (after 60 seconds if there are 1000 records changed).
- Configure the slave master using
slave-priorityis another key configuration you need to consider if you are planning to use the sentinel. If the master is down, then based on this configuration, the sentinel process identifies the next master. The lower-priority one will be considered as next master.
- Note: Do not set it to
0. The value indicates that this Redis instance will not be promoted to master, it will be always considered as slave.
- Note: Do not set it to
- If you have separate IPs (private and public), then — based on the challenges I faced — always use the private IP for binding to Redis server and configure the
announce-IPconfiguration to announce the IP of a Redis server with a public IP address.
- Always set the
- We found this very useful when we performed load tests on the Redis server to see how Redis performs with huge data sets.
maxmemorytells what is the max memory allocation for the Redis.
maxmemory-policytells Redis what to do when the
maxmemorythreshold is reached. Redis has provided different strategies you can choose from.
Once this is done you can start all the Redis servers by using
You can ensure whether all your slaves are connected to slaves by following below steps.
Connect to your master with below command, its simple arguments, IP, and port, and then the master passphrase:
sudo src/redis-cli -h 22.214.171.124 -p 50668 –a MyLongPassword.
Once you are connected to the Redis master, run the
info command, which will give you the details of running instance. Look for the
synchronization section. You will see
This way, you can ensure that your configuration is running as expected.
For high availability and fail safe, you really need to understand the sentinel configuration. Sentinel configuration is maintained in the
sentinel.conf file. Below are the details of a few lines from the sentinel configuration:
bind 10.0.0.13 logfile "./redis/Sentinel.log" port 26379 sentinel monitor MyProdMaster 126.96.36.199 78965 2 sentinel auth-pass MyProdMaster MyLongPassword sentinel config-epoch MyProdMaster 20 sentinel leader-epoch MyProdMaster 20 sentinel announce-ip 188.8.131.52
bindbinds the sentinel process to the private IP address.
logfileprovides the logfile location — please do provide the full path. This is required as you can look into the sentinel activity.
- Line 3 is the port number where the sentinel will listen.
- At line 4, provide the name of the server to refer to. I provided it as
MyProdMaster. Then, specify the master IP (this is the public IP, which you configured through the announced IP configuration in the
redis.conffile). The last part of the line is the quorum. In my case, two sentinels need to agree if the server is down.
- Line 5 specifies the passphrase to connect to a master or slave.
leader-epochis the time in seconds. The sentinel will mark the server as down if the sentinel observes the server down for 20 seconds.
- According to line 7, if a number of sentinels that you set as a quorum agree that the master is down, the master selecting will start after 20 seconds.
You can start the sentinel server by executing
sudo src/redis-sentinel sentinel.conf .
To ensure your sentinel is configured and running as expected, you can connect to the sentinel by executing the following command (use your "announced" IP and the custom port to connect; refer your
sentinel.conf file to get these details):
sudo src/redis-sentinel -h 184.108.40.206 -p 23988 –a MyLongPassword.
Once you are connected to the sentinel, you can run the
INFO command to get the sentinel information. Below is a snippet of the result of the
Now your setup is ready. The next steps to consider are:
- Test your setup on the lower environment and the non-prod environment. Put down your master and analyze how the sentinel polls and elects the master; analyze the sentinel logs for details.
- Try to put the load on your setup and then test the configuration. You can use Gatling to test the end-to-end flow; in this case, the REST service call and then Redis Cache.
- Implement the monitoring process to get insights from your setup.
Opinions expressed by DZone contributors are their own.