DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports Events Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Partner Zones AWS Cloud
by AWS Developer Relations
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Partner Zones
AWS Cloud
by AWS Developer Relations
Building Scalable Real-Time Apps with AstraDB and Vaadin
Register Now

Trending

  • Is Podman a Drop-in Replacement for Docker?
  • The SPACE Framework for Developer Productivity
  • Unlocking the Power of AIOps: Enhancing DevOps With Intelligent Automation for Optimized IT Operations
  • A Complete Guide to AWS File Handling and How It Is Revolutionizing Cloud Storage

Trending

  • Is Podman a Drop-in Replacement for Docker?
  • The SPACE Framework for Developer Productivity
  • Unlocking the Power of AIOps: Enhancing DevOps With Intelligent Automation for Optimized IT Operations
  • A Complete Guide to AWS File Handling and How It Is Revolutionizing Cloud Storage
  1. DZone
  2. Software Design and Architecture
  3. Microservices
  4. Top Redis Headaches for DevOps: Replication Buffer

Top Redis Headaches for DevOps: Replication Buffer

Itamar Haber user avatar by
Itamar Haber
·
Aug. 05, 14 · Interview
Like (0)
Save
Tweet
Share
10.16K Views

Join the DZone community and get the full member experience.

Join For Free

originally written by yaron dolev

redis provides a wide variety of tools directed at improving and maintaining efficient in-memory database usage. while its unique data types and commands fine-tune databases to serve application requests without any additional processing at the application level, misconfiguration, or rather, using out-of-the-box configuration, can (and does) lead to operational challenges and performance issues. despite the setbacks that have been the cause of quite a few headaches, solutions do exist, and may be even simpler than anticipated.

this series of installments will highlight some of the most irritating issues that come up when using redis, along with tips on how to solve them. they are based on our real-life experience of running thousands of redis database instances.

the replication buffer limit

replication buffers are memory buffers that hold data while a slave redis server synchronizes with the master server. in a full master-slave synchronization, changes performed to the data during the initial phase of the synchronization are held in the replication buffer by the master server. after the completion of the initial phase, the contents of the buffer are sent to the slave. there is a limitation to the size of the buffer that can be used in this procedure, causing replication to start from the beginning when the maximum is reached, as mentioned in our post on endless redis replication loops . in order to prevent this from happening, an initial configuration of the buffer needs to take place according to the amount and types of changes expected to be made during the replication process. for example, a low volume of changes and/or smaller data in the changes can get by with a smaller buffer, whereas if there are a lot of changes and/or the changes are big, a large buffer is needed. a more comprehensive solution entails setting the buffers to a very high level to offset the possibility of a lengthy or heavy replication process that will eventually exhaust the buffer (if the latter is too small). ultimately, this solution requires fine-tuning the specific database at hand.

redis default setting:

> config get client-output-buffer-limit
1) "client-output-buffer-limit"
2) "normal 1073741824 536870912 30 slave 268435456 67108864 60 pubsub 33554432 8388608 60"

as explained here , this default configuration replication link will be broken (causing the synchronization to start from the beginning) once the 256mb hard limit is reached, or if a soft limit of 67mb is reached and held for a continuous 60 seconds. in many cases, especially with a high ‘write’ load and insufficient bandwidth to the slave server, the replication process will never finish. this can lead to an infinite loop situation where the master redis is constantly forking and snapshotting the entire dataset to disk, which can cause up to triple the amount of extra memory to be used together with a high rate of i/o operations. additionally, this infinite loop situation results in the slave never being able to catch up and fully synchronize with the master redis server.

a simple solution that offers an immediate improvement results from increasing the size of the output slave buffer by setting both the hard and soft limits to 512mb:

> config set client-output-buffer-limit "slave 536870912 536870912 0"

as with many reconfigurations, it is important to understand that:

  1. before increasing the size of the replication buffers you must make sure you have enough memory on your machine.
  2. the redis memory usage calculation does not take the replication buffer size into account.

that brings us to the end of our first installment of the top redis operational headaches. as pointed out above, in terms of replication buffer limits, proper configuration can go a long way. be sure to keep an eye out for the next post in this compilation covering replication timeouts and how to handle them accordingly.

continue reading our next post in this series: replication timeouts


Buffer (application) Replication (computing) Redis (company) DevOps

Published at DZone with permission of Itamar Haber, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

Trending

  • Is Podman a Drop-in Replacement for Docker?
  • The SPACE Framework for Developer Productivity
  • Unlocking the Power of AIOps: Enhancing DevOps With Intelligent Automation for Optimized IT Operations
  • A Complete Guide to AWS File Handling and How It Is Revolutionizing Cloud Storage

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com

Let's be friends: