DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

Zones

Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks

Last call! Secure your stack and shape the future! Help dev teams across the globe navigate their software supply chain security challenges.

Modernize your data layer. Learn how to design cloud-native database architectures to meet the evolving demands of AI and GenAI workloads.

Releasing software shouldn't be stressful or risky. Learn how to leverage progressive delivery techniques to ensure safer deployments.

Avoid machine learning mistakes and boost model performance! Discover key ML patterns, anti-patterns, data strategies, and more.

Related

  • How to Repair Corrupt MySQL Database Tables Step-by-Step
  • Harnessing the Power of AWS Aurora for Scalable and Reliable Databases
  • Optimizing MySQL Performance: Best Practices for Database Efficiency
  • Fine-Tuning Performance, Resolving Common Issues in FinTech Application With MySQL

Trending

  • Start Coding With Google Cloud Workstations
  • Is Agile Right for Every Project? When To Use It and When To Avoid It
  • Automating Data Pipelines: Generating PySpark and SQL Jobs With LLMs in Cloudera
  • 5 Subtle Indicators Your Development Environment Is Under Siege
  1. DZone
  2. Data Engineering
  3. Databases
  4. Managing MySQL Server Logs: Rotate, Compress, Retain, and Delete

Managing MySQL Server Logs: Rotate, Compress, Retain, and Delete

In this blog post, we describe some best practices for setting up and managing MySQL error logs, general logs, and slow query logs for your MySQL deployments.

By 
Prasad Nagaraj user avatar
Prasad Nagaraj
·
May. 14, 19 · Tutorial
Likes (1)
Comment
Save
Tweet
Share
17.1K Views

Join the DZone community and get the full member experience.

Join For Free

MySQL Server generates several logs that can help you monitor the activities of the server. However, once these logs are enabled, they can grow in size and start taking up too much disk space. This is why it’s important to have an automated way of archiving and preserving MySQL log files for a certain duration, as well as deleting the old ones. In this blog post, we describe some best practices for setting up and managing MySQL error logs, general logs and slow query logs for your MySQL deployments.

Setting Up MySQL Server Logging

Let’s look at how to set up the following 3 types of logs:

Error Log

Logs all the problems encountered during starting, running, or stopping mysqld. This log can be enabled by having the following option in /etc/my.cnf file:

  • log_error=/var/log/mysql/mysqld.log

General Query Log

Logs established client connections and statements received from clients. This log can be enabled by having the following option in /etc/my.cnf file:

  • general_log=ON
  • general_log_file=/var/log/mysql/general.log

Slow Query Log

Logs queries that took more than long_query_time seconds to execute. This log can be enabled by the following option in /etc/my.cnf file:

  • slow_query_log=ON
  • slow_query_log_file=/var/log/mysql/mysql-slowquery.log

Setting Up Criteria For Log Rotation

As an example, let’s have some criteria for managing general MySQL query logs. We can come up with a suitable set of criteria for log management by asking the following questions:

Q: What is the maximum size that the log file can grow?

A: Let’s say it can grow up to 300 MB after which it needs to be rotated and compressed.

Q: What is the frequency that you want the log file to be rotated?

A: We can say that we want logs to be rotated on a daily basis.

Q: How many old log files you want to retain?

A: We would like to retain the last 30 log files.

Based on the above criteria, the overall disk space required for general query log management is about 1.2 GB. Assuming a 90 percent compression ratio — we will have 30 compressed log files of size 30 MB each and a live log file of about 300 MB.

Managing The Logs Using Linux logrotate Utility

logrotate is a Linux utility that helps with the efficient administration of log files and provides options for automatic rotation, compression, and removal of log files. The criteria established above can be configured for logrotate utility by creating a configuration file in the /etc/logrotate.d folder.

Let’s call this configuration file mysqlgeneral and the contents of the file will be:

/var/log/mysql/general.log{
        compress
        dateext
        maxsize 300M
        copytruncate
        maxage 365
        dateformat -%Y%m%d%s
        daily
        rotate 30
        notifempty
}

With the above options for logrotate, the general query logs get rotated either on a daily basis or when the log file size exceeds 300 MB. The old logs are compressed and 30 such files will be preserved. Log rotation will be skipped if the log file is empty due to the setting ‘notifempty’.

The ‘copytruncate’ option is to ensure that current log file is never deleted during rotation and only its contents get truncated. This is important since some applications expect that the log file is always available and it’s not possible to delete the log without stopping the application first.

Now that the log rotation configuration is set for the general query log, the logrotate utility has to be run so that the above configuration is executed. This is typically done through a cron job. We can set this to be running every hour by placing the logrotate script in /etc/cron.hourly directory:

#!/bin/sh

/usr/sbin/logrotate /etc/logrotate.conf
EXITVALUE=$?
if [ $EXITVALUE != 0 ]; then
    /usr/bin/logger -t logrotate "ALERT exited abnormally with [$EXITVALUE]"
fi
exit 0

So, with a few simple steps, we have set up log rotation for MySQL general logs based on our criteria. The same approach can be followed for MySQL error logs and slow query logs as well. Check out these other posts to learn more about optimizing your MySQL deployments:

  • Calculating InnoDB Buffer Pool Size for your MySQL Server
  • MySQL Tutorial – Configuring and Managing SSL On Your MySQL Server
  • MySQL High Availability Framework Explained – Part I: Introduction
MySQL Log rotation Database

Published at DZone with permission of Prasad Nagaraj, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

Related

  • How to Repair Corrupt MySQL Database Tables Step-by-Step
  • Harnessing the Power of AWS Aurora for Scalable and Reliable Databases
  • Optimizing MySQL Performance: Best Practices for Database Efficiency
  • Fine-Tuning Performance, Resolving Common Issues in FinTech Application With MySQL

Partner Resources

×

Comments

The likes didn't load as expected. Please refresh the page and try again.

ABOUT US

  • About DZone
  • Support and feedback
  • Community research
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 100
  • Nashville, TN 37211
  • support@dzone.com

Let's be friends: