Important Health Checks for Your MySQL Master-Slave Servers
In this article, we explain some basic health checks you can do on your MySQL master and slave nodes.
Join the DZone community and get the full member experience.Join For Free
In a MySQL master-slave high availability (HA) setup, it is important to continuously monitor the health of the master and slave servers so you can detect potential issues and take corrective actions. In this blog post, we explain some basic health checks you can do on your MySQL master and slave nodes to ensure your setup is healthy. The monitoring program or script must alert the high availability framework in case any of the health checks fail, enabling the high availability framework to take corrective actions to ensure service availability.
You may also like: Pros and Cons of MySQL Replication Types
MySQL Master Server Health Checks
We recommended that your MySQL master monitoring program or scripts run at frequent intervals. Assuming that the monitoring script is running on the same server as your MySQL server, you can check for the following:
Ensure the MySQL Service Is Running
This can be done using a simple command like:
Ensure You Can Connect to MySQL and Do a Simple Query
We recommended having a short timeout for these commands so you can quickly detect if MySQL is unresponsive. This can be achieved from a call like:
/usr/bin/timeout 5 mysql -u testuser -ptestpswd -e 'select * from mysql.test’
Be sure to examine the exit value of the above command:
If the command times out, it means that the MySQL service is not responsive enough. We advise you to retry after some time to avoid false-negative results. If the exit code indicates a failure, the return code from MySQL will tell us the failure reason. One example of a failure is the ‘Too many connections’ error from MySQL which happens if the number of connections to the server exceeds your ‘max_connections’ configuration value.
Ensure the MySQL Master Is Running in Read-Write Mode
You can use the following command to ensure the MySQL master is running in read-write mode:
/usr/bin/timeout 5 mysql -u testuser -ptestpswd -e "SELECT @@global.read_only"
The master is expected to be always running in read-write mode, and hence, the value of read_only should be ‘OFF’.
It is also possible to club this step with step 2, and instead of doing the test query 'select * from MySQL.test, we can just do the query to get the read_only value.
MySQL Slave Server Health Checks
You can run the monitoring for your MySQL slaves at a lesser frequency compared to the master, as they are not handling data writes. The first 3 steps for your slave health check can be the same as that of the master, except that we need to ensure the slave is running in a read-only mode — the value of the variable read_only should be ‘ON’ in step-3.
Also, we can do more checks on the slave to ensure its replication status is healthy, such as:
- The slave is configured to replicate from the right master.
- The slave’s connection to the master is healthy.
- The slave can apply the master events it has received.
It's possible to check for all the above using the ‘show slave status’ command. For example:
- The Master_Host value indicates the master server is configured for replication.
- For the Slave_IO_Running value, "Yes" indicates that the slave has connected to the master and is receiving the replication stream.
- For the Slave_SQL_Running value, "Yes" indicates that the slave’s applier is running and able to apply all the events received from the master.
In this blog post, we discussed some simple checks that can detect if there are basic issues in your MySQL master and slave servers. In general, the failure detection mechanism in a high availability setup is a complex subject and needs a robust high availability framework through which health check monitoring should be implemented.
Published at DZone with permission of Prasad Nagaraj, DZone MVB. See the original article here.
Opinions expressed by DZone contributors are their own.