Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Auto-bootstrapping an all-down cluster with Percona XtraDB Cluster

DZone's Guide to

Auto-bootstrapping an all-down cluster with Percona XtraDB Cluster

· Java Zone
Free Resource

What every Java engineer should know about microservices: Reactive Microservices Architecture.  Brought to you in partnership with Lightbend.

Originally Written by

One new feature in Percona XtraDB Cluster (PXC) in recent releases was the inclusion of the ability for an existing cluster to auto-bootstrap after an all-node-down event.  Suppose you lose power on all nodes simultaneously or something else similar happens to your cluster. Traditionally, this meant manually re-bootstrapping the cluster, but not any more.

How it works

Given the above all-down situation, if all nodes are able to restart and see each other such that they all agree what the state was and that all nodes have returned, then the nodes will make a decision that it is safe for them to recover PRIMARY state as a whole.

This requires:

  • All nodes went down hard — that is; a kill -9, kernel panic, server power failure, or similar event
  • All nodes from the last PRIMARY component are restarted and are able to see each other again.

Demonstration

Suppose I have a 3 node cluster in a stable state. I then kill all nodes simultaneously (simulating a power failure or similar event):

[root@node1 ~]# killall -9 mysqld
[root@node2 ~]# killall -9 mysqld
[root@node3 ~]# killall -9 mysqld

I can see that each node maintained a state file in its datadir called ‘gvwstate.dat’. This contains the last known view of the cluster:

[root@node1 ~]# cat /var/lib/mysql/gvwstate.dat
my_uuid: 78caedfe-75a5-11e4-ac69-fb694ee06530
#vwbeg
view_id: 3 78caedfe-75a5-11e4-ac69-fb694ee06530 9
bootstrap: 0
member: 78caedfe-75a5-11e4-ac69-fb694ee06530 0
member: 87da2387-75a5-11e4-900f-ba49ecdce584 0
member: 8a25acd8-75a5-11e4-9e22-97412a1263ac 0
#vwend

This file will not exist on a node if it was shutdown cleanly, only if the mysqld was uncleanly terminated. This file should exist and be the same on all the nodes for the auto-recovery to work.

I can now restart all 3 nodes more or less at the same time. Note that none of these nodes are bootstrapping and all of the nodes have the wsrep_cluster_address set to a proper list of the nodes in the cluster:

[root@node1 ~]# service mysql start
[root@node2 ~]# service mysql start
[root@node3 ~]# service mysql start

I can indeed see that they all start successfully and enter the primary state:

[root@node1 ~]# mysql -e "show global status like 'wsrep_cluster%'"
+--------------------------+--------------------------------------+
| Variable_name            | Value                                |
+--------------------------+--------------------------------------+
| wsrep_cluster_conf_id    | 0                                    |
| wsrep_cluster_size       | 3                                    |
| wsrep_cluster_state_uuid | 1ba6f69a-759b-11e4-89ba-62a713a26cd1 |
| wsrep_cluster_status     | Primary                              |
+--------------------------+--------------------------------------+
[root@node2 ~]# mysql -e "show global status like 'wsrep_cluster%'"
+--------------------------+--------------------------------------+
| Variable_name            | Value                                |
+--------------------------+--------------------------------------+
| wsrep_cluster_conf_id    | 0                                    |
| wsrep_cluster_size       | 3                                    |
| wsrep_cluster_state_uuid | 1ba6f69a-759b-11e4-89ba-62a713a26cd1 |
| wsrep_cluster_status     | Primary                              |
+--------------------------+--------------------------------------+
[root@node3 ~]# mysql -e "show global status like 'wsrep_cluster%'"
+--------------------------+--------------------------------------+
| Variable_name            | Value                                |
+--------------------------+--------------------------------------+
| wsrep_cluster_conf_id    | 0                                    |
| wsrep_cluster_size       | 3                                    |
| wsrep_cluster_state_uuid | 1ba6f69a-759b-11e4-89ba-62a713a26cd1 |
| wsrep_cluster_status     | Primary                              |
+--------------------------+--------------------------------------+

Checking the logs, I can see this indication that the feature is working:

2014-11-26 19:59:36 1809 [Note] WSREP: promote to primary component
2014-11-26 19:59:36 1809 [Note] WSREP: view(view_id(PRIM,78caedfe,13) memb {
78caedfe,0
87da2387,0
8a25acd8,0
} joined {
} left {
} partitioned {
})
2014-11-26 19:59:36 1809 [Note] WSREP: save pc into disk
2014-11-26 19:59:36 1809 [Note] WSREP: clear restored view

Changing this behavior

This feature is enabled by default, but you can toggle it off with the pc.recovery setting in the wsrep_provider_options

This feature helps cover an edge case where manual bootstrapping was necessary in the past to recovery properly. This feature was added in Percona XtraDB Cluster version 5.6.19, but was broken due to this bug.  It was fixed in PXC 5.6.21


Microservices for Java, explained. Revitalize your legacy systems (and your career) with Reactive Microservices Architecture, a free O'Reilly book. Brought to you in partnership with Lightbend.

Topics:

Published at DZone with permission of Peter Zaitsev, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

THE DZONE NEWSLETTER

Dev Resources & Solutions Straight to Your Inbox

Thanks for subscribing!

Awesome! Check your inbox to verify your email so you can start receiving the latest in tech news and resources.

X

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}