# [译]PXC7中故障场景及恢复方法

## 情景3

or in packages using systemd service manager (Centos7 at the moment):

In older PXC versions, to bootstrap cluster, you had to edit my.cnf and replace previous wsrep_cluster_address line with empty value like this: wsrep_cluster_address=gcomm:// and start mysql normally. More details to be found here.

Please note that even if you bootstrap from the most advanced node, so the other nodes have lower sequence number, they will have to still join via full-SST because the Galera Cache is not retained on restart. For that reason, it is recommended to stop writes to the cluster before it’s full shutdown, so that all nodes stop in the same position. Edit: This changes since Galera 3.19 thanks to gcache-recover option

## 情景5

A,B节点均从集群中消失. The node C is not able to form the quorum alone, 集群切换到non-primary模式, 拒绝所有应用请求. 在这种情景下, C节点的mysqld进程仍在运行, 你可以连接到C节点的mysql, 但是执行任何语句都将失败

However, you should double check in order to be very sure the other nodes are really down before doing that! Otherwise, you will most likely end up with two clusters having different data.

## 情景6

So the last committed transaction sequence number on this node was 2. Now you just need to bootstrap from the latest node first and then start the others.

However, the above procedure won’t be needed in the recent Galera versions (3.6+?), available since PXC 5.6.19. There is a new option – pc.recovery (enabled by default), which saves the cluster state into a file named gvwstate.dat on each member node. As the variable name says (pc – primary component), it saves only a cluster being in PRIMARY state. An example content of that file may look like this:

We can see three node cluster above with all members being up. Thanks to this new feature, in the case of power outage in our datacenter, after power is back, the nodes will read the last state on startup and will try to restore primary component once all the members again start to see each other. This makes the PXC cluster to automatically recover from being powered down without any manual intervention! In the logs we will see:

## 情景7

Cluster lost it’s primary state due to split brain situation.为了举例, 我们假设由偶数个节点形成- 6个, ABC在一个机房, DEF在另一个机房, 两个机房之间的网络中断了. 当然最好的方案是避免这种拓扑结构, 如果没机器了实在不行还可以用arbitrator (garbd) node或者调整pc.weight参数.但是，当分裂大脑以任何方式发生时，所有分离的组都不能维持法定人数 - 所有节点都必须停止提供请求，并且这两个部分只是不断尝试重新连接。如果要在恢复网络链接之前恢复服务，则可以使用与方案5中相同的命令使其中一个组再次成为主服务器：

After that, you are able to work on the manually restored part of the cluster, and the second half should be able to automatically re-join using incremental state transfer (IST) once the network link is restored. But beware: if you set the bootstrap option on both the separated parts, you will end up with two living cluster instances, with data likely diverging away from each other. Restoring network link in that case won’t make them to re-join until nodes are restarted and try to re-connect to members specified in configuration file. Then, as Galera replication model truly cares about data consistency – once the inconsistency will be detected, nodes that cannot execute row change statement due to different data – will perform emergency shutdown and the only way to bring them back to the cluster will be via full SST.

I hope I covered most of the possible failure scenarios of Galera-based clusters, and made the recovery procedures bit more clear.

https://www.percona.com/blog/2014/09/01/galera-replication-how-to-recover-a-pxc-cluster/

#