On 25/11/15 15:22, Jonathan Davies wrote: > Hi, > > I'm experimenting with corosync+dlm+gfs2 (approximately following > http://people.redhat.com/teigland/cluster4-gfs2-dlm.txt) and am trying > to establish whether it meets my requirements. I have a query about a > node rejoining a cluster after failure, and want to make sure I'm not > overlooking something. > > I have a three-node cluster and deliberately cause token loss by > firewalling one of them (call it node A) out of the network for longer > than the token timeout. At this point, the other two hosts (B and C) > decide that A has disappeared and continue with quorum. That is fine. > > When I unfirewall node A, dlm tries to reconnect to its peers on B and > C. But then I see the following on host B: > > 16:29:25.823496 nodeb dlm_controld[6548]: 908 daemon node 85 stateful merge > 16:29:25.823529 nodeb dlm_controld[6548]: 908 daemon node 85 kill due to > stateful merge > 16:29:25.823543 nodeb dlm_controld[6548]: 908 tell corosync to remove > nodeid 85 from cluster > 16:29:25.823696 nodeb corosync[6536]: [CFG ] request to kill node > 85(us=83): xxx > > and then the following on node A: > > 16:29:25.828547 nodea corosync[3896]: [CFG ] Killed by node 83: > dlm_controld > 16:29:25.828575 nodea corosync[3896]: [MAIN ] Corosync Cluster Engine > exiting with status -1 at cfg.c:530. > 16:29:25.834828 nodea dlm_controld[3466]: 1183 process_cluster_cfg > cfg_dispatch 2 > 16:29:25.834871 nodea dlm_controld[3466]: 1183 cluster is down, exiting > 16:29:25.834886 nodea dlm_controld[3466]: 1183 process_cluster > quorum_dispatch 2 > 16:29:25.834903 nodea dlm_controld[3466]: 1183 daemon cpg_dispatch error 2 > 16:29:25.834917 nodea dlm_controld[3466]: 1183 cpg_dispatch error 2 > 16:29:25.837152 nodea dlm_controld[3466]: 1183 abandoned lockspace mygfs2 > > resulting in both corosync and dlm_controld exiting on node A. > > Later, if I try to manually restart corosync and dlm on node A, I see > the following: > > 16:32:08.382871 nodea dlm_controld[20483]: 2872 dlm_controld 4.0.2 started > 16:32:08.392453 nodea dlm_controld[20483]: 2872 found uncontrolled > lockspace mygfs2 > 16:32:08.392477 nodea dlm_controld[20483]: 2872 tell corosync to remove > nodeid 85 from cluster > 16:32:08.394965 nodea corosync[20456]: [CFG ] request to kill node > 85(us=85): xxx > 16:32:08.394998 nodea corosync[20456]: [CFG ] Killed by node 85: > dlm_controld > > The only way of making A rejoin the cluster is to reboot. > Yes. You need to implement fencing, so that the node will automatically be restarted when it leaves the cluster. CHrissie > I would be grateful if you could confirm the following statements: > (a) The "stateful merge" is unavoidable when node A leaves the cluster > for longer than the token timeout then tries to rejoin. > (b) Killing corosync on node A is unavoidable when node B sees the > "stateful merge". > (c) dlm exiting is unavoidable when corosync dies. > (d) Restarting corosync then dlm on node A will necessarily result in > "found uncontrolled lockspace". > (e) The only way to recover from "found uncontrolled lockspace" (for a > gfs2 lockspace) is to reboot. > > I'm hoping that I'm overlooking something and that at least one of > (a)--(e) is false! I'm not comfortable with a reboot being the only > means of recovery when the token timeout is exceeded. > > Thanks, > Jonathan > -- Linux-cluster mailing list Linux-cluster@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/linux-cluster
