Node4 was fenced again, I was able to get some debug logs (below), a new message : "Jun 12 14:01:56 corosync [TOTEM ] The token was lost in the OPERATIONAL state.“ Rest of corosync logs http://pastebin.com/iYFbkbhb Jun 12 14:44:49 corosync [TOTEM ] entering OPERATIONAL state. Jun 12 14:44:49 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed. Jun 12 14:44:49 corosync [TOTEM ] waiting_trans_ack changed to 0 Jun 12 14:44:49 corosync [TOTEM ] Process pause detected for 32947 ms, flushing membership messages. Jun 12 14:44:49 corosync [TOTEM ] entering GATHER state from 12. Jun 12 14:44:49 corosync [TOTEM ] Process pause detected for 32947 ms, flushing membership messages. Jun 12 14:44:49 corosync [TOTEM ] Process pause detected for 32947 ms, flushing membership messages. Jun 12 14:44:49 corosync [TOTEM ] Process pause detected for 33016 ms, flushing membership messages. Jun 12 14:44:49 corosync [TOTEM ] Process pause detected for 33016 ms, flushing membership messages. Jun 12 14:44:49 corosync [TOTEM ] Process pause detected for 33016 ms, flushing membership messages. Jun 12 14:44:49 corosync [TOTEM ] Process pause detected for 33016 ms, flushing membership messages. Jun 12 14:44:49 corosync [TOTEM ] Process pause detected for 33086 ms, flushing membership messages. Jun 12 14:44:49 corosync [TOTEM ] Process pause detected for 33086 ms, flushing membership messages. Jun 12 14:44:49 corosync [TOTEM ] Process pause detected for 33086 ms, flushing membership messages. Jun 12 14:44:49 corosync [TOTEM ] Process pause detected for 33086 ms, flushing membership messages. Jun 12 14:44:49 corosync [TOTEM ] Process pause detected for 33155 ms, flushing membership messages. Jun 12 14:44:49 corosync [TOTEM ] Process pause detected for 33155 ms, flushing membership messages. Jun 12 14:44:49 corosync [TOTEM ] Process pause detected for 33155 ms, flushing membership messages. Jun 12 14:44:49 corosync [TOTEM ] Process pause detected for 33155 ms, flushing membership messages. Jun 12 14:44:50 corosync [TOTEM ] Process pause detected for 33224 ms, flushing membership messages. Jun 12 14:44:50 corosync [TOTEM ] Process pause detected for 33224 ms, flushing membership messages. Jun 12 14:44:50 corosync [TOTEM ] Process pause detected for 33225 ms, flushing membership messages. Jun 12 14:44:50 corosync [TOTEM ] Process pause detected for 33225 ms, flushing membership messages. Jun 12 14:44:50 corosync [TOTEM ] Process pause detected for 33294 ms, flushing membership messages. Jun 12 14:44:50 corosync [TOTEM ] Process pause detected for 33294 ms, flushing membership messages. Jun 12 14:44:50 corosync [TOTEM ] Process pause detected for 33294 ms, flushing membership messages. Jun 12 14:44:50 corosync [TOTEM ] Process pause detected for 33294 ms, flushing membership messages. Jun 12 14:44:50 corosync [TOTEM ] Process pause detected for 33363 ms, flushing membership messages. Jun 12 14:44:50 corosync [TOTEM ] Process pause detected for 33363 ms, flushing membership messages. Jun 12 14:44:50 corosync [TOTEM ] Process pause detected for 33363 ms, flushing membership messages. Jun 12 14:44:50 corosync [TOTEM ] Process pause detected for 33432 ms, flushing membership messages. Jun 12 14:44:50 corosync [TOTEM ] Process pause detected for 33432 ms, flushing membership messages. Jun 12 14:44:50 corosync [TOTEM ] Process pause detected for 33432 ms, flushing membership messages. Jun 12 14:44:50 corosync [TOTEM ] Process pause detected for 33494 ms, flushing membership messages. Jun 12 14:44:50 corosync [TOTEM ] Process pause detected for 33495 ms, flushing membership messages. Jun 12 14:44:50 corosync [TOTEM ] Process pause detected for 33495 ms, flushing membership messages. Jun 12 14:44:50 corosync [TOTEM ] Process pause detected for 33495 ms, flushing membership messages. Jun 12 14:44:50 corosync [TOTEM ] Process pause detected for 33564 ms, flushing membership messages. Jun 12 14:44:50 corosync [TOTEM ] Process pause detected for 33564 ms, flushing membership messages. Jun 12 14:44:50 corosync [TOTEM ] Process pause detected for 33564 ms, flushing membership messages. Jun 12 14:44:50 corosync [TOTEM ] Process pause detected for 33564 ms, flushing membership messages. Jun 12 14:44:50 corosync [TOTEM ] got commit token Jun 12 14:44:50 corosync [TOTEM ] Saving state aru 86 high seq received 86 Jun 12 14:44:50 corosync [TOTEM ] Storing new sequence id for ring 6324 Jun 12 14:44:50 corosync [TOTEM ] entering COMMIT state. Jun 12 14:44:50 corosync [TOTEM ] got commit token Jun 12 14:44:50 corosync [TOTEM ] entering RECOVERY state. Jun 12 14:44:50 corosync [TOTEM ] TRANS [0] member 10.70.100.101: Jun 12 14:44:50 corosync [TOTEM ] TRANS [1] member 10.70.100.102: Jun 12 14:44:50 corosync [TOTEM ] TRANS [2] member 10.70.100.103: Jun 12 14:44:50 corosync [TOTEM ] TRANS [3] member 10.70.100.104: Jun 12 14:44:50 corosync [TOTEM ] position [0] member 10.70.100.101: Jun 12 14:44:50 corosync [TOTEM ] previous ring seq 25376 rep 10.70.100.101 Jun 12 14:44:50 corosync [TOTEM ] aru 86 high delivered 86 received flag 1 Jun 12 14:44:50 corosync [TOTEM ] position [1] member 10.70.100.102: Jun 12 14:44:50 corosync [TOTEM ] previous ring seq 25376 rep 10.70.100.101 Jun 12 14:44:50 corosync [TOTEM ] aru 86 high delivered 86 received flag 1 Jun 12 14:44:50 corosync [TOTEM ] position [2] member 10.70.100.103: Jun 12 14:44:50 corosync [TOTEM ] previous ring seq 25376 rep 10.70.100.101 Jun 12 14:44:50 corosync [TOTEM ] aru 86 high delivered 86 received flag 1 Jun 12 14:44:50 corosync [TOTEM ] position [3] member 10.70.100.104: Jun 12 14:44:50 corosync [TOTEM ] previous ring seq 25376 rep 10.70.100.101 Jun 12 14:44:50 corosync [TOTEM ] aru 86 high delivered 86 received flag 1 Jun 12 14:44:50 corosync [TOTEM ] Did not need to originate any messages in recovery. Jun 12 14:44:50 corosync [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 0, aru ffffffff Jun 12 14:44:50 corosync [TOTEM ] install seq 0 aru 0 high seq received 0 Jun 12 14:44:50 corosync [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 1, aru 0 Jun 12 14:44:50 corosync [TOTEM ] install seq 0 aru 0 high seq received 0 Jun 12 14:44:50 corosync [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 2, aru 0 Jun 12 14:44:50 corosync [TOTEM ] install seq 0 aru 0 high seq received 0 Jun 12 14:44:50 corosync [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 3, aru 0 Jun 12 14:44:50 corosync [TOTEM ] install seq 0 aru 0 high seq received 0 Jun 12 14:44:50 corosync [TOTEM ] retrans flag count 4 token aru 0 install seq 0 aru 0 0 Jun 12 14:44:50 corosync [TOTEM ] Resetting old ring state Jun 12 14:44:50 corosync [TOTEM ] recovery to regular 1-0 Jun 12 14:44:50 corosync [TOTEM ] waiting_trans_ack changed to 1 Jun 12 14:44:50 corosync [TOTEM ] entering OPERATIONAL state. Jun 12 14:44:50 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed. Jun 12 14:44:50 corosync [TOTEM ] waiting_trans_ack changed to 0 Jun 12 14:44:51 corosync [TOTEM ] Process pause detected for 34338 ms, flushing membership messages. Jun 12 14:44:51 corosync [TOTEM ] entering GATHER state from 12. Jun 12 14:44:51 corosync [TOTEM ] Process pause detected for 34338 ms, flushing membership messages. Jun 12 14:44:51 corosync [TOTEM ] Process pause detected for 34338 ms, flushing membership messages. Jun 12 14:44:51 corosync [TOTEM ] Process pause detected for 34338 ms, flushing membership messages. Jun 12 14:44:51 corosync [TOTEM ] Process pause detected for 34407 ms, flushing membership messages. Jun 12 14:44:51 corosync [TOTEM ] Process pause detected for 34407 ms, flushing membership messages. Jun 12 14:44:51 corosync [TOTEM ] Process pause detected for 34407 ms, flushing membership messages. Jun 12 14:44:51 corosync [TOTEM ] Process pause detected for 34407 ms, flushing membership messages. Jun 12 14:44:51 corosync [TOTEM ] got commit token Jun 12 14:44:51 corosync [TOTEM ] Saving state aru 86 high seq received 86 Jun 12 14:44:51 corosync [TOTEM ] Storing new sequence id for ring 6328 Jun 12 14:44:51 corosync [TOTEM ] entering COMMIT state. Jun 12 14:44:51 corosync [TOTEM ] got commit token Jun 12 14:44:51 corosync [TOTEM ] entering RECOVERY state. Jun 12 14:44:51 corosync [TOTEM ] TRANS [0] member 10.70.100.101: Jun 12 14:44:51 corosync [TOTEM ] TRANS [1] member 10.70.100.102: Jun 12 14:44:51 corosync [TOTEM ] TRANS [2] member 10.70.100.103: Jun 12 14:44:51 corosync [TOTEM ] TRANS [3] member 10.70.100.104: Jun 12 14:44:51 corosync [TOTEM ] position [0] member 10.70.100.101: Jun 12 14:44:51 corosync [TOTEM ] previous ring seq 25380 rep 10.70.100.101 Jun 12 14:44:51 corosync [TOTEM ] aru 86 high delivered 86 received flag 1 Jun 12 14:44:51 corosync [TOTEM ] position [1] member 10.70.100.102: Jun 12 14:44:51 corosync [TOTEM ] previous ring seq 25380 rep 10.70.100.101 Jun 12 14:44:51 corosync [TOTEM ] aru 86 high delivered 86 received flag 1 Jun 12 14:44:51 corosync [TOTEM ] position [2] member 10.70.100.103: Jun 12 14:44:51 corosync [TOTEM ] previous ring seq 25380 rep 10.70.100.101 Jun 12 14:44:51 corosync [TOTEM ] aru 86 high delivered 86 received flag 1 Jun 12 14:44:51 corosync [TOTEM ] position [3] member 10.70.100.104: Jun 12 14:44:51 corosync [TOTEM ] previous ring seq 25380 rep 10.70.100.101 Jun 12 14:44:51 corosync [TOTEM ] aru 86 high delivered 86 received flag 1 Jun 12 14:44:51 corosync [TOTEM ] Did not need to originate any messages in recovery. Jun 12 14:44:51 corosync [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 0, aru ffffffff Jun 12 14:44:51 corosync [TOTEM ] install seq 0 aru 0 high seq received 0 Jun 12 14:44:51 corosync [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 1, aru 0 Jun 12 14:44:51 corosync [TOTEM ] install seq 0 aru 0 high seq received 0 Jun 12 14:44:51 corosync [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 2, aru 0 Jun 12 14:44:51 corosync [TOTEM ] install seq 0 aru 0 high seq received 0 Jun 12 14:44:51 corosync [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 3, aru 0 Jun 12 14:44:51 corosync [TOTEM ] install seq 0 aru 0 high seq received 0 Jun 12 14:44:51 corosync [TOTEM ] retrans flag count 4 token aru 0 install seq 0 aru 0 0 Jun 12 14:44:51 corosync [TOTEM ] Resetting old ring state Jun 12 14:44:51 corosync [TOTEM ] recovery to regular 1-0 Jun 12 14:44:51 corosync [TOTEM ] waiting_trans_ack changed to 1 Jun 12 14:44:51 corosync [TOTEM ] entering OPERATIONAL state. Jun 12 14:44:51 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed. Jun 12 14:44:51 corosync [TOTEM ] waiting_trans_ack changed to 0 Jun 12 14:44:52 corosync [TOTEM ] Process pause detected for 35177 ms, flushing membership messages. Jun 12 14:44:52 corosync [TOTEM ] entering GATHER state from 12. Jun 12 14:44:52 corosync [TOTEM ] Process pause detected for 35177 ms, flushing membership messages. Jun 12 14:44:52 corosync [TOTEM ] Process pause detected for 35246 ms, flushing membership messages. Jun 12 14:44:52 corosync [TOTEM ] Process pause detected for 35246 ms, flushing membership messages. Jun 12 14:44:52 corosync [TOTEM ] Process pause detected for 35316 ms, flushing membership messages. Jun 12 14:44:52 corosync [TOTEM ] Process pause detected for 35316 ms, flushing membership messages. Jun 12 14:44:52 corosync [TOTEM ] Process pause detected for 35385 ms, flushing membership messages. Jun 12 14:44:52 corosync [TOTEM ] Process pause detected for 35385 ms, flushing membership messages. Jun 12 14:44:52 corosync [TOTEM ] Process pause detected for 35385 ms, flushing membership messages. Jun 12 14:44:52 corosync [TOTEM ] Process pause detected for 35454 ms, flushing membership messages. Jun 12 14:44:52 corosync [TOTEM ] Process pause detected for 35454 ms, flushing membership messages. Jun 12 14:44:52 corosync [TOTEM ] Process pause detected for 35454 ms, flushing membership messages. Jun 12 14:44:52 corosync [TOTEM ] Process pause detected for 35455 ms, flushing membership messages. Jun 12 14:44:52 corosync [TOTEM ] got commit token Jun 12 14:44:52 corosync [TOTEM ] Saving state aru 86 high seq received 86 Jun 12 14:44:52 corosync [TOTEM ] Storing new sequence id for ring 632c Jun 12 14:44:52 corosync [TOTEM ] entering COMMIT state. Jun 12 14:44:52 corosync [TOTEM ] got commit token Jun 12 14:44:52 corosync [TOTEM ] entering RECOVERY state. Jun 12 14:44:52 corosync [TOTEM ] TRANS [0] member 10.70.100.101: Jun 12 14:44:52 corosync [TOTEM ] TRANS [1] member 10.70.100.102: Jun 12 14:44:52 corosync [TOTEM ] TRANS [2] member 10.70.100.103: Jun 12 14:44:52 corosync [TOTEM ] TRANS [3] member 10.70.100.104: Jun 12 14:44:52 corosync [TOTEM ] position [0] member 10.70.100.101: Jun 12 14:44:52 corosync [TOTEM ] previous ring seq 25384 rep 10.70.100.101 Jun 12 14:44:52 corosync [TOTEM ] aru 86 high delivered 86 received flag 1 Jun 12 14:44:52 corosync [TOTEM ] position [1] member 10.70.100.102: Jun 12 14:44:52 corosync [TOTEM ] previous ring seq 25384 rep 10.70.100.101 Jun 12 14:44:52 corosync [TOTEM ] aru 86 high delivered 86 received flag 1 Jun 12 14:44:52 corosync [TOTEM ] position [2] member 10.70.100.103: Jun 12 14:44:52 corosync [TOTEM ] previous ring seq 25384 rep 10.70.100.101 Jun 12 14:44:52 corosync [TOTEM ] aru 86 high delivered 86 received flag 1 Jun 12 14:44:52 corosync [TOTEM ] position [3] member 10.70.100.104: Jun 12 14:44:52 corosync [TOTEM ] previous ring seq 25384 rep 10.70.100.101 Jun 12 14:44:52 corosync [TOTEM ] aru 86 high delivered 86 received flag 1 Jun 12 14:44:52 corosync [TOTEM ] Did not need to originate any messages in recovery. Jun 12 14:44:52 corosync [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 0, aru ffffffff Jun 12 14:44:52 corosync [TOTEM ] install seq 0 aru 0 high seq received 0 Jun 12 14:44:52 corosync [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 1, aru 0 Jun 12 14:44:52 corosync [TOTEM ] install seq 0 aru 0 high seq received 0 Jun 12 14:44:52 corosync [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 2, aru 0 Jun 12 14:44:52 corosync [TOTEM ] install seq 0 aru 0 high seq received 0 Jun 12 14:44:52 corosync [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 3, aru 0 Jun 12 14:44:52 corosync [TOTEM ] install seq 0 aru 0 high seq received 0 Jun 12 14:44:52 corosync [TOTEM ] retrans flag count 4 token aru 0 install seq 0 aru 0 0 Jun 12 14:44:52 corosync [TOTEM ] Resetting old ring state Jun 12 14:44:52 corosync [TOTEM ] recovery to regular 1-0 Jun 12 14:44:52 corosync [TOTEM ] waiting_trans_ack changed to 1 Jun 12 14:44:52 corosync [TOTEM ] entering OPERATIONAL state. Jun 12 14:44:52 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed. Jun 12 14:44:52 corosync [TOTEM ] waiting_trans_ack changed to 0 Jun 12 14:44:53 corosync [TOTEM ] Process pause detected for 36223 ms, flushing membership messages. Jun 12 14:44:53 corosync [TOTEM ] entering GATHER state from 12. Jun 12 14:44:53 corosync [TOTEM ] Process pause detected for 36224 ms, flushing membership messages. Jun 12 14:44:53 corosync [TOTEM ] Process pause detected for 36293 ms, flushing membership messages. Jun 12 14:44:53 corosync [TOTEM ] Process pause detected for 36293 ms, flushing membership messages. Jun 12 14:44:53 corosync [TOTEM ] Process pause detected for 36293 ms, flushing membership messages. Jun 12 14:44:53 corosync [TOTEM ] Process pause detected for 36293 ms, flushing membership messages. Jun 12 14:44:53 corosync [TOTEM ] Process pause detected for 36362 ms, flushing membership messages. Jun 12 14:44:53 corosync [TOTEM ] Process pause detected for 36362 ms, flushing membership messages. Jun 12 14:44:53 corosync [TOTEM ] Process pause detected for 36362 ms, flushing membership messages. Jun 12 14:44:53 corosync [TOTEM ] Process pause detected for 36362 ms, flushing membership messages. Jun 12 14:44:53 corosync [TOTEM ] Process pause detected for 36431 ms, flushing membership messages. Jun 12 14:44:53 corosync [TOTEM ] Process pause detected for 36431 ms, flushing membership messages. Jun 12 14:44:53 corosync [TOTEM ] Process pause detected for 36432 ms, flushing membership messages. Jun 12 14:44:53 corosync [TOTEM ] Process pause detected for 36432 ms, flushing membership messages. Jun 12 14:44:53 corosync [TOTEM ] Process pause detected for 36501 ms, flushing membership messages. Jun 12 14:44:53 corosync [TOTEM ] Process pause detected for 36501 ms, flushing membership messages. Jun 12 14:44:53 corosync [TOTEM ] Process pause detected for 36501 ms, flushing membership messages. Jun 12 14:44:53 corosync [TOTEM ] Process pause detected for 36501 ms, flushing membership messages. Jun 12 14:44:53 corosync [TOTEM ] got commit token Jun 12 14:44:53 corosync [TOTEM ] Saving state aru 86 high seq received 86 Jun 12 14:44:53 corosync [TOTEM ] Storing new sequence id for ring 6330 Jun 12 14:44:53 corosync [TOTEM ] entering COMMIT state. Jun 12 14:44:53 corosync [TOTEM ] got commit token Jun 12 14:44:53 corosync [TOTEM ] entering RECOVERY state. Jun 12 14:44:53 corosync [TOTEM ] TRANS [0] member 10.70.100.101: Jun 12 14:44:53 corosync [TOTEM ] TRANS [1] member 10.70.100.102: Jun 12 14:44:53 corosync [TOTEM ] TRANS [2] member 10.70.100.103: Jun 12 14:44:53 corosync [TOTEM ] TRANS [3] member 10.70.100.104: Jun 12 14:44:53 corosync [TOTEM ] position [0] member 10.70.100.101: Jun 12 14:44:53 corosync [TOTEM ] previous ring seq 25388 rep 10.70.100.101 Jun 12 14:44:53 corosync [TOTEM ] aru 86 high delivered 86 received flag 1 Jun 12 14:44:53 corosync [TOTEM ] position [1] member 10.70.100.102: Jun 12 14:44:53 corosync [TOTEM ] previous ring seq 25388 rep 10.70.100.101 Jun 12 14:44:53 corosync [TOTEM ] aru 86 high delivered 86 received flag 1 Jun 12 14:44:53 corosync [TOTEM ] position [2] member 10.70.100.103: Jun 12 14:44:53 corosync [TOTEM ] previous ring seq 25388 rep 10.70.100.101 Jun 12 14:44:53 corosync [TOTEM ] aru 86 high delivered 86 received flag 1 Jun 12 14:44:53 corosync [TOTEM ] position [3] member 10.70.100.104: Jun 12 14:44:53 corosync [TOTEM ] previous ring seq 25388 rep 10.70.100.101 Jun 12 14:44:53 corosync [TOTEM ] aru 86 high delivered 86 received flag 1 Jun 12 14:44:53 corosync [TOTEM ] Did not need to originate any messages in recovery. Jun 12 14:44:53 corosync [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 0, aru ffffffff Jun 12 14:44:53 corosync [TOTEM ] install seq 0 aru 0 high seq received 0 Jun 12 14:44:53 corosync [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 1, aru 0 Jun 12 14:44:53 corosync [TOTEM ] install seq 0 aru 0 high seq received 0 Jun 12 14:44:53 corosync [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 2, aru 0 Jun 12 14:44:53 corosync [TOTEM ] install seq 0 aru 0 high seq received 0 Jun 12 14:44:53 corosync [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 3, aru 0 Jun 12 14:44:53 corosync [TOTEM ] install seq 0 aru 0 high seq received 0 Jun 12 14:44:53 corosync [TOTEM ] retrans flag count 4 token aru 0 install seq 0 aru 0 0 Jun 12 14:44:53 corosync [TOTEM ] Resetting old ring state Jun 12 14:44:53 corosync [TOTEM ] recovery to regular 1-0 Jun 12 14:44:53 corosync [TOTEM ] waiting_trans_ack changed to 1 Jun 12 14:44:53 corosync [TOTEM ] entering OPERATIONAL state. Jun 12 14:44:53 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed. Jun 12 14:44:53 corosync [TOTEM ] waiting_trans_ack changed to 0 Jun 12 14:44:54 corosync [TOTEM ] Process pause detected for 37267 ms, flushing membership messages. Jun 12 14:44:54 corosync [TOTEM ] entering GATHER state from 12. Jun 12 14:44:54 corosync [TOTEM ] Process pause detected for 37267 ms, flushing membership messages. Jun 12 14:44:54 corosync [TOTEM ] Process pause detected for 37268 ms, flushing membership messages. Jun 12 14:44:54 corosync [TOTEM ] Process pause detected for 37268 ms, flushing membership messages. Jun 12 14:44:54 corosync [TOTEM ] Process pause detected for 37337 ms, flushing membership messages. Jun 12 14:44:54 corosync [TOTEM ] Process pause detected for 37337 ms, flushing membership messages. Jun 12 14:44:54 corosync [TOTEM ] got commit token Jun 12 14:44:54 corosync [TOTEM ] Saving state aru 86 high seq received 86 Jun 12 14:44:54 corosync [TOTEM ] Storing new sequence id for ring 6334 Jun 12 14:44:54 corosync [TOTEM ] entering COMMIT state. Jun 12 14:44:54 corosync [TOTEM ] got commit token Jun 12 14:44:54 corosync [TOTEM ] entering RECOVERY state. Jun 12 14:44:54 corosync [TOTEM ] TRANS [0] member 10.70.100.101: Jun 12 14:44:54 corosync [TOTEM ] TRANS [1] member 10.70.100.102: Jun 12 14:44:54 corosync [TOTEM ] TRANS [2] member 10.70.100.103: Jun 12 14:44:54 corosync [TOTEM ] TRANS [3] member 10.70.100.104: Jun 12 14:44:54 corosync [TOTEM ] position [0] member 10.70.100.101: Jun 12 14:44:54 corosync [TOTEM ] previous ring seq 25392 rep 10.70.100.101 Jun 12 14:44:54 corosync [TOTEM ] aru 86 high delivered 86 received flag 1 Jun 12 14:44:54 corosync [TOTEM ] position [1] member 10.70.100.102: Jun 12 14:44:54 corosync [TOTEM ] previous ring seq 25392 rep 10.70.100.101 Jun 12 14:44:54 corosync [TOTEM ] aru 86 high delivered 86 received flag 1 Jun 12 14:44:54 corosync [TOTEM ] position [2] member 10.70.100.103: Jun 12 14:44:54 corosync [TOTEM ] previous ring seq 25392 rep 10.70.100.101 Jun 12 14:44:54 corosync [TOTEM ] aru 86 high delivered 86 received flag 1 Jun 12 14:44:54 corosync [TOTEM ] position [3] member 10.70.100.104: Jun 12 14:44:54 corosync [TOTEM ] previous ring seq 25392 rep 10.70.100.101 Jun 12 14:44:54 corosync [TOTEM ] aru 86 high delivered 86 received flag 1 Jun 12 14:44:54 corosync [TOTEM ] Did not need to originate any messages in recovery. Jun 12 14:44:54 corosync [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 0, aru ffffffff Jun 12 14:44:54 corosync [TOTEM ] install seq 0 aru 0 high seq received 0 Jun 12 14:44:54 corosync [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 1, aru 0 Jun 12 14:44:54 corosync [TOTEM ] install seq 0 aru 0 high seq received 0 Jun 12 14:44:54 corosync [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 2, aru 0 Jun 12 14:44:54 corosync [TOTEM ] install seq 0 aru 0 high seq received 0 Jun 12 14:44:54 corosync [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 3, aru 0 Jun 12 14:44:54 corosync [TOTEM ] install seq 0 aru 0 high seq received 0 Jun 12 14:44:54 corosync [TOTEM ] retrans flag count 4 token aru 0 install seq 0 aru 0 0 Jun 12 14:44:54 corosync [TOTEM ] Resetting old ring state Jun 12 14:44:54 corosync [TOTEM ] recovery to regular 1-0 Jun 12 14:44:54 corosync [TOTEM ] waiting_trans_ack changed to 1 Jun 12 14:44:54 corosync [TOTEM ] entering OPERATIONAL state. Jun 12 14:44:54 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed. Jun 12 14:44:54 corosync [TOTEM ] waiting_trans_ack changed to 0 Jun 12 14:44:54 corosync [TOTEM ] Process pause detected for 38108 ms, flushing membership messages. Jun 12 14:44:54 corosync [TOTEM ] entering GATHER state from 12. Jun 12 14:44:54 corosync [TOTEM ] Process pause detected for 38108 ms, flushing membership messages. Jun 12 14:44:54 corosync [TOTEM ] Process pause detected for 38108 ms, flushing membership messages. Jun 12 14:44:54 corosync [TOTEM ] Process pause detected for 38109 ms, flushing membership messages. On 6/12/14, 1:55 PM, "Schaefer, Micah" <Micah.Schaefer@xxxxxxxxxx> wrote: >I just found that the clock on node1 was off by about a minute and a half >compared to the rest of the nodes. > >I am running ntp, so not sure why the time wasn’t synced up. Wonder if >node1 being behind, would think it was not receiving updates from the >other nodes? > > > > > > > >On 6/12/14, 1:29 PM, "Digimer" <lists@xxxxxxxxxx> wrote: > >>Even if the token changes stop the immediate fencing, don't leave it >>please. There is something fundamentally wrong that you need to >>identify/fix. >> >>Keep us posted! >> >>On 12/06/14 01:24 PM, Schaefer, Micah wrote: >>> The servers do not run any tasks other than the tasks in the cluster >>> service group. >>> >>> Nodes 3 and 4 are physical servers with a lot of horsepower and nodes 1 >>> and 2 are virtual machines with much less resources available. >>> >>> I adjusted the token settings and will watch for any change. >>> >>> >>> >>> >>> >>> >>> >>> >>> On 6/12/14, 1:08 PM, "Digimer" <lists@xxxxxxxxxx> wrote: >>> >>>> On 12/06/14 12:48 PM, Schaefer, Micah wrote: >>>>> As far as the switch goes, both are Cisco Catalyst 6509-E, no >>>>>spanning >>>>> tree changes are happening and all the ports have port-fast enabled >>>>>for >>>>> these servers. My switch logging level is very high and I have no >>>>> messages >>>>> in relation to the time frames or ports. >>>>> >>>>> TOTEM reports that ³A processor joined or left the membershipŠ², but >>>>> that >>>>> isn¹t enough detail. >>>>> >>>>> Also note that I did not have these issues until adding new servers: >>>>> node3 >>>>> and node4 to the cluster. Node1 and node2 do not fence each other >>>>> (unless >>>>> a real issue is there), and they are on different switches. >>>> >>>> Then I can't imagine it being network anymore. Seeing as both node 3 >>>>and >>>> 4 get fenced, it's likely not hardware either. Are the workloads on 3 >>>> and 4 much higher (or are the computers much slower) than 1 and 2? I'm >>>> wondering if the nodes are simply not keeping up with corosync >>>>traffic. >>>> You might try adjusting the corosync token timeout and retransmit >>>>counts >>>> to see if that reduces the node loses. >>>> >>>> -- >>>> Digimer >>>> Papers and Projects: https://alteeve.ca/w/ >>>> What if the cure for cancer is trapped in the mind of a person without >>>> access to education? >>>> >>>> -- >>>> Linux-cluster mailing list >>>> Linux-cluster@xxxxxxxxxx >>>> https://www.redhat.com/mailman/listinfo/linux-cluster >>> >>> >> >> >>-- >>Digimer >>Papers and Projects: https://alteeve.ca/w/ >>What if the cure for cancer is trapped in the mind of a person without >>access to education? >> >>-- >>Linux-cluster mailing list >>Linux-cluster@xxxxxxxxxx >>https://www.redhat.com/mailman/listinfo/linux-cluster > > >-- >Linux-cluster mailing list >Linux-cluster@xxxxxxxxxx >https://www.redhat.com/mailman/listinfo/linux-cluster -- Linux-cluster mailing list Linux-cluster@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/linux-cluster
