I am attaching the relevant log entries from a node that is rebooting
about twice per day on average. I can't seem to pinpoint why? I was
hoping someone could Identify something I'm overlooking. I will gladly
provide more information as needed.
I'm running a two-node nfs cluster on Centos 5 with the following packages:
rgmanager-2.0.31-1.el5.centos
gfs2-utils-0.1.38-1.el5
gfs-utils-0.1.12-1.el5
system-config-lvm-1.0.22-1.0.el5
cman-2.0.73-1.el5_1.1
lvm2-cluster-2.02.26-1.el5
kmod-gfs-0.1.16-6.2.6.18_8.1.15.el5
lvm2-2.02.26-3.el5
kmod-gfs-0.1.19-7.el5_1.1
kernel-2.6.18-53.1.6.el5
Thanks!
Randy
Feb 12 00:01:16 nfs1-cluster openais[2605]: [TOTEM] The token was lost in the OPERATIONAL state.
Feb 12 00:01:16 nfs1-cluster openais[2605]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes).
Feb 12 00:01:16 nfs1-cluster openais[2605]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes).
Feb 12 00:01:16 nfs1-cluster openais[2605]: [TOTEM] entering GATHER state from 2.
Feb 12 00:01:16 nfs1-cluster openais[2605]: [TOTEM] Creating commit token because I am the rep.
Feb 12 00:01:16 nfs1-cluster openais[2605]: [TOTEM] Saving state aru 1e high seq received 1e
Feb 12 00:01:16 nfs1-cluster openais[2605]: [TOTEM] Storing new sequence id for ring 1054
Feb 12 00:01:16 nfs1-cluster openais[2605]: [TOTEM] entering COMMIT state.
Feb 12 00:01:19 nfs1-cluster openais[2605]: [TOTEM] entering RECOVERY state.
Feb 12 00:01:19 nfs1-cluster openais[2605]: [TOTEM] position [0] member 140.90.91.240:
Feb 12 00:01:19 nfs1-cluster openais[2605]: [TOTEM] previous ring seq 4176 rep 140.90.91.240
Feb 12 00:01:19 nfs1-cluster openais[2605]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 00:01:19 nfs1-cluster openais[2605]: [TOTEM] position [1] member 140.90.91.242:
Feb 12 00:01:19 nfs1-cluster openais[2605]: [TOTEM] previous ring seq 4176 rep 140.90.91.240
Feb 12 00:01:19 nfs1-cluster openais[2605]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 00:01:19 nfs1-cluster openais[2605]: [TOTEM] Did not need to originate any messages in recovery.
Feb 12 00:01:19 nfs1-cluster openais[2605]: [TOTEM] Sending initial ORF token
Feb 12 00:01:19 nfs1-cluster openais[2605]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 00:01:19 nfs1-cluster openais[2605]: [CLM ] New Configuration:
Feb 12 00:01:19 nfs1-cluster openais[2605]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 00:01:19 nfs1-cluster openais[2605]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 00:01:19 nfs1-cluster openais[2605]: [CLM ] Members Left:
Feb 12 00:01:19 nfs1-cluster openais[2605]: [CLM ] Members Joined:
Feb 12 00:01:19 nfs1-cluster openais[2605]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 00:01:19 nfs1-cluster openais[2605]: [CLM ] New Configuration:
Feb 12 00:01:19 nfs1-cluster openais[2605]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 00:01:19 nfs1-cluster openais[2605]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 00:01:19 nfs1-cluster openais[2605]: [CLM ] Members Left:
Feb 12 00:01:19 nfs1-cluster openais[2605]: [CLM ] Members Joined:
Feb 12 00:01:19 nfs1-cluster openais[2605]: [SYNC ] This node is within the primary component and will provide service.
Feb 12 00:01:19 nfs1-cluster openais[2605]: [TOTEM] entering OPERATIONAL state.
Feb 12 00:01:19 nfs1-cluster openais[2605]: [CLM ] got nodejoin message 140.90.91.240
Feb 12 00:01:19 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2
Feb 12 00:01:19 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2 3
Feb 12 00:01:19 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2 3 4
Feb 12 00:01:19 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2 3 4
Feb 12 00:01:19 nfs1-cluster openais[2605]: [CLM ] got nodejoin message 140.90.91.242
Feb 12 00:01:19 nfs1-cluster openais[2605]: [CPG ] got joinlist message from node 1
Feb 12 00:01:19 nfs1-cluster openais[2605]: [CPG ] got joinlist message from node 2
Feb 12 00:28:23 nfs1-cluster openais[2605]: [TOTEM] The token was lost in the OPERATIONAL state.
Feb 12 00:28:23 nfs1-cluster openais[2605]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes).
Feb 12 00:28:23 nfs1-cluster openais[2605]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes).
Feb 12 00:28:23 nfs1-cluster openais[2605]: [TOTEM] entering GATHER state from 2.
Feb 12 00:28:23 nfs1-cluster openais[2605]: [TOTEM] Creating commit token because I am the rep.
Feb 12 00:28:23 nfs1-cluster openais[2605]: [TOTEM] Saving state aru 1e high seq received 1e
Feb 12 00:28:23 nfs1-cluster openais[2605]: [TOTEM] Storing new sequence id for ring 1058
Feb 12 00:28:23 nfs1-cluster openais[2605]: [TOTEM] entering COMMIT state.
Feb 12 00:28:26 nfs1-cluster openais[2605]: [TOTEM] entering RECOVERY state.
Feb 12 00:28:26 nfs1-cluster openais[2605]: [TOTEM] position [0] member 140.90.91.240:
Feb 12 00:28:26 nfs1-cluster openais[2605]: [TOTEM] previous ring seq 4180 rep 140.90.91.240
Feb 12 00:28:26 nfs1-cluster openais[2605]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 00:28:26 nfs1-cluster openais[2605]: [TOTEM] position [1] member 140.90.91.242:
Feb 12 00:28:26 nfs1-cluster openais[2605]: [TOTEM] previous ring seq 4180 rep 140.90.91.240
Feb 12 00:28:26 nfs1-cluster openais[2605]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 00:28:26 nfs1-cluster openais[2605]: [TOTEM] Did not need to originate any messages in recovery.
Feb 12 00:28:26 nfs1-cluster openais[2605]: [TOTEM] Sending initial ORF token
Feb 12 00:28:26 nfs1-cluster openais[2605]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 00:28:26 nfs1-cluster openais[2605]: [CLM ] New Configuration:
Feb 12 00:28:26 nfs1-cluster openais[2605]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 00:28:26 nfs1-cluster openais[2605]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 00:28:26 nfs1-cluster openais[2605]: [CLM ] Members Left:
Feb 12 00:28:26 nfs1-cluster openais[2605]: [CLM ] Members Joined:
Feb 12 00:28:26 nfs1-cluster openais[2605]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 00:28:26 nfs1-cluster openais[2605]: [CLM ] New Configuration:
Feb 12 00:28:26 nfs1-cluster openais[2605]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 00:28:26 nfs1-cluster openais[2605]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 00:28:27 nfs1-cluster openais[2605]: [CLM ] Members Left:
Feb 12 00:28:27 nfs1-cluster openais[2605]: [CLM ] Members Joined:
Feb 12 00:28:27 nfs1-cluster openais[2605]: [SYNC ] This node is within the primary component and will provide service.
Feb 12 00:28:27 nfs1-cluster openais[2605]: [TOTEM] entering OPERATIONAL state.
Feb 12 00:28:27 nfs1-cluster openais[2605]: [CLM ] got nodejoin message 140.90.91.240
Feb 12 00:28:27 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2
Feb 12 00:28:27 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2 3
Feb 12 00:28:27 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2 3 4
Feb 12 00:28:27 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2 3 4
Feb 12 00:28:27 nfs1-cluster openais[2605]: [CLM ] got nodejoin message 140.90.91.242
Feb 12 00:28:27 nfs1-cluster openais[2605]: [CPG ] got joinlist message from node 1
Feb 12 00:28:27 nfs1-cluster openais[2605]: [CPG ] got joinlist message from node 2
Feb 12 00:44:09 nfs1-cluster openais[2605]: [TOTEM] The token was lost in the OPERATIONAL state.
Feb 12 00:44:09 nfs1-cluster openais[2605]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes).
Feb 12 00:44:09 nfs1-cluster openais[2605]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes).
Feb 12 00:44:09 nfs1-cluster openais[2605]: [TOTEM] entering GATHER state from 2.
Feb 12 00:44:09 nfs1-cluster openais[2605]: [TOTEM] Creating commit token because I am the rep.
Feb 12 00:44:09 nfs1-cluster openais[2605]: [TOTEM] Saving state aru 1e high seq received 1e
Feb 12 00:44:09 nfs1-cluster openais[2605]: [TOTEM] Storing new sequence id for ring 105c
Feb 12 00:44:09 nfs1-cluster openais[2605]: [TOTEM] entering COMMIT state.
Feb 12 00:44:10 nfs1-cluster openais[2605]: [TOTEM] entering RECOVERY state.
Feb 12 00:44:10 nfs1-cluster openais[2605]: [TOTEM] position [0] member 140.90.91.240:
Feb 12 00:44:10 nfs1-cluster openais[2605]: [TOTEM] previous ring seq 4184 rep 140.90.91.240
Feb 12 00:44:10 nfs1-cluster openais[2605]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 00:44:10 nfs1-cluster openais[2605]: [TOTEM] position [1] member 140.90.91.242:
Feb 12 00:44:10 nfs1-cluster openais[2605]: [TOTEM] previous ring seq 4184 rep 140.90.91.240
Feb 12 00:44:10 nfs1-cluster openais[2605]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 00:44:10 nfs1-cluster openais[2605]: [TOTEM] Did not need to originate any messages in recovery.
Feb 12 00:44:10 nfs1-cluster openais[2605]: [TOTEM] Sending initial ORF token
Feb 12 00:44:10 nfs1-cluster openais[2605]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 00:44:10 nfs1-cluster openais[2605]: [CLM ] New Configuration:
Feb 12 00:44:10 nfs1-cluster openais[2605]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 00:44:10 nfs1-cluster openais[2605]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 00:44:10 nfs1-cluster openais[2605]: [CLM ] Members Left:
Feb 12 00:44:10 nfs1-cluster openais[2605]: [CLM ] Members Joined:
Feb 12 00:44:10 nfs1-cluster openais[2605]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 00:44:10 nfs1-cluster openais[2605]: [CLM ] New Configuration:
Feb 12 00:44:10 nfs1-cluster openais[2605]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 00:44:10 nfs1-cluster openais[2605]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 00:44:10 nfs1-cluster openais[2605]: [CLM ] Members Left:
Feb 12 00:44:10 nfs1-cluster openais[2605]: [CLM ] Members Joined:
Feb 12 00:44:10 nfs1-cluster openais[2605]: [SYNC ] This node is within the primary component and will provide service.
Feb 12 00:44:10 nfs1-cluster openais[2605]: [TOTEM] entering OPERATIONAL state.
Feb 12 00:44:10 nfs1-cluster openais[2605]: [CLM ] got nodejoin message 140.90.91.240
Feb 12 00:44:10 nfs1-cluster openais[2605]: [CLM ] got nodejoin message 140.90.91.242
Feb 12 00:44:10 nfs1-cluster openais[2605]: [CPG ] got joinlist message from node 1
Feb 12 00:44:10 nfs1-cluster openais[2605]: [CPG ] got joinlist message from node 2
Feb 12 00:50:27 nfs1-cluster openais[2605]: [TOTEM] The token was lost in the OPERATIONAL state.
Feb 12 00:50:27 nfs1-cluster openais[2605]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes).
Feb 12 00:50:27 nfs1-cluster openais[2605]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes).
Feb 12 00:50:27 nfs1-cluster openais[2605]: [TOTEM] entering GATHER state from 2.
Feb 12 00:50:27 nfs1-cluster openais[2605]: [TOTEM] Creating commit token because I am the rep.
Feb 12 00:50:27 nfs1-cluster openais[2605]: [TOTEM] Saving state aru 1e high seq received 1e
Feb 12 00:50:27 nfs1-cluster openais[2605]: [TOTEM] Storing new sequence id for ring 1060
Feb 12 00:50:27 nfs1-cluster openais[2605]: [TOTEM] entering COMMIT state.
Feb 12 00:50:29 nfs1-cluster openais[2605]: [TOTEM] entering RECOVERY state.
Feb 12 00:50:29 nfs1-cluster openais[2605]: [TOTEM] position [0] member 140.90.91.240:
Feb 12 00:50:29 nfs1-cluster openais[2605]: [TOTEM] previous ring seq 4188 rep 140.90.91.240
Feb 12 00:50:29 nfs1-cluster openais[2605]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 00:50:29 nfs1-cluster openais[2605]: [TOTEM] position [1] member 140.90.91.242:
Feb 12 00:50:29 nfs1-cluster openais[2605]: [TOTEM] previous ring seq 4188 rep 140.90.91.240
Feb 12 00:50:29 nfs1-cluster openais[2605]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 00:50:29 nfs1-cluster openais[2605]: [TOTEM] Did not need to originate any messages in recovery.
Feb 12 00:50:29 nfs1-cluster openais[2605]: [TOTEM] Sending initial ORF token
Feb 12 00:50:29 nfs1-cluster openais[2605]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 00:50:29 nfs1-cluster openais[2605]: [CLM ] New Configuration:
Feb 12 00:50:29 nfs1-cluster openais[2605]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 00:50:29 nfs1-cluster openais[2605]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 00:50:29 nfs1-cluster openais[2605]: [CLM ] Members Left:
Feb 12 00:50:29 nfs1-cluster openais[2605]: [CLM ] Members Joined:
Feb 12 00:50:29 nfs1-cluster openais[2605]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 00:50:29 nfs1-cluster openais[2605]: [CLM ] New Configuration:
Feb 12 00:50:29 nfs1-cluster openais[2605]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 00:50:29 nfs1-cluster openais[2605]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 00:50:29 nfs1-cluster openais[2605]: [CLM ] Members Left:
Feb 12 00:50:29 nfs1-cluster openais[2605]: [CLM ] Members Joined:
Feb 12 00:50:29 nfs1-cluster openais[2605]: [SYNC ] This node is within the primary component and will provide service.
Feb 12 00:50:29 nfs1-cluster openais[2605]: [TOTEM] entering OPERATIONAL state.
Feb 12 00:50:29 nfs1-cluster openais[2605]: [CLM ] got nodejoin message 140.90.91.240
Feb 12 00:50:29 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2
Feb 12 00:50:29 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2 3
Feb 12 00:50:29 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2 3 4
Feb 12 00:50:29 nfs1-cluster openais[2605]: [CLM ] got nodejoin message 140.90.91.242
Feb 12 00:50:29 nfs1-cluster openais[2605]: [CPG ] got joinlist message from node 1
Feb 12 00:50:29 nfs1-cluster openais[2605]: [CPG ] got joinlist message from node 2
Feb 12 00:53:31 nfs1-cluster openais[2605]: [TOTEM] The token was lost in the OPERATIONAL state.
Feb 12 00:53:31 nfs1-cluster openais[2605]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes).
Feb 12 00:53:31 nfs1-cluster openais[2605]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes).
Feb 12 00:53:31 nfs1-cluster openais[2605]: [TOTEM] entering GATHER state from 2.
Feb 12 00:53:31 nfs1-cluster openais[2605]: [TOTEM] Creating commit token because I am the rep.
Feb 12 00:53:31 nfs1-cluster openais[2605]: [TOTEM] Saving state aru 1e high seq received 1e
Feb 12 00:53:31 nfs1-cluster openais[2605]: [TOTEM] Storing new sequence id for ring 1064
Feb 12 00:53:31 nfs1-cluster openais[2605]: [TOTEM] entering COMMIT state.
Feb 12 00:53:32 nfs1-cluster openais[2605]: [TOTEM] entering RECOVERY state.
Feb 12 00:53:32 nfs1-cluster openais[2605]: [TOTEM] position [0] member 140.90.91.240:
Feb 12 00:53:32 nfs1-cluster openais[2605]: [TOTEM] previous ring seq 4192 rep 140.90.91.240
Feb 12 00:53:32 nfs1-cluster openais[2605]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 00:53:32 nfs1-cluster openais[2605]: [TOTEM] position [1] member 140.90.91.242:
Feb 12 00:53:32 nfs1-cluster openais[2605]: [TOTEM] previous ring seq 4192 rep 140.90.91.240
Feb 12 00:53:32 nfs1-cluster openais[2605]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 00:53:32 nfs1-cluster openais[2605]: [TOTEM] Did not need to originate any messages in recovery.
Feb 12 00:53:32 nfs1-cluster openais[2605]: [TOTEM] Sending initial ORF token
Feb 12 00:53:32 nfs1-cluster openais[2605]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 00:53:32 nfs1-cluster openais[2605]: [CLM ] New Configuration:
Feb 12 00:53:32 nfs1-cluster openais[2605]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 00:53:32 nfs1-cluster openais[2605]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 00:53:32 nfs1-cluster openais[2605]: [CLM ] Members Left:
Feb 12 00:53:32 nfs1-cluster openais[2605]: [CLM ] Members Joined:
Feb 12 00:53:32 nfs1-cluster openais[2605]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 00:53:32 nfs1-cluster openais[2605]: [CLM ] New Configuration:
Feb 12 00:53:32 nfs1-cluster openais[2605]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 00:53:32 nfs1-cluster openais[2605]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 00:53:32 nfs1-cluster openais[2605]: [CLM ] Members Left:
Feb 12 00:53:32 nfs1-cluster openais[2605]: [CLM ] Members Joined:
Feb 12 00:53:32 nfs1-cluster openais[2605]: [SYNC ] This node is within the primary component and will provide service.
Feb 12 00:53:32 nfs1-cluster openais[2605]: [TOTEM] entering OPERATIONAL state.
Feb 12 00:53:32 nfs1-cluster openais[2605]: [CLM ] got nodejoin message 140.90.91.240
Feb 12 00:53:32 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2
Feb 12 00:53:32 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2 3
Feb 12 00:53:32 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2 3 4
Feb 12 00:53:32 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2 3 4
Feb 12 00:53:32 nfs1-cluster openais[2605]: [CLM ] got nodejoin message 140.90.91.242
Feb 12 00:53:32 nfs1-cluster openais[2605]: [CPG ] got joinlist message from node 1
Feb 12 00:53:32 nfs1-cluster openais[2605]: [CPG ] got joinlist message from node 2
Feb 12 01:06:55 nfs1-cluster openais[2605]: [TOTEM] entering GATHER state from 12.
Feb 12 01:06:55 nfs1-cluster openais[2605]: [TOTEM] Creating commit token because I am the rep.
Feb 12 01:06:55 nfs1-cluster openais[2605]: [TOTEM] Saving state aru 1e high seq received 1e
Feb 12 01:06:55 nfs1-cluster openais[2605]: [TOTEM] Storing new sequence id for ring 1068
Feb 12 01:06:55 nfs1-cluster openais[2605]: [TOTEM] entering COMMIT state.
Feb 12 01:06:57 nfs1-cluster openais[2605]: [TOTEM] entering RECOVERY state.
Feb 12 01:06:57 nfs1-cluster openais[2605]: [TOTEM] position [0] member 140.90.91.240:
Feb 12 01:06:57 nfs1-cluster openais[2605]: [TOTEM] previous ring seq 4196 rep 140.90.91.240
Feb 12 01:06:57 nfs1-cluster openais[2605]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 01:06:57 nfs1-cluster openais[2605]: [TOTEM] position [1] member 140.90.91.242:
Feb 12 01:06:57 nfs1-cluster openais[2605]: [TOTEM] previous ring seq 4196 rep 140.90.91.240
Feb 12 01:06:57 nfs1-cluster openais[2605]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 01:06:57 nfs1-cluster openais[2605]: [TOTEM] Did not need to originate any messages in recovery.
Feb 12 01:06:57 nfs1-cluster openais[2605]: [TOTEM] Sending initial ORF token
Feb 12 01:06:57 nfs1-cluster openais[2605]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 01:06:57 nfs1-cluster openais[2605]: [CLM ] New Configuration:
Feb 12 01:06:57 nfs1-cluster openais[2605]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 01:06:57 nfs1-cluster openais[2605]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 01:06:57 nfs1-cluster openais[2605]: [CLM ] Members Left:
Feb 12 01:06:57 nfs1-cluster openais[2605]: [CLM ] Members Joined:
Feb 12 01:06:57 nfs1-cluster openais[2605]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 01:06:57 nfs1-cluster openais[2605]: [CLM ] New Configuration:
Feb 12 01:06:57 nfs1-cluster openais[2605]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 01:06:57 nfs1-cluster openais[2605]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 01:06:57 nfs1-cluster openais[2605]: [CLM ] Members Left:
Feb 12 01:06:57 nfs1-cluster openais[2605]: [CLM ] Members Joined:
Feb 12 01:06:57 nfs1-cluster openais[2605]: [SYNC ] This node is within the primary component and will provide service.
Feb 12 01:06:57 nfs1-cluster openais[2605]: [TOTEM] entering OPERATIONAL state.
Feb 12 01:06:57 nfs1-cluster openais[2605]: [CLM ] got nodejoin message 140.90.91.240
Feb 12 01:06:57 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2
Feb 12 01:06:57 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2 3
Feb 12 01:06:57 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2 3 4
Feb 12 01:06:57 nfs1-cluster openais[2605]: [CLM ] got nodejoin message 140.90.91.242
Feb 12 01:06:57 nfs1-cluster openais[2605]: [CPG ] got joinlist message from node 1
Feb 12 01:06:57 nfs1-cluster openais[2605]: [CPG ] got joinlist message from node 2
Feb 12 01:46:13 nfs1-cluster openais[2605]: [TOTEM] The token was lost in the OPERATIONAL state.
Feb 12 01:46:14 nfs1-cluster openais[2605]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes).
Feb 12 01:46:14 nfs1-cluster openais[2605]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes).
Feb 12 01:46:14 nfs1-cluster openais[2605]: [TOTEM] entering GATHER state from 2.
Feb 12 01:46:14 nfs1-cluster openais[2605]: [TOTEM] Creating commit token because I am the rep.
Feb 12 01:46:14 nfs1-cluster openais[2605]: [TOTEM] Saving state aru 1e high seq received 1e
Feb 12 01:46:14 nfs1-cluster openais[2605]: [TOTEM] Storing new sequence id for ring 106c
Feb 12 01:46:14 nfs1-cluster openais[2605]: [TOTEM] entering COMMIT state.
Feb 12 01:46:14 nfs1-cluster openais[2605]: [TOTEM] entering RECOVERY state.
Feb 12 01:46:14 nfs1-cluster openais[2605]: [TOTEM] position [0] member 140.90.91.240:
Feb 12 01:46:14 nfs1-cluster openais[2605]: [TOTEM] previous ring seq 4200 rep 140.90.91.240
Feb 12 01:46:14 nfs1-cluster openais[2605]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 01:46:14 nfs1-cluster openais[2605]: [TOTEM] position [1] member 140.90.91.242:
Feb 12 01:46:14 nfs1-cluster openais[2605]: [TOTEM] previous ring seq 4200 rep 140.90.91.240
Feb 12 01:46:14 nfs1-cluster openais[2605]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 01:46:14 nfs1-cluster openais[2605]: [TOTEM] Did not need to originate any messages in recovery.
Feb 12 01:46:14 nfs1-cluster openais[2605]: [TOTEM] Sending initial ORF token
Feb 12 01:46:14 nfs1-cluster openais[2605]: [TOTEM] Sending initial ORF token
Feb 12 01:46:14 nfs1-cluster openais[2605]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 01:46:14 nfs1-cluster openais[2605]: [CLM ] New Configuration:
Feb 12 01:46:14 nfs1-cluster openais[2605]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 01:46:14 nfs1-cluster openais[2605]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 01:46:14 nfs1-cluster openais[2605]: [CLM ] Members Left:
Feb 12 01:46:14 nfs1-cluster openais[2605]: [CLM ] Members Joined:
Feb 12 01:46:14 nfs1-cluster openais[2605]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 01:46:14 nfs1-cluster openais[2605]: [CLM ] New Configuration:
Feb 12 01:46:14 nfs1-cluster openais[2605]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 01:46:14 nfs1-cluster openais[2605]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 01:46:14 nfs1-cluster openais[2605]: [CLM ] Members Left:
Feb 12 01:46:14 nfs1-cluster openais[2605]: [CLM ] Members Joined:
Feb 12 01:46:14 nfs1-cluster openais[2605]: [SYNC ] This node is within the primary component and will provide service.
Feb 12 01:46:14 nfs1-cluster openais[2605]: [TOTEM] entering OPERATIONAL state.
Feb 12 01:46:14 nfs1-cluster openais[2605]: [CLM ] got nodejoin message 140.90.91.240
Feb 12 01:46:14 nfs1-cluster openais[2605]: [CLM ] got nodejoin message 140.90.91.242
Feb 12 01:46:14 nfs1-cluster openais[2605]: [CPG ] got joinlist message from node 1
Feb 12 01:46:14 nfs1-cluster openais[2605]: [CPG ] got joinlist message from node 2
Feb 12 02:25:14 nfs1-cluster openais[2605]: [TOTEM] The token was lost in the OPERATIONAL state.
Feb 12 02:25:14 nfs1-cluster openais[2605]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes).
Feb 12 02:25:14 nfs1-cluster openais[2605]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes).
Feb 12 02:25:14 nfs1-cluster openais[2605]: [TOTEM] entering GATHER state from 2.
Feb 12 02:25:14 nfs1-cluster openais[2605]: [TOTEM] Creating commit token because I am the rep.
Feb 12 02:25:14 nfs1-cluster openais[2605]: [TOTEM] Saving state aru 1e high seq received 1e
Feb 12 02:25:14 nfs1-cluster openais[2605]: [TOTEM] Storing new sequence id for ring 1070
Feb 12 02:25:14 nfs1-cluster openais[2605]: [TOTEM] entering COMMIT state.
Feb 12 02:25:15 nfs1-cluster openais[2605]: [TOTEM] entering RECOVERY state.
Feb 12 02:25:15 nfs1-cluster openais[2605]: [TOTEM] position [0] member 140.90.91.240:
Feb 12 02:25:15 nfs1-cluster openais[2605]: [TOTEM] previous ring seq 4204 rep 140.90.91.240
Feb 12 02:25:15 nfs1-cluster openais[2605]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 02:25:15 nfs1-cluster openais[2605]: [TOTEM] position [1] member 140.90.91.242:
Feb 12 02:25:15 nfs1-cluster openais[2605]: [TOTEM] previous ring seq 4204 rep 140.90.91.240
Feb 12 02:25:15 nfs1-cluster openais[2605]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 02:25:15 nfs1-cluster openais[2605]: [TOTEM] Did not need to originate any messages in recovery.
Feb 12 02:25:15 nfs1-cluster openais[2605]: [TOTEM] Sending initial ORF token
Feb 12 02:25:15 nfs1-cluster openais[2605]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 02:25:15 nfs1-cluster openais[2605]: [CLM ] New Configuration:
Feb 12 02:25:15 nfs1-cluster openais[2605]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 02:25:15 nfs1-cluster openais[2605]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 02:25:15 nfs1-cluster openais[2605]: [CLM ] Members Left:
Feb 12 02:25:15 nfs1-cluster openais[2605]: [CLM ] Members Joined:
Feb 12 02:25:15 nfs1-cluster openais[2605]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 02:25:15 nfs1-cluster openais[2605]: [CLM ] New Configuration:
Feb 12 02:25:15 nfs1-cluster openais[2605]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 02:25:15 nfs1-cluster openais[2605]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 02:25:15 nfs1-cluster openais[2605]: [CLM ] Members Left:
Feb 12 02:25:15 nfs1-cluster openais[2605]: [CLM ] Members Joined:
Feb 12 02:25:15 nfs1-cluster openais[2605]: [SYNC ] This node is within the primary component and will provide service.
Feb 12 02:25:15 nfs1-cluster openais[2605]: [TOTEM] entering OPERATIONAL state.
Feb 12 02:25:15 nfs1-cluster openais[2605]: [CLM ] got nodejoin message 140.90.91.240
Feb 12 02:25:15 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2
Feb 12 02:25:15 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2 3
Feb 12 02:25:15 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2 3 4
Feb 12 02:25:15 nfs1-cluster openais[2605]: [CLM ] got nodejoin message 140.90.91.242
Feb 12 02:25:15 nfs1-cluster openais[2605]: [CPG ] got joinlist message from node 1
Feb 12 02:25:15 nfs1-cluster openais[2605]: [CPG ] got joinlist message from node 2
Feb 12 02:36:52 nfs1-cluster openais[2605]: [TOTEM] The token was lost in the OPERATIONAL state.
Feb 12 02:36:52 nfs1-cluster openais[2605]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes).
Feb 12 02:36:52 nfs1-cluster openais[2605]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes).
Feb 12 02:36:52 nfs1-cluster openais[2605]: [TOTEM] entering GATHER state from 2.
Feb 12 02:36:52 nfs1-cluster openais[2605]: [TOTEM] Creating commit token because I am the rep.
Feb 12 02:36:52 nfs1-cluster openais[2605]: [TOTEM] Saving state aru 1e high seq received 1e
Feb 12 02:36:52 nfs1-cluster openais[2605]: [TOTEM] Storing new sequence id for ring 1074
Feb 12 02:36:52 nfs1-cluster openais[2605]: [TOTEM] entering COMMIT state.
Feb 12 02:36:54 nfs1-cluster openais[2605]: [TOTEM] entering RECOVERY state.
Feb 12 02:36:54 nfs1-cluster openais[2605]: [TOTEM] position [0] member 140.90.91.240:
Feb 12 02:36:54 nfs1-cluster openais[2605]: [TOTEM] previous ring seq 4208 rep 140.90.91.240
Feb 12 02:36:54 nfs1-cluster openais[2605]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 02:36:54 nfs1-cluster openais[2605]: [TOTEM] position [1] member 140.90.91.242:
Feb 12 02:36:54 nfs1-cluster openais[2605]: [TOTEM] previous ring seq 4208 rep 140.90.91.240
Feb 12 02:36:54 nfs1-cluster openais[2605]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 02:36:54 nfs1-cluster openais[2605]: [TOTEM] Did not need to originate any messages in recovery.
Feb 12 02:36:54 nfs1-cluster openais[2605]: [TOTEM] Sending initial ORF token
Feb 12 02:36:54 nfs1-cluster openais[2605]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 02:36:54 nfs1-cluster openais[2605]: [CLM ] New Configuration:
Feb 12 02:36:54 nfs1-cluster openais[2605]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 02:36:54 nfs1-cluster openais[2605]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 02:36:54 nfs1-cluster openais[2605]: [CLM ] Members Left:
Feb 12 02:36:54 nfs1-cluster openais[2605]: [CLM ] Members Joined:
Feb 12 02:36:54 nfs1-cluster openais[2605]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 02:36:54 nfs1-cluster openais[2605]: [CLM ] New Configuration:
Feb 12 02:36:54 nfs1-cluster openais[2605]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 02:36:54 nfs1-cluster openais[2605]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 02:36:54 nfs1-cluster openais[2605]: [CLM ] Members Left:
Feb 12 02:36:54 nfs1-cluster openais[2605]: [CLM ] Members Joined:
Feb 12 02:36:54 nfs1-cluster openais[2605]: [SYNC ] This node is within the primary component and will provide service.
Feb 12 02:36:54 nfs1-cluster openais[2605]: [TOTEM] entering OPERATIONAL state.
Feb 12 02:36:54 nfs1-cluster openais[2605]: [CLM ] got nodejoin message 140.90.91.240
Feb 12 02:36:54 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2
Feb 12 02:36:54 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2 3
Feb 12 02:36:54 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2 3 4
Feb 12 02:36:54 nfs1-cluster openais[2605]: [CLM ] got nodejoin message 140.90.91.242
Feb 12 02:36:54 nfs1-cluster openais[2605]: [CPG ] got joinlist message from node 1
Feb 12 02:36:54 nfs1-cluster openais[2605]: [CPG ] got joinlist message from node 2
Feb 12 02:41:27 nfs1-cluster openais[2605]: [TOTEM] The token was lost in the OPERATIONAL state.
Feb 12 02:41:27 nfs1-cluster openais[2605]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes).
Feb 12 02:41:27 nfs1-cluster openais[2605]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes).
Feb 12 02:41:27 nfs1-cluster openais[2605]: [TOTEM] entering GATHER state from 2.
Feb 12 02:41:27 nfs1-cluster openais[2605]: [TOTEM] Creating commit token because I am the rep.
Feb 12 02:41:27 nfs1-cluster openais[2605]: [TOTEM] Saving state aru 1e high seq received 1e
Feb 12 02:41:27 nfs1-cluster openais[2605]: [TOTEM] Storing new sequence id for ring 1078
Feb 12 02:41:27 nfs1-cluster openais[2605]: [TOTEM] entering COMMIT state.
Feb 12 02:41:27 nfs1-cluster openais[2605]: [TOTEM] entering RECOVERY state.
Feb 12 02:41:27 nfs1-cluster openais[2605]: [TOTEM] position [0] member 140.90.91.240:
Feb 12 02:41:27 nfs1-cluster openais[2605]: [TOTEM] previous ring seq 4212 rep 140.90.91.240
Feb 12 02:41:27 nfs1-cluster openais[2605]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 02:41:27 nfs1-cluster openais[2605]: [TOTEM] position [1] member 140.90.91.242:
Feb 12 02:41:27 nfs1-cluster openais[2605]: [TOTEM] previous ring seq 4212 rep 140.90.91.240
Feb 12 02:41:27 nfs1-cluster openais[2605]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 02:41:27 nfs1-cluster openais[2605]: [TOTEM] Did not need to originate any messages in recovery.
Feb 12 02:41:27 nfs1-cluster openais[2605]: [TOTEM] Sending initial ORF token
Feb 12 02:41:27 nfs1-cluster openais[2605]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 02:41:27 nfs1-cluster openais[2605]: [CLM ] New Configuration:
Feb 12 02:41:27 nfs1-cluster openais[2605]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 02:41:27 nfs1-cluster openais[2605]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 02:41:27 nfs1-cluster openais[2605]: [CLM ] Members Left:
Feb 12 02:41:27 nfs1-cluster openais[2605]: [CLM ] Members Joined:
Feb 12 02:41:27 nfs1-cluster openais[2605]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 02:41:27 nfs1-cluster openais[2605]: [CLM ] New Configuration:
Feb 12 02:41:27 nfs1-cluster openais[2605]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 02:41:27 nfs1-cluster openais[2605]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 02:41:27 nfs1-cluster openais[2605]: [CLM ] Members Left:
Feb 12 02:41:27 nfs1-cluster openais[2605]: [CLM ] Members Joined:
Feb 12 02:41:27 nfs1-cluster openais[2605]: [SYNC ] This node is within the primary component and will provide service.
Feb 12 02:41:27 nfs1-cluster openais[2605]: [TOTEM] entering OPERATIONAL state.
Feb 12 02:41:27 nfs1-cluster openais[2605]: [CLM ] got nodejoin message 140.90.91.240
Feb 12 02:41:27 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2
Feb 12 02:41:27 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2 3
Feb 12 02:41:27 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2 3 4
Feb 12 02:41:27 nfs1-cluster openais[2605]: [CLM ] got nodejoin message 140.90.91.242
Feb 12 02:41:27 nfs1-cluster openais[2605]: [CPG ] got joinlist message from node 1
Feb 12 02:41:27 nfs1-cluster openais[2605]: [CPG ] got joinlist message from node 2
Feb 12 02:44:56 nfs1-cluster openais[2605]: [TOTEM] entering GATHER state from 12.
Feb 12 02:44:56 nfs1-cluster openais[2605]: [TOTEM] Creating commit token because I am the rep.
Feb 12 02:44:56 nfs1-cluster openais[2605]: [TOTEM] Saving state aru 1e high seq received 1e
Feb 12 02:44:56 nfs1-cluster openais[2605]: [TOTEM] Storing new sequence id for ring 107c
Feb 12 02:44:56 nfs1-cluster openais[2605]: [TOTEM] entering COMMIT state.
Feb 12 02:45:01 nfs1-cluster openais[2605]: [TOTEM] entering RECOVERY state.
Feb 12 02:45:01 nfs1-cluster openais[2605]: [TOTEM] position [0] member 140.90.91.240:
Feb 12 02:45:01 nfs1-cluster openais[2605]: [TOTEM] previous ring seq 4216 rep 140.90.91.240
Feb 12 02:45:01 nfs1-cluster openais[2605]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 02:45:01 nfs1-cluster openais[2605]: [TOTEM] position [1] member 140.90.91.242:
Feb 12 02:45:01 nfs1-cluster openais[2605]: [TOTEM] previous ring seq 4216 rep 140.90.91.240
Feb 12 02:45:01 nfs1-cluster openais[2605]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 02:45:01 nfs1-cluster openais[2605]: [TOTEM] Did not need to originate any messages in recovery.
Feb 12 02:45:01 nfs1-cluster openais[2605]: [TOTEM] Sending initial ORF token
Feb 12 02:45:01 nfs1-cluster openais[2605]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 02:45:01 nfs1-cluster openais[2605]: [CLM ] New Configuration:
Feb 12 02:45:01 nfs1-cluster openais[2605]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 02:45:01 nfs1-cluster openais[2605]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 02:45:01 nfs1-cluster openais[2605]: [CLM ] Members Left:
Feb 12 02:45:01 nfs1-cluster openais[2605]: [CLM ] Members Joined:
Feb 12 02:45:01 nfs1-cluster openais[2605]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 02:45:01 nfs1-cluster openais[2605]: [CLM ] New Configuration:
Feb 12 02:45:01 nfs1-cluster openais[2605]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 02:45:01 nfs1-cluster openais[2605]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 02:45:01 nfs1-cluster openais[2605]: [CLM ] Members Left:
Feb 12 02:45:01 nfs1-cluster openais[2605]: [CLM ] Members Joined:
Feb 12 02:45:01 nfs1-cluster openais[2605]: [SYNC ] This node is within the primary component and will provide service.
Feb 12 02:45:01 nfs1-cluster openais[2605]: [TOTEM] entering OPERATIONAL state.
Feb 12 02:45:01 nfs1-cluster openais[2605]: [CLM ] got nodejoin message 140.90.91.240
Feb 12 02:45:01 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2
Feb 12 02:45:01 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2 3
Feb 12 02:45:01 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2 3 4
Feb 12 02:45:01 nfs1-cluster openais[2605]: [CLM ] got nodejoin message 140.90.91.242
Feb 12 02:45:01 nfs1-cluster openais[2605]: [CPG ] got joinlist message from node 1
Feb 12 02:45:01 nfs1-cluster openais[2605]: [CPG ] got joinlist message from node 2
Feb 12 03:12:47 nfs1-cluster openais[2605]: [TOTEM] entering GATHER state from 12.
Feb 12 03:12:47 nfs1-cluster openais[2605]: [TOTEM] Creating commit token because I am the rep.
Feb 12 03:12:47 nfs1-cluster openais[2605]: [TOTEM] Saving state aru 1e high seq received 1e
Feb 12 03:12:47 nfs1-cluster openais[2605]: [TOTEM] Storing new sequence id for ring 1080
Feb 12 03:12:47 nfs1-cluster openais[2605]: [TOTEM] entering COMMIT state.
Feb 12 03:12:57 nfs1-cluster openais[2605]: [TOTEM] The token was lost in the COMMIT state.
Feb 12 03:12:57 nfs1-cluster openais[2605]: [TOTEM] entering GATHER state from 4.
Feb 12 03:12:57 nfs1-cluster openais[2605]: [TOTEM] Creating commit token because I am the rep.
Feb 12 03:12:57 nfs1-cluster openais[2605]: [TOTEM] Storing new sequence id for ring 1084
Feb 12 03:12:57 nfs1-cluster openais[2605]: [TOTEM] entering COMMIT state.
Feb 12 03:12:57 nfs1-cluster openais[2605]: [TOTEM] entering RECOVERY state.
Feb 12 03:12:57 nfs1-cluster openais[2605]: [TOTEM] position [0] member 140.90.91.240:
Feb 12 03:12:57 nfs1-cluster openais[2605]: [TOTEM] previous ring seq 4220 rep 140.90.91.240
Feb 12 03:12:57 nfs1-cluster openais[2605]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 03:12:57 nfs1-cluster openais[2605]: [TOTEM] position [1] member 140.90.91.242:
Feb 12 03:12:57 nfs1-cluster openais[2605]: [TOTEM] previous ring seq 4224 rep 140.90.91.242
Feb 12 03:12:57 nfs1-cluster openais[2605]: [TOTEM] aru 32 high delivered 32 received flag 1
Feb 12 03:12:57 nfs1-cluster openais[2605]: [TOTEM] Did not need to originate any messages in recovery.
Feb 12 03:12:57 nfs1-cluster openais[2605]: [TOTEM] Sending initial ORF token
Feb 12 03:12:57 nfs1-cluster openais[2605]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 03:12:57 nfs1-cluster openais[2605]: [CLM ] New Configuration:
Feb 12 03:12:57 nfs1-cluster openais[2605]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 03:12:57 nfs1-cluster openais[2605]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 03:12:57 nfs1-cluster openais[2605]: [CLM ] Members Left:
Feb 12 03:12:57 nfs1-cluster openais[2605]: [CLM ] Members Joined:
Feb 12 03:12:57 nfs1-cluster openais[2605]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 03:12:57 nfs1-cluster openais[2605]: [CLM ] New Configuration:
Feb 12 03:12:57 nfs1-cluster openais[2605]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 03:12:57 nfs1-cluster openais[2605]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 03:12:57 nfs1-cluster openais[2605]: [CLM ] Members Left:
Feb 12 03:12:57 nfs1-cluster openais[2605]: [CLM ] Members Joined:
Feb 12 03:12:57 nfs1-cluster openais[2605]: [SYNC ] This node is within the primary component and will provide service.
Feb 12 03:12:57 nfs1-cluster openais[2605]: [TOTEM] entering OPERATIONAL state.
Feb 12 03:12:57 nfs1-cluster openais[2605]: [CMAN ] cman killed by node 2 because we rejoined the cluster without a full restart
Feb 12 03:16:20 nfs1-cluster openais[2605]: [MAIN ] AIS Executive Service RELEASE 'subrev 1358 version 0.80.3'
Feb 12 03:16:20 nfs1-cluster openais[2605]: [MAIN ] Copyright (C) 2002-2006 MontaVista Software, Inc and contributors.
Feb 12 03:16:20 nfs1-cluster openais[2605]: [MAIN ] Copyright (C) 2006 Red Hat, Inc.
Feb 12 03:16:20 nfs1-cluster openais[2605]: [MAIN ] AIS Executive Service: started and ready to provide service.
Feb 12 03:16:20 nfs1-cluster openais[2605]: [MAIN ] Using default multicast address of 239.192.83.8
Feb 12 03:16:20 nfs1-cluster openais[2605]: [MAIN ] openais component openais_cpg loaded.
Feb 12 03:16:20 nfs1-cluster openais[2605]: [MAIN ] Registering service handler 'openais cluster closed process group service v1.01'
Feb 12 03:16:20 nfs1-cluster openais[2605]: [MAIN ] openais component openais_cfg loaded.
Feb 12 03:16:20 nfs1-cluster openais[2605]: [MAIN ] Registering service handler 'openais configuration service'
Feb 12 03:16:20 nfs1-cluster openais[2605]: [MAIN ] openais component openais_msg loaded.
Feb 12 03:16:20 nfs1-cluster openais[2605]: [MAIN ] Registering service handler 'openais message service B.01.01'
Feb 12 03:16:20 nfs1-cluster openais[2605]: [MAIN ] openais component openais_lck loaded.
Feb 12 03:16:20 nfs1-cluster openais[2605]: [MAIN ] Registering service handler 'openais distributed locking service B.01.01'
Feb 12 03:16:20 nfs1-cluster openais[2605]: [MAIN ] openais component openais_evt loaded.
Feb 12 03:16:20 nfs1-cluster openais[2605]: [MAIN ] Registering service handler 'openais event service B.01.01'
Feb 12 03:16:20 nfs1-cluster openais[2605]: [MAIN ] openais component openais_ckpt loaded.
Feb 12 03:16:20 nfs1-cluster openais[2605]: [MAIN ] Registering service handler 'openais checkpoint service B.01.01'
Feb 12 03:16:20 nfs1-cluster openais[2605]: [MAIN ] openais component openais_amf loaded.
Feb 12 03:16:20 nfs1-cluster openais[2605]: [MAIN ] Registering service handler 'openais availability management framework B.01.01'
Feb 12 03:16:20 nfs1-cluster openais[2605]: [MAIN ] openais component openais_clm loaded.
Feb 12 03:16:20 nfs1-cluster openais[2605]: [MAIN ] Registering service handler 'openais cluster membership service B.01.01'
Feb 12 03:16:20 nfs1-cluster openais[2605]: [MAIN ] openais component openais_evs loaded.
Feb 12 03:16:20 nfs1-cluster openais[2605]: [MAIN ] Registering service handler 'openais extended virtual synchrony service'
Feb 12 03:16:20 nfs1-cluster openais[2605]: [MAIN ] openais component openais_cman loaded.
Feb 12 03:16:20 nfs1-cluster openais[2605]: [MAIN ] Registering service handler 'openais CMAN membership service 2.01'
Feb 12 03:16:20 nfs1-cluster openais[2605]: [TOTEM] Token Timeout (10000 ms) retransmit timeout (495 ms)
Feb 12 03:16:20 nfs1-cluster openais[2605]: [TOTEM] token hold (386 ms) retransmits before loss (20 retrans)
Feb 12 03:16:20 nfs1-cluster openais[2605]: [TOTEM] join (60 ms) send_join (0 ms) consensus (4800 ms) merge (200 ms)
Feb 12 03:16:20 nfs1-cluster openais[2605]: [TOTEM] downcheck (1000 ms) fail to recv const (50 msgs)
Feb 12 03:16:20 nfs1-cluster openais[2605]: [TOTEM] seqno unchanged const (30 rotations) Maximum network MTU 1500
Feb 12 03:16:20 nfs1-cluster openais[2605]: [TOTEM] window size per rotation (50 messages) maximum messages per rotation (17 messages)
Feb 12 03:16:20 nfs1-cluster openais[2605]: [TOTEM] send threads (0 threads)
Feb 12 03:16:20 nfs1-cluster openais[2605]: [TOTEM] RRP token expired timeout (495 ms)
Feb 12 03:16:20 nfs1-cluster openais[2605]: [TOTEM] RRP token problem counter (2000 ms)
Feb 12 03:16:20 nfs1-cluster openais[2605]: [TOTEM] RRP threshold (10 problem count)
Feb 12 03:16:20 nfs1-cluster openais[2605]: [TOTEM] RRP mode set to none.
Feb 12 03:16:20 nfs1-cluster openais[2605]: [TOTEM] heartbeat_failures_allowed (0)
Feb 12 03:16:20 nfs1-cluster openais[2605]: [TOTEM] max_network_delay (50 ms)
Feb 12 03:16:20 nfs1-cluster openais[2605]: [TOTEM] HeartBeat is Disabled. To enable set heartbeat_failures_allowed > 0
Feb 12 03:16:20 nfs1-cluster openais[2605]: [TOTEM] Receive multicast socket recv buffer size (262142 bytes).
Feb 12 03:16:20 nfs1-cluster openais[2605]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes).
Feb 12 03:16:20 nfs1-cluster openais[2605]: [TOTEM] The network interface [140.90.91.240] is now up.
Feb 12 03:16:20 nfs1-cluster openais[2605]: [TOTEM] Created or loaded sequence id 4228.140.90.91.240 for this ring.
Feb 12 03:16:20 nfs1-cluster openais[2605]: [TOTEM] entering GATHER state from 15.
Feb 12 03:16:20 nfs1-cluster openais[2605]: [SERV ] Initialising service handler 'openais extended virtual synchrony service'
Feb 12 03:16:20 nfs1-cluster openais[2605]: [SERV ] Initialising service handler 'openais cluster membership service B.01.01'
Feb 12 03:16:20 nfs1-cluster openais[2605]: [SERV ] Initialising service handler 'openais availability management framework B.01.01'
Feb 12 03:16:20 nfs1-cluster openais[2605]: [SERV ] Initialising service handler 'openais checkpoint service B.01.01'
Feb 12 03:16:20 nfs1-cluster openais[2605]: [SERV ] Initialising service handler 'openais event service B.01.01'
Feb 12 03:16:20 nfs1-cluster openais[2605]: [SERV ] Initialising service handler 'openais distributed locking service B.01.01'
Feb 12 03:16:20 nfs1-cluster openais[2605]: [SERV ] Initialising service handler 'openais message service B.01.01'
Feb 12 03:16:20 nfs1-cluster openais[2605]: [SERV ] Initialising service handler 'openais configuration service'
Feb 12 03:16:20 nfs1-cluster openais[2605]: [SERV ] Initialising service handler 'openais cluster closed process group service v1.01'
Feb 12 03:16:20 nfs1-cluster openais[2605]: [SERV ] Initialising service handler 'openais CMAN membership service 2.01'
Feb 12 03:16:20 nfs1-cluster openais[2605]: [CMAN ] CMAN 2.0.73 (built Nov 29 2007 18:41:45) started
Feb 12 03:16:20 nfs1-cluster openais[2605]: [SYNC ] Not using a virtual synchrony filter.
Feb 12 03:16:20 nfs1-cluster openais[2605]: [TOTEM] Creating commit token because I am the rep.
Feb 12 03:16:20 nfs1-cluster openais[2605]: [TOTEM] Saving state aru 0 high seq received 0
Feb 12 03:16:20 nfs1-cluster openais[2605]: [TOTEM] Storing new sequence id for ring 1088
Feb 12 03:16:20 nfs1-cluster openais[2605]: [TOTEM] entering COMMIT state.
Feb 12 03:16:20 nfs1-cluster openais[2605]: [TOTEM] entering RECOVERY state.
Feb 12 03:16:20 nfs1-cluster openais[2605]: [TOTEM] position [0] member 140.90.91.240:
Feb 12 03:16:20 nfs1-cluster openais[2605]: [TOTEM] previous ring seq 4228 rep 140.90.91.240
Feb 12 03:16:20 nfs1-cluster openais[2605]: [TOTEM] aru 0 high delivered 0 received flag 1
Feb 12 03:16:20 nfs1-cluster openais[2605]: [TOTEM] Did not need to originate any messages in recovery.
Feb 12 03:16:20 nfs1-cluster openais[2605]: [TOTEM] Sending initial ORF token
Feb 12 03:16:20 nfs1-cluster openais[2605]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 03:16:20 nfs1-cluster openais[2605]: [CLM ] New Configuration:
Feb 12 03:16:20 nfs1-cluster openais[2605]: [CLM ] Members Left:
Feb 12 03:16:20 nfs1-cluster openais[2605]: [CLM ] Members Joined:
Feb 12 03:16:20 nfs1-cluster openais[2605]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 03:16:20 nfs1-cluster openais[2605]: [CLM ] New Configuration:
Feb 12 03:16:20 nfs1-cluster openais[2605]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 03:16:20 nfs1-cluster openais[2605]: [CLM ] Members Left:
Feb 12 03:16:20 nfs1-cluster openais[2605]: [CLM ] Members Joined:
Feb 12 03:16:20 nfs1-cluster openais[2605]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 03:16:20 nfs1-cluster openais[2605]: [SYNC ] This node is within the primary component and will provide service.
Feb 12 03:16:20 nfs1-cluster openais[2605]: [TOTEM] entering OPERATIONAL state.
Feb 12 03:16:20 nfs1-cluster openais[2605]: [CMAN ] quorum regained, resuming activity
Feb 12 03:16:20 nfs1-cluster openais[2605]: [TOTEM] entering GATHER state from 11.
Feb 12 03:16:21 nfs1-cluster openais[2605]: [TOTEM] Creating commit token because I am the rep.
Feb 12 03:16:21 nfs1-cluster openais[2605]: [TOTEM] Saving state aru 3 high seq received 3
Feb 12 03:16:21 nfs1-cluster openais[2605]: [TOTEM] Storing new sequence id for ring 108c
Feb 12 03:16:21 nfs1-cluster openais[2605]: [TOTEM] entering COMMIT state.
Feb 12 03:16:21 nfs1-cluster openais[2605]: [TOTEM] entering RECOVERY state.
Feb 12 03:16:21 nfs1-cluster openais[2605]: [TOTEM] position [0] member 140.90.91.240:
Feb 12 03:16:21 nfs1-cluster openais[2605]: [TOTEM] previous ring seq 4232 rep 140.90.91.240
Feb 12 03:16:21 nfs1-cluster openais[2605]: [TOTEM] aru 3 high delivered 1 received flag 1
Feb 12 03:16:21 nfs1-cluster openais[2605]: [TOTEM] position [1] member 140.90.91.242:
Feb 12 03:16:21 nfs1-cluster openais[2605]: [TOTEM] previous ring seq 4232 rep 140.90.91.242
Feb 12 03:16:21 nfs1-cluster openais[2605]: [TOTEM] aru 3c high delivered 3c received flag 1
Feb 12 03:16:21 nfs1-cluster openais[2605]: [TOTEM] Did not need to originate any messages in recovery.
Feb 12 03:16:21 nfs1-cluster openais[2605]: [TOTEM] Sending initial ORF token
Feb 12 03:16:21 nfs1-cluster openais[2605]: [CLM ] got nodejoin message 140.90.91.240
Feb 12 03:16:21 nfs1-cluster openais[2605]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 03:16:21 nfs1-cluster openais[2605]: [CLM ] New Configuration:
Feb 12 03:16:21 nfs1-cluster openais[2605]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 03:16:21 nfs1-cluster openais[2605]: [CLM ] Members Left:
Feb 12 03:16:21 nfs1-cluster openais[2605]: [CLM ] Members Joined:
Feb 12 03:16:21 nfs1-cluster openais[2605]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 03:16:21 nfs1-cluster openais[2605]: [CLM ] New Configuration:
Feb 12 03:16:21 nfs1-cluster openais[2605]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 03:16:21 nfs1-cluster openais[2605]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 03:16:21 nfs1-cluster openais[2605]: [CLM ] Members Left:
Feb 12 03:16:21 nfs1-cluster openais[2605]: [CLM ] Members Joined:
Feb 12 03:16:21 nfs1-cluster openais[2605]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 03:16:21 nfs1-cluster openais[2605]: [SYNC ] This node is within the primary component and will provide service.
Feb 12 03:16:21 nfs1-cluster openais[2605]: [TOTEM] entering OPERATIONAL state.
Feb 12 03:16:21 nfs1-cluster openais[2605]: [CLM ] got nodejoin message 140.90.91.240
Feb 12 03:16:21 nfs1-cluster openais[2605]: [CLM ] got nodejoin message 140.90.91.242
Feb 12 03:16:21 nfs1-cluster openais[2605]: [CPG ] got joinlist message from node 2
Feb 12 03:33:14 nfs1-cluster openais[2605]: [TOTEM] entering GATHER state from 12.
Feb 12 03:33:14 nfs1-cluster openais[2605]: [TOTEM] Creating commit token because I am the rep.
Feb 12 03:33:14 nfs1-cluster openais[2605]: [TOTEM] Saving state aru fd high seq received fd
Feb 12 03:33:14 nfs1-cluster openais[2605]: [TOTEM] Storing new sequence id for ring 1090
Feb 12 03:33:14 nfs1-cluster openais[2605]: [TOTEM] entering COMMIT state.
Feb 12 03:33:17 nfs1-cluster openais[2605]: [TOTEM] entering RECOVERY state.
Feb 12 03:33:17 nfs1-cluster openais[2605]: [TOTEM] position [0] member 140.90.91.240:
Feb 12 03:33:17 nfs1-cluster openais[2605]: [TOTEM] previous ring seq 4236 rep 140.90.91.240
Feb 12 03:33:17 nfs1-cluster openais[2605]: [TOTEM] aru fd high delivered fd received flag 1
Feb 12 03:33:17 nfs1-cluster openais[2605]: [TOTEM] position [1] member 140.90.91.242:
Feb 12 03:33:17 nfs1-cluster openais[2605]: [TOTEM] previous ring seq 4236 rep 140.90.91.240
Feb 12 03:33:17 nfs1-cluster openais[2605]: [TOTEM] aru fd high delivered fd received flag 1
Feb 12 03:33:17 nfs1-cluster openais[2605]: [TOTEM] Did not need to originate any messages in recovery.
Feb 12 03:33:17 nfs1-cluster openais[2605]: [TOTEM] Sending initial ORF token
Feb 12 03:33:17 nfs1-cluster openais[2605]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 03:33:17 nfs1-cluster openais[2605]: [CLM ] New Configuration:
Feb 12 03:33:17 nfs1-cluster openais[2605]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 03:33:17 nfs1-cluster openais[2605]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 03:33:17 nfs1-cluster openais[2605]: [CLM ] Members Left:
Feb 12 03:33:17 nfs1-cluster openais[2605]: [CLM ] Members Joined:
Feb 12 03:33:17 nfs1-cluster openais[2605]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 03:33:17 nfs1-cluster openais[2605]: [CLM ] New Configuration:
Feb 12 03:33:17 nfs1-cluster openais[2605]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 03:33:17 nfs1-cluster openais[2605]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 03:33:17 nfs1-cluster openais[2605]: [CLM ] Members Left:
Feb 12 03:33:17 nfs1-cluster openais[2605]: [CLM ] Members Joined:
Feb 12 03:33:17 nfs1-cluster openais[2605]: [SYNC ] This node is within the primary component and will provide service.
Feb 12 03:33:17 nfs1-cluster openais[2605]: [TOTEM] entering OPERATIONAL state.
Feb 12 03:33:17 nfs1-cluster openais[2605]: [CLM ] got nodejoin message 140.90.91.240
Feb 12 03:33:17 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2
Feb 12 03:33:17 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2 3
Feb 12 03:33:17 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2 3 4
Feb 12 03:33:17 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2 3 4
Feb 12 03:33:17 nfs1-cluster openais[2605]: [CLM ] got nodejoin message 140.90.91.242
Feb 12 03:33:17 nfs1-cluster openais[2605]: [CPG ] got joinlist message from node 1
Feb 12 03:33:17 nfs1-cluster openais[2605]: [CPG ] got joinlist message from node 2
Feb 12 03:35:15 nfs1-cluster openais[2605]: [TOTEM] The token was lost in the OPERATIONAL state.
Feb 12 03:35:15 nfs1-cluster openais[2605]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes).
Feb 12 03:35:15 nfs1-cluster openais[2605]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes).
Feb 12 03:35:15 nfs1-cluster openais[2605]: [TOTEM] entering GATHER state from 2.
Feb 12 03:35:15 nfs1-cluster openais[2605]: [TOTEM] Creating commit token because I am the rep.
Feb 12 03:35:15 nfs1-cluster openais[2605]: [TOTEM] Saving state aru 1e high seq received 1e
Feb 12 03:35:15 nfs1-cluster openais[2605]: [TOTEM] Storing new sequence id for ring 1094
Feb 12 03:35:15 nfs1-cluster openais[2605]: [TOTEM] entering COMMIT state.
Feb 12 03:35:18 nfs1-cluster openais[2605]: [TOTEM] entering RECOVERY state.
Feb 12 03:35:18 nfs1-cluster openais[2605]: [TOTEM] position [0] member 140.90.91.240:
Feb 12 03:35:18 nfs1-cluster openais[2605]: [TOTEM] previous ring seq 4240 rep 140.90.91.240
Feb 12 03:35:18 nfs1-cluster openais[2605]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 03:35:18 nfs1-cluster openais[2605]: [TOTEM] position [1] member 140.90.91.242:
Feb 12 03:35:18 nfs1-cluster openais[2605]: [TOTEM] previous ring seq 4240 rep 140.90.91.240
Feb 12 03:35:18 nfs1-cluster openais[2605]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 03:35:18 nfs1-cluster openais[2605]: [TOTEM] Did not need to originate any messages in recovery.
Feb 12 03:35:18 nfs1-cluster openais[2605]: [TOTEM] Sending initial ORF token
Feb 12 03:35:18 nfs1-cluster openais[2605]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 03:35:18 nfs1-cluster openais[2605]: [CLM ] New Configuration:
Feb 12 03:35:18 nfs1-cluster openais[2605]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 03:35:18 nfs1-cluster openais[2605]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 03:35:18 nfs1-cluster openais[2605]: [CLM ] Members Left:
Feb 12 03:35:18 nfs1-cluster openais[2605]: [CLM ] Members Joined:
Feb 12 03:35:18 nfs1-cluster openais[2605]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 03:35:18 nfs1-cluster openais[2605]: [CLM ] New Configuration:
Feb 12 03:35:18 nfs1-cluster openais[2605]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 03:35:18 nfs1-cluster openais[2605]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 03:35:18 nfs1-cluster openais[2605]: [CLM ] Members Left:
Feb 12 03:35:18 nfs1-cluster openais[2605]: [CLM ] Members Joined:
Feb 12 03:35:18 nfs1-cluster openais[2605]: [SYNC ] This node is within the primary component and will provide service.
Feb 12 03:35:18 nfs1-cluster openais[2605]: [TOTEM] entering OPERATIONAL state.
Feb 12 03:35:18 nfs1-cluster openais[2605]: [CLM ] got nodejoin message 140.90.91.240
Feb 12 03:35:18 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2
Feb 12 03:35:18 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2 3
Feb 12 03:35:18 nfs1-cluster openais[2605]: [TOTEM] Retransmit List: 2 3 4
Feb 12 03:35:18 nfs1-cluster openais[2605]: [CLM ] got nodejoin message 140.90.91.242
Feb 12 03:35:18 nfs1-cluster openais[2605]: [CPG ] got joinlist message from node 1
Feb 12 03:35:18 nfs1-cluster openais[2605]: [CPG ] got joinlist message from node 2
Feb 12 03:36:14 nfs1-cluster openais[2605]: [TOTEM] entering GATHER state from 12.
Feb 12 03:36:14 nfs1-cluster openais[2605]: [TOTEM] Creating commit token because I am the rep.
Feb 12 03:36:14 nfs1-cluster openais[2605]: [TOTEM] Saving state aru 1e high seq received 1e
Feb 12 03:36:14 nfs1-cluster openais[2605]: [TOTEM] Storing new sequence id for ring 1098
Feb 12 03:36:14 nfs1-cluster openais[2605]: [TOTEM] entering COMMIT state.
Feb 12 03:36:24 nfs1-cluster openais[2605]: [TOTEM] The token was lost in the COMMIT state.
Feb 12 03:36:24 nfs1-cluster openais[2605]: [TOTEM] entering GATHER state from 4.
Feb 12 03:36:24 nfs1-cluster openais[2605]: [TOTEM] Creating commit token because I am the rep.
Feb 12 03:36:24 nfs1-cluster openais[2605]: [TOTEM] Storing new sequence id for ring 109c
Feb 12 03:36:24 nfs1-cluster openais[2605]: [TOTEM] entering COMMIT state.
Feb 12 03:36:24 nfs1-cluster openais[2605]: [TOTEM] entering RECOVERY state.
Feb 12 03:36:24 nfs1-cluster openais[2605]: [TOTEM] position [0] member 140.90.91.240:
Feb 12 03:36:24 nfs1-cluster openais[2605]: [TOTEM] previous ring seq 4244 rep 140.90.91.240
Feb 12 03:36:24 nfs1-cluster openais[2605]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 03:36:24 nfs1-cluster openais[2605]: [TOTEM] position [1] member 140.90.91.242:
Feb 12 03:36:24 nfs1-cluster openais[2605]: [TOTEM] previous ring seq 4248 rep 140.90.91.242
Feb 12 03:36:24 nfs1-cluster openais[2605]: [TOTEM] aru 32 high delivered 32 received flag 1
Feb 12 03:36:24 nfs1-cluster openais[2605]: [TOTEM] Did not need to originate any messages in recovery.
Feb 12 03:36:24 nfs1-cluster openais[2605]: [TOTEM] Sending initial ORF token
Feb 12 03:36:24 nfs1-cluster openais[2605]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 03:36:24 nfs1-cluster openais[2605]: [CLM ] New Configuration:
Feb 12 03:36:24 nfs1-cluster openais[2605]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 03:36:24 nfs1-cluster openais[2605]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 03:36:24 nfs1-cluster openais[2605]: [CLM ] Members Left:
Feb 12 03:36:24 nfs1-cluster openais[2605]: [CLM ] Members Joined:
Feb 12 03:36:24 nfs1-cluster openais[2605]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 03:36:24 nfs1-cluster openais[2605]: [CLM ] New Configuration:
Feb 12 03:36:24 nfs1-cluster openais[2605]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 03:36:24 nfs1-cluster openais[2605]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 03:36:24 nfs1-cluster openais[2605]: [CLM ] Members Left:
Feb 12 03:36:24 nfs1-cluster openais[2605]: [CLM ] Members Joined:
Feb 12 03:36:24 nfs1-cluster openais[2605]: [SYNC ] This node is within the primary component and will provide service.
Feb 12 03:36:24 nfs1-cluster openais[2605]: [TOTEM] entering OPERATIONAL state.
Feb 12 03:36:24 nfs1-cluster openais[2605]: [CMAN ] cman killed by node 2 because we rejoined the cluster without a full restart
Feb 12 03:39:48 nfs1-cluster openais[2604]: [MAIN ] AIS Executive Service RELEASE 'subrev 1358 version 0.80.3'
Feb 12 03:39:48 nfs1-cluster openais[2604]: [MAIN ] Copyright (C) 2002-2006 MontaVista Software, Inc and contributors.
Feb 12 03:39:48 nfs1-cluster openais[2604]: [MAIN ] Copyright (C) 2006 Red Hat, Inc.
Feb 12 03:39:48 nfs1-cluster openais[2604]: [MAIN ] AIS Executive Service: started and ready to provide service.
Feb 12 03:39:48 nfs1-cluster openais[2604]: [MAIN ] Using default multicast address of 239.192.83.8
Feb 12 03:39:48 nfs1-cluster openais[2604]: [MAIN ] openais component openais_cpg loaded.
Feb 12 03:39:48 nfs1-cluster openais[2604]: [MAIN ] Registering service handler 'openais cluster closed process group service v1.01'
Feb 12 03:39:48 nfs1-cluster openais[2604]: [MAIN ] openais component openais_cfg loaded.
Feb 12 03:39:48 nfs1-cluster openais[2604]: [MAIN ] Registering service handler 'openais configuration service'
Feb 12 03:39:48 nfs1-cluster openais[2604]: [MAIN ] openais component openais_msg loaded.
Feb 12 03:39:48 nfs1-cluster openais[2604]: [MAIN ] Registering service handler 'openais message service B.01.01'
Feb 12 03:39:48 nfs1-cluster openais[2604]: [MAIN ] openais component openais_lck loaded.
Feb 12 03:39:48 nfs1-cluster openais[2604]: [MAIN ] Registering service handler 'openais distributed locking service B.01.01'
Feb 12 03:39:48 nfs1-cluster openais[2604]: [MAIN ] openais component openais_evt loaded.
Feb 12 03:39:48 nfs1-cluster openais[2604]: [MAIN ] Registering service handler 'openais event service B.01.01'
Feb 12 03:39:48 nfs1-cluster openais[2604]: [MAIN ] openais component openais_ckpt loaded.
Feb 12 03:39:48 nfs1-cluster openais[2604]: [MAIN ] Registering service handler 'openais checkpoint service B.01.01'
Feb 12 03:39:48 nfs1-cluster openais[2604]: [MAIN ] openais component openais_amf loaded.
Feb 12 03:39:48 nfs1-cluster openais[2604]: [MAIN ] Registering service handler 'openais availability management framework B.01.01'
Feb 12 03:39:48 nfs1-cluster openais[2604]: [MAIN ] openais component openais_clm loaded.
Feb 12 03:39:48 nfs1-cluster openais[2604]: [MAIN ] Registering service handler 'openais cluster membership service B.01.01'
Feb 12 03:39:48 nfs1-cluster openais[2604]: [MAIN ] openais component openais_evs loaded.
Feb 12 03:39:48 nfs1-cluster openais[2604]: [MAIN ] Registering service handler 'openais extended virtual synchrony service'
Feb 12 03:39:48 nfs1-cluster openais[2604]: [MAIN ] openais component openais_cman loaded.
Feb 12 03:39:48 nfs1-cluster openais[2604]: [MAIN ] Registering service handler 'openais CMAN membership service 2.01'
Feb 12 03:39:48 nfs1-cluster openais[2604]: [TOTEM] Token Timeout (10000 ms) retransmit timeout (495 ms)
Feb 12 03:39:48 nfs1-cluster openais[2604]: [TOTEM] token hold (386 ms) retransmits before loss (20 retrans)
Feb 12 03:39:48 nfs1-cluster openais[2604]: [TOTEM] join (60 ms) send_join (0 ms) consensus (4800 ms) merge (200 ms)
Feb 12 03:39:48 nfs1-cluster openais[2604]: [TOTEM] downcheck (1000 ms) fail to recv const (50 msgs)
Feb 12 03:39:48 nfs1-cluster openais[2604]: [TOTEM] seqno unchanged const (30 rotations) Maximum network MTU 1500
Feb 12 03:39:48 nfs1-cluster openais[2604]: [TOTEM] window size per rotation (50 messages) maximum messages per rotation (17 messages)
Feb 12 03:39:48 nfs1-cluster openais[2604]: [TOTEM] send threads (0 threads)
Feb 12 03:39:48 nfs1-cluster openais[2604]: [TOTEM] RRP token expired timeout (495 ms)
Feb 12 03:39:48 nfs1-cluster openais[2604]: [TOTEM] RRP token problem counter (2000 ms)
Feb 12 03:39:48 nfs1-cluster openais[2604]: [TOTEM] RRP threshold (10 problem count)
Feb 12 03:39:48 nfs1-cluster openais[2604]: [TOTEM] RRP mode set to none.
Feb 12 03:39:48 nfs1-cluster openais[2604]: [TOTEM] heartbeat_failures_allowed (0)
Feb 12 03:39:48 nfs1-cluster openais[2604]: [TOTEM] max_network_delay (50 ms)
Feb 12 03:39:48 nfs1-cluster openais[2604]: [TOTEM] HeartBeat is Disabled. To enable set heartbeat_failures_allowed > 0
Feb 12 03:39:48 nfs1-cluster openais[2604]: [TOTEM] Receive multicast socket recv buffer size (262142 bytes).
Feb 12 03:39:48 nfs1-cluster openais[2604]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes).
Feb 12 03:39:48 nfs1-cluster openais[2604]: [TOTEM] The network interface [140.90.91.240] is now up.
Feb 12 03:39:48 nfs1-cluster openais[2604]: [TOTEM] Created or loaded sequence id 4252.140.90.91.240 for this ring.
Feb 12 03:39:48 nfs1-cluster openais[2604]: [TOTEM] entering GATHER state from 15.
Feb 12 03:39:48 nfs1-cluster openais[2604]: [SERV ] Initialising service handler 'openais extended virtual synchrony service'
Feb 12 03:39:48 nfs1-cluster openais[2604]: [SERV ] Initialising service handler 'openais cluster membership service B.01.01'
Feb 12 03:39:48 nfs1-cluster openais[2604]: [SERV ] Initialising service handler 'openais availability management framework B.01.01'
Feb 12 03:39:48 nfs1-cluster openais[2604]: [SERV ] Initialising service handler 'openais checkpoint service B.01.01'
Feb 12 03:39:48 nfs1-cluster openais[2604]: [SERV ] Initialising service handler 'openais event service B.01.01'
Feb 12 03:39:48 nfs1-cluster openais[2604]: [SERV ] Initialising service handler 'openais distributed locking service B.01.01'
Feb 12 03:39:48 nfs1-cluster openais[2604]: [SERV ] Initialising service handler 'openais message service B.01.01'
Feb 12 03:39:48 nfs1-cluster openais[2604]: [SERV ] Initialising service handler 'openais configuration service'
Feb 12 03:39:48 nfs1-cluster openais[2604]: [SERV ] Initialising service handler 'openais cluster closed process group service v1.01'
Feb 12 03:39:48 nfs1-cluster openais[2604]: [SERV ] Initialising service handler 'openais CMAN membership service 2.01'
Feb 12 03:39:48 nfs1-cluster openais[2604]: [CMAN ] CMAN 2.0.73 (built Nov 29 2007 18:41:45) started
Feb 12 03:39:48 nfs1-cluster openais[2604]: [SYNC ] Not using a virtual synchrony filter.
Feb 12 03:39:48 nfs1-cluster openais[2604]: [TOTEM] Creating commit token because I am the rep.
Feb 12 03:39:48 nfs1-cluster openais[2604]: [TOTEM] Saving state aru 0 high seq received 0
Feb 12 03:39:48 nfs1-cluster openais[2604]: [TOTEM] Storing new sequence id for ring 10a0
Feb 12 03:39:48 nfs1-cluster openais[2604]: [TOTEM] entering COMMIT state.
Feb 12 03:39:48 nfs1-cluster openais[2604]: [TOTEM] entering RECOVERY state.
Feb 12 03:39:48 nfs1-cluster openais[2604]: [TOTEM] position [0] member 140.90.91.240:
Feb 12 03:39:48 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4252 rep 140.90.91.240
Feb 12 03:39:48 nfs1-cluster openais[2604]: [TOTEM] aru 0 high delivered 0 received flag 1
Feb 12 03:39:48 nfs1-cluster openais[2604]: [TOTEM] Did not need to originate any messages in recovery.
Feb 12 03:39:48 nfs1-cluster openais[2604]: [TOTEM] Sending initial ORF token
Feb 12 03:39:48 nfs1-cluster openais[2604]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 03:39:48 nfs1-cluster openais[2604]: [CLM ] New Configuration:
Feb 12 03:39:48 nfs1-cluster openais[2604]: [CLM ] Members Left:
Feb 12 03:39:48 nfs1-cluster openais[2604]: [CLM ] Members Joined:
Feb 12 03:39:49 nfs1-cluster openais[2604]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 03:39:49 nfs1-cluster openais[2604]: [CLM ] New Configuration:
Feb 12 03:39:49 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 03:39:49 nfs1-cluster openais[2604]: [CLM ] Members Left:
Feb 12 03:39:49 nfs1-cluster openais[2604]: [CLM ] Members Joined:
Feb 12 03:39:49 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 03:39:49 nfs1-cluster openais[2604]: [SYNC ] This node is within the primary component and will provide service.
Feb 12 03:39:49 nfs1-cluster openais[2604]: [TOTEM] entering OPERATIONAL state.
Feb 12 03:39:49 nfs1-cluster openais[2604]: [CMAN ] quorum regained, resuming activity
Feb 12 03:39:49 nfs1-cluster openais[2604]: [TOTEM] entering GATHER state from 11.
Feb 12 03:39:49 nfs1-cluster openais[2604]: [TOTEM] Creating commit token because I am the rep.
Feb 12 03:39:49 nfs1-cluster openais[2604]: [TOTEM] Saving state aru 3 high seq received 3
Feb 12 03:39:49 nfs1-cluster openais[2604]: [TOTEM] Storing new sequence id for ring 10a4
Feb 12 03:39:49 nfs1-cluster openais[2604]: [TOTEM] entering COMMIT state.
Feb 12 03:39:49 nfs1-cluster openais[2604]: [TOTEM] entering RECOVERY state.
Feb 12 03:39:49 nfs1-cluster openais[2604]: [TOTEM] position [0] member 140.90.91.240:
Feb 12 03:39:49 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4256 rep 140.90.91.240
Feb 12 03:39:49 nfs1-cluster openais[2604]: [TOTEM] aru 3 high delivered 1 received flag 1
Feb 12 03:39:49 nfs1-cluster openais[2604]: [TOTEM] position [1] member 140.90.91.242:
Feb 12 03:39:49 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4256 rep 140.90.91.242
Feb 12 03:39:49 nfs1-cluster openais[2604]: [TOTEM] aru 3d high delivered 3d received flag 1
Feb 12 03:39:49 nfs1-cluster openais[2604]: [TOTEM] Did not need to originate any messages in recovery.
Feb 12 03:39:49 nfs1-cluster openais[2604]: [TOTEM] Sending initial ORF token
Feb 12 03:39:49 nfs1-cluster openais[2604]: [CLM ] got nodejoin message 140.90.91.240
Feb 12 03:39:49 nfs1-cluster openais[2604]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 03:39:49 nfs1-cluster openais[2604]: [CLM ] New Configuration:
Feb 12 03:39:49 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 03:39:49 nfs1-cluster openais[2604]: [CLM ] Members Left:
Feb 12 03:39:49 nfs1-cluster openais[2604]: [CLM ] Members Joined:
Feb 12 03:39:49 nfs1-cluster openais[2604]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 03:39:49 nfs1-cluster openais[2604]: [CLM ] New Configuration:
Feb 12 03:39:49 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 03:39:49 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 03:39:49 nfs1-cluster openais[2604]: [CLM ] Members Left:
Feb 12 03:39:49 nfs1-cluster openais[2604]: [CLM ] Members Joined:
Feb 12 03:39:49 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 03:39:49 nfs1-cluster openais[2604]: [SYNC ] This node is within the primary component and will provide service.
Feb 12 03:39:49 nfs1-cluster openais[2604]: [TOTEM] entering OPERATIONAL state.
Feb 12 03:39:49 nfs1-cluster openais[2604]: [CLM ] got nodejoin message 140.90.91.240
Feb 12 03:39:49 nfs1-cluster openais[2604]: [CLM ] got nodejoin message 140.90.91.242
Feb 12 03:39:49 nfs1-cluster openais[2604]: [CPG ] got joinlist message from node 2
Feb 12 03:40:01 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: bd
Feb 12 03:40:44 nfs1-cluster openais[2604]: [TOTEM] The token was lost in the OPERATIONAL state.
Feb 12 03:40:44 nfs1-cluster openais[2604]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes).
Feb 12 03:40:44 nfs1-cluster openais[2604]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes).
Feb 12 03:40:44 nfs1-cluster openais[2604]: [TOTEM] entering GATHER state from 2.
Feb 12 03:40:44 nfs1-cluster openais[2604]: [TOTEM] Creating commit token because I am the rep.
Feb 12 03:40:44 nfs1-cluster openais[2604]: [TOTEM] Saving state aru fa high seq received fa
Feb 12 03:40:44 nfs1-cluster openais[2604]: [TOTEM] Storing new sequence id for ring 10a8
Feb 12 03:40:44 nfs1-cluster openais[2604]: [TOTEM] entering COMMIT state.
Feb 12 03:40:47 nfs1-cluster openais[2604]: [TOTEM] entering RECOVERY state.
Feb 12 03:40:47 nfs1-cluster openais[2604]: [TOTEM] position [0] member 140.90.91.240:
Feb 12 03:40:47 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4260 rep 140.90.91.240
Feb 12 03:40:47 nfs1-cluster openais[2604]: [TOTEM] aru fa high delivered fa received flag 1
Feb 12 03:40:47 nfs1-cluster openais[2604]: [TOTEM] position [1] member 140.90.91.242:
Feb 12 03:40:47 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4260 rep 140.90.91.240
Feb 12 03:40:47 nfs1-cluster openais[2604]: [TOTEM] aru fa high delivered fa received flag 1
Feb 12 03:40:47 nfs1-cluster openais[2604]: [TOTEM] Did not need to originate any messages in recovery.
Feb 12 03:40:47 nfs1-cluster openais[2604]: [TOTEM] Sending initial ORF token
Feb 12 03:40:47 nfs1-cluster openais[2604]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 03:40:47 nfs1-cluster openais[2604]: [CLM ] New Configuration:
Feb 12 03:40:47 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 03:40:47 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 03:40:47 nfs1-cluster openais[2604]: [CLM ] Members Left:
Feb 12 03:40:47 nfs1-cluster openais[2604]: [CLM ] Members Joined:
Feb 12 03:40:47 nfs1-cluster openais[2604]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 03:40:47 nfs1-cluster openais[2604]: [CLM ] New Configuration:
Feb 12 03:40:47 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 03:40:47 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 03:40:47 nfs1-cluster openais[2604]: [CLM ] Members Left:
Feb 12 03:40:47 nfs1-cluster openais[2604]: [CLM ] Members Joined:
Feb 12 03:40:47 nfs1-cluster openais[2604]: [SYNC ] This node is within the primary component and will provide service.
Feb 12 03:40:47 nfs1-cluster openais[2604]: [TOTEM] entering OPERATIONAL state.
Feb 12 03:40:47 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2
Feb 12 03:40:47 nfs1-cluster openais[2604]: [CLM ] got nodejoin message 140.90.91.240
Feb 12 03:40:47 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2
Feb 12 03:40:47 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 3
Feb 12 03:40:47 nfs1-cluster openais[2604]: [CLM ] got nodejoin message 140.90.91.242
Feb 12 03:40:47 nfs1-cluster openais[2604]: [CPG ] got joinlist message from node 1
Feb 12 03:40:47 nfs1-cluster openais[2604]: [CPG ] got joinlist message from node 2
Feb 12 03:51:37 nfs1-cluster openais[2604]: [TOTEM] The token was lost in the OPERATIONAL state.
Feb 12 03:51:37 nfs1-cluster openais[2604]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes).
Feb 12 03:51:37 nfs1-cluster openais[2604]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes).
Feb 12 03:51:37 nfs1-cluster openais[2604]: [TOTEM] entering GATHER state from 2.
Feb 12 03:51:37 nfs1-cluster openais[2604]: [TOTEM] Creating commit token because I am the rep.
Feb 12 03:51:37 nfs1-cluster openais[2604]: [TOTEM] Saving state aru 1e high seq received 1e
Feb 12 03:51:37 nfs1-cluster openais[2604]: [TOTEM] Storing new sequence id for ring 10ac
Feb 12 03:51:37 nfs1-cluster openais[2604]: [TOTEM] entering COMMIT state.
Feb 12 03:51:41 nfs1-cluster openais[2604]: [TOTEM] entering RECOVERY state.
Feb 12 03:51:41 nfs1-cluster openais[2604]: [TOTEM] position [0] member 140.90.91.240:
Feb 12 03:51:41 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4264 rep 140.90.91.240
Feb 12 03:51:41 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 03:51:41 nfs1-cluster openais[2604]: [TOTEM] position [1] member 140.90.91.242:
Feb 12 03:51:41 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4264 rep 140.90.91.240
Feb 12 03:51:41 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 03:51:41 nfs1-cluster openais[2604]: [TOTEM] Did not need to originate any messages in recovery.
Feb 12 03:51:41 nfs1-cluster openais[2604]: [TOTEM] Sending initial ORF token
Feb 12 03:51:41 nfs1-cluster openais[2604]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 03:51:41 nfs1-cluster openais[2604]: [CLM ] New Configuration:
Feb 12 03:51:41 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 03:51:41 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 03:51:41 nfs1-cluster openais[2604]: [CLM ] Members Left:
Feb 12 03:51:41 nfs1-cluster openais[2604]: [CLM ] Members Joined:
Feb 12 03:51:41 nfs1-cluster openais[2604]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 03:51:41 nfs1-cluster openais[2604]: [CLM ] New Configuration:
Feb 12 03:51:41 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 03:51:41 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 03:51:41 nfs1-cluster openais[2604]: [CLM ] Members Left:
Feb 12 03:51:41 nfs1-cluster openais[2604]: [CLM ] Members Joined:
Feb 12 03:51:41 nfs1-cluster openais[2604]: [SYNC ] This node is within the primary component and will provide service.
Feb 12 03:51:41 nfs1-cluster openais[2604]: [TOTEM] entering OPERATIONAL state.
Feb 12 03:51:41 nfs1-cluster openais[2604]: [CLM ] got nodejoin message 140.90.91.240
Feb 12 03:51:41 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2
Feb 12 03:51:41 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2 3
Feb 12 03:51:41 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2 3 4
Feb 12 03:51:41 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2 3 4
Feb 12 03:51:41 nfs1-cluster openais[2604]: [CLM ] got nodejoin message 140.90.91.242
Feb 12 03:51:41 nfs1-cluster openais[2604]: [CPG ] got joinlist message from node 1
Feb 12 03:51:41 nfs1-cluster openais[2604]: [CPG ] got joinlist message from node 2
Feb 12 03:53:41 nfs1-cluster openais[2604]: [TOTEM] The token was lost in the OPERATIONAL state.
Feb 12 03:53:41 nfs1-cluster openais[2604]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes).
Feb 12 03:53:41 nfs1-cluster openais[2604]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes).
Feb 12 03:53:41 nfs1-cluster openais[2604]: [TOTEM] entering GATHER state from 2.
Feb 12 03:53:41 nfs1-cluster openais[2604]: [TOTEM] Creating commit token because I am the rep.
Feb 12 03:53:41 nfs1-cluster openais[2604]: [TOTEM] Saving state aru 1e high seq received 1e
Feb 12 03:53:41 nfs1-cluster openais[2604]: [TOTEM] Storing new sequence id for ring 10b0
Feb 12 03:53:41 nfs1-cluster openais[2604]: [TOTEM] entering COMMIT state.
Feb 12 03:53:42 nfs1-cluster openais[2604]: [TOTEM] entering RECOVERY state.
Feb 12 03:53:42 nfs1-cluster openais[2604]: [TOTEM] position [0] member 140.90.91.240:
Feb 12 03:53:42 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4268 rep 140.90.91.240
Feb 12 03:53:42 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 03:53:42 nfs1-cluster openais[2604]: [TOTEM] position [1] member 140.90.91.242:
Feb 12 03:53:42 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4268 rep 140.90.91.240
Feb 12 03:53:42 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 03:53:42 nfs1-cluster openais[2604]: [TOTEM] Did not need to originate any messages in recovery.
Feb 12 03:53:42 nfs1-cluster openais[2604]: [TOTEM] Sending initial ORF token
Feb 12 03:53:42 nfs1-cluster openais[2604]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 03:53:42 nfs1-cluster openais[2604]: [CLM ] New Configuration:
Feb 12 03:53:42 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 03:53:42 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 03:53:42 nfs1-cluster openais[2604]: [CLM ] Members Left:
Feb 12 03:53:42 nfs1-cluster openais[2604]: [CLM ] Members Joined:
Feb 12 03:53:42 nfs1-cluster openais[2604]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 03:53:42 nfs1-cluster openais[2604]: [CLM ] New Configuration:
Feb 12 03:53:42 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 03:53:42 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 03:53:42 nfs1-cluster openais[2604]: [CLM ] Members Left:
Feb 12 03:53:42 nfs1-cluster openais[2604]: [CLM ] Members Joined:
Feb 12 03:53:42 nfs1-cluster openais[2604]: [SYNC ] This node is within the primary component and will provide service.
Feb 12 03:53:42 nfs1-cluster openais[2604]: [TOTEM] entering OPERATIONAL state.
Feb 12 03:53:42 nfs1-cluster openais[2604]: [CLM ] got nodejoin message 140.90.91.240
Feb 12 03:53:42 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2
Feb 12 03:53:42 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2 3
Feb 12 03:53:42 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2 3 4
Feb 12 03:53:42 nfs1-cluster openais[2604]: [CLM ] got nodejoin message 140.90.91.242
Feb 12 03:53:42 nfs1-cluster openais[2604]: [CPG ] got joinlist message from node 1
Feb 12 03:53:42 nfs1-cluster openais[2604]: [CPG ] got joinlist message from node 2
Feb 12 04:10:54 nfs1-cluster openais[2604]: [TOTEM] The token was lost in the OPERATIONAL state.
Feb 12 04:10:54 nfs1-cluster openais[2604]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes).
Feb 12 04:10:54 nfs1-cluster openais[2604]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes).
Feb 12 04:10:54 nfs1-cluster openais[2604]: [TOTEM] entering GATHER state from 2.
Feb 12 04:10:54 nfs1-cluster openais[2604]: [TOTEM] Creating commit token because I am the rep.
Feb 12 04:10:54 nfs1-cluster openais[2604]: [TOTEM] Saving state aru 1e high seq received 1e
Feb 12 04:10:54 nfs1-cluster openais[2604]: [TOTEM] Storing new sequence id for ring 10b4
Feb 12 04:10:54 nfs1-cluster openais[2604]: [TOTEM] entering COMMIT state.
Feb 12 04:10:56 nfs1-cluster openais[2604]: [TOTEM] entering RECOVERY state.
Feb 12 04:10:56 nfs1-cluster openais[2604]: [TOTEM] position [0] member 140.90.91.240:
Feb 12 04:10:56 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4272 rep 140.90.91.240
Feb 12 04:10:56 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 04:10:56 nfs1-cluster openais[2604]: [TOTEM] position [1] member 140.90.91.242:
Feb 12 04:10:56 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4272 rep 140.90.91.240
Feb 12 04:10:56 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 04:10:56 nfs1-cluster openais[2604]: [TOTEM] Did not need to originate any messages in recovery.
Feb 12 04:10:56 nfs1-cluster openais[2604]: [TOTEM] Sending initial ORF token
Feb 12 04:10:56 nfs1-cluster openais[2604]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 04:10:56 nfs1-cluster openais[2604]: [CLM ] New Configuration:
Feb 12 04:10:56 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 04:10:56 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 04:10:56 nfs1-cluster openais[2604]: [CLM ] Members Left:
Feb 12 04:10:56 nfs1-cluster openais[2604]: [CLM ] Members Joined:
Feb 12 04:10:56 nfs1-cluster openais[2604]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 04:10:56 nfs1-cluster openais[2604]: [CLM ] New Configuration:
Feb 12 04:10:56 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 04:10:56 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 04:10:56 nfs1-cluster openais[2604]: [CLM ] Members Left:
Feb 12 04:10:56 nfs1-cluster openais[2604]: [CLM ] Members Joined:
Feb 12 04:10:56 nfs1-cluster openais[2604]: [SYNC ] This node is within the primary component and will provide service.
Feb 12 04:10:56 nfs1-cluster openais[2604]: [TOTEM] entering OPERATIONAL state.
Feb 12 04:10:56 nfs1-cluster openais[2604]: [CLM ] got nodejoin message 140.90.91.240
Feb 12 04:10:56 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2
Feb 12 04:10:56 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2 3
Feb 12 04:10:56 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2 3 4
Feb 12 04:10:57 nfs1-cluster openais[2604]: [CLM ] got nodejoin message 140.90.91.242
Feb 12 04:10:57 nfs1-cluster openais[2604]: [CPG ] got joinlist message from node 1
Feb 12 04:10:57 nfs1-cluster openais[2604]: [CPG ] got joinlist message from node 2
Feb 12 04:11:08 nfs1-cluster openais[2604]: [TOTEM] The token was lost in the OPERATIONAL state.
Feb 12 04:11:08 nfs1-cluster openais[2604]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes).
Feb 12 04:11:08 nfs1-cluster openais[2604]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes).
Feb 12 04:11:08 nfs1-cluster openais[2604]: [TOTEM] entering GATHER state from 2.
Feb 12 04:11:08 nfs1-cluster openais[2604]: [TOTEM] Creating commit token because I am the rep.
Feb 12 04:11:08 nfs1-cluster openais[2604]: [TOTEM] Saving state aru 1e high seq received 1e
Feb 12 04:11:08 nfs1-cluster openais[2604]: [TOTEM] Storing new sequence id for ring 10b8
Feb 12 04:11:08 nfs1-cluster openais[2604]: [TOTEM] entering COMMIT state.
Feb 12 04:11:11 nfs1-cluster openais[2604]: [TOTEM] entering RECOVERY state.
Feb 12 04:11:11 nfs1-cluster openais[2604]: [TOTEM] position [0] member 140.90.91.240:
Feb 12 04:11:11 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4276 rep 140.90.91.240
Feb 12 04:11:11 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 04:11:11 nfs1-cluster openais[2604]: [TOTEM] position [1] member 140.90.91.242:
Feb 12 04:11:11 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4276 rep 140.90.91.240
Feb 12 04:11:11 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 04:11:11 nfs1-cluster openais[2604]: [TOTEM] Did not need to originate any messages in recovery.
Feb 12 04:11:11 nfs1-cluster openais[2604]: [TOTEM] Sending initial ORF token
Feb 12 04:11:11 nfs1-cluster openais[2604]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 04:11:11 nfs1-cluster openais[2604]: [CLM ] New Configuration:
Feb 12 04:11:11 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 04:11:11 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 04:11:11 nfs1-cluster openais[2604]: [CLM ] Members Left:
Feb 12 04:11:11 nfs1-cluster openais[2604]: [CLM ] Members Joined:
Feb 12 04:11:11 nfs1-cluster openais[2604]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 04:11:11 nfs1-cluster openais[2604]: [CLM ] New Configuration:
Feb 12 04:11:11 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 04:11:11 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 04:11:11 nfs1-cluster openais[2604]: [CLM ] Members Left:
Feb 12 04:11:11 nfs1-cluster openais[2604]: [CLM ] Members Joined:
Feb 12 04:11:11 nfs1-cluster openais[2604]: [SYNC ] This node is within the primary component and will provide service.
Feb 12 04:11:11 nfs1-cluster openais[2604]: [TOTEM] entering OPERATIONAL state.
Feb 12 04:11:11 nfs1-cluster openais[2604]: [CLM ] got nodejoin message 140.90.91.240
Feb 12 04:11:11 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2
Feb 12 04:11:11 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2 3
Feb 12 04:11:11 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2 3 4
Feb 12 04:11:11 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2 3 4
Feb 12 04:11:11 nfs1-cluster openais[2604]: [CLM ] got nodejoin message 140.90.91.242
Feb 12 04:11:11 nfs1-cluster openais[2604]: [CPG ] got joinlist message from node 1
Feb 12 04:11:11 nfs1-cluster openais[2604]: [CPG ] got joinlist message from node 2
Feb 12 04:16:58 nfs1-cluster openais[2604]: [TOTEM] The token was lost in the OPERATIONAL state.
Feb 12 04:16:58 nfs1-cluster openais[2604]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes).
Feb 12 04:16:58 nfs1-cluster openais[2604]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes).
Feb 12 04:16:58 nfs1-cluster openais[2604]: [TOTEM] entering GATHER state from 2.
Feb 12 04:16:58 nfs1-cluster openais[2604]: [TOTEM] Creating commit token because I am the rep.
Feb 12 04:16:58 nfs1-cluster openais[2604]: [TOTEM] Saving state aru 1e high seq received 1e
Feb 12 04:16:58 nfs1-cluster openais[2604]: [TOTEM] Storing new sequence id for ring 10bc
Feb 12 04:16:58 nfs1-cluster openais[2604]: [TOTEM] entering COMMIT state.
Feb 12 04:17:01 nfs1-cluster openais[2604]: [TOTEM] entering RECOVERY state.
Feb 12 04:17:01 nfs1-cluster openais[2604]: [TOTEM] position [0] member 140.90.91.240:
Feb 12 04:17:01 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4280 rep 140.90.91.240
Feb 12 04:17:01 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 04:17:01 nfs1-cluster openais[2604]: [TOTEM] position [1] member 140.90.91.242:
Feb 12 04:17:01 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4280 rep 140.90.91.240
Feb 12 04:17:01 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 04:17:01 nfs1-cluster openais[2604]: [TOTEM] Did not need to originate any messages in recovery.
Feb 12 04:17:01 nfs1-cluster openais[2604]: [TOTEM] Sending initial ORF token
Feb 12 04:17:01 nfs1-cluster openais[2604]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 04:17:01 nfs1-cluster openais[2604]: [CLM ] New Configuration:
Feb 12 04:17:01 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 04:17:01 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 04:17:01 nfs1-cluster openais[2604]: [CLM ] Members Left:
Feb 12 04:17:01 nfs1-cluster openais[2604]: [CLM ] Members Joined:
Feb 12 04:17:01 nfs1-cluster openais[2604]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 04:17:01 nfs1-cluster openais[2604]: [CLM ] New Configuration:
Feb 12 04:17:01 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 04:17:01 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 04:17:01 nfs1-cluster openais[2604]: [CLM ] Members Left:
Feb 12 04:17:01 nfs1-cluster openais[2604]: [CLM ] Members Joined:
Feb 12 04:17:01 nfs1-cluster openais[2604]: [SYNC ] This node is within the primary component and will provide service.
Feb 12 04:17:01 nfs1-cluster openais[2604]: [TOTEM] entering OPERATIONAL state.
Feb 12 04:17:01 nfs1-cluster openais[2604]: [CLM ] got nodejoin message 140.90.91.240
Feb 12 04:17:01 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2
Feb 12 04:17:01 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2 3
Feb 12 04:17:01 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2 3 4
Feb 12 04:17:01 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2 3 4
Feb 12 04:17:01 nfs1-cluster openais[2604]: [CLM ] got nodejoin message 140.90.91.242
Feb 12 04:17:01 nfs1-cluster openais[2604]: [CPG ] got joinlist message from node 1
Feb 12 04:17:01 nfs1-cluster openais[2604]: [CPG ] got joinlist message from node 2
Feb 12 04:28:54 nfs1-cluster openais[2604]: [TOTEM] The token was lost in the OPERATIONAL state.
Feb 12 04:28:54 nfs1-cluster openais[2604]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes).
Feb 12 04:28:54 nfs1-cluster openais[2604]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes).
Feb 12 04:28:54 nfs1-cluster openais[2604]: [TOTEM] entering GATHER state from 2.
Feb 12 04:28:54 nfs1-cluster openais[2604]: [TOTEM] Creating commit token because I am the rep.
Feb 12 04:28:54 nfs1-cluster openais[2604]: [TOTEM] Saving state aru 1e high seq received 1e
Feb 12 04:28:54 nfs1-cluster openais[2604]: [TOTEM] Storing new sequence id for ring 10c0
Feb 12 04:28:54 nfs1-cluster openais[2604]: [TOTEM] entering COMMIT state.
Feb 12 04:28:56 nfs1-cluster openais[2604]: [TOTEM] entering RECOVERY state.
Feb 12 04:28:56 nfs1-cluster openais[2604]: [TOTEM] position [0] member 140.90.91.240:
Feb 12 04:28:56 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4284 rep 140.90.91.240
Feb 12 04:28:56 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 04:28:56 nfs1-cluster openais[2604]: [TOTEM] position [1] member 140.90.91.242:
Feb 12 04:28:56 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4284 rep 140.90.91.240
Feb 12 04:28:56 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 04:28:56 nfs1-cluster openais[2604]: [TOTEM] Did not need to originate any messages in recovery.
Feb 12 04:28:56 nfs1-cluster openais[2604]: [TOTEM] Sending initial ORF token
Feb 12 04:28:56 nfs1-cluster openais[2604]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 04:28:56 nfs1-cluster openais[2604]: [CLM ] New Configuration:
Feb 12 04:28:56 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 04:28:56 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 04:28:56 nfs1-cluster openais[2604]: [CLM ] Members Left:
Feb 12 04:28:56 nfs1-cluster openais[2604]: [CLM ] Members Joined:
Feb 12 04:28:56 nfs1-cluster openais[2604]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 04:28:56 nfs1-cluster openais[2604]: [CLM ] New Configuration:
Feb 12 04:28:56 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 04:28:56 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 04:28:56 nfs1-cluster openais[2604]: [CLM ] Members Left:
Feb 12 04:28:56 nfs1-cluster openais[2604]: [CLM ] Members Joined:
Feb 12 04:28:56 nfs1-cluster openais[2604]: [SYNC ] This node is within the primary component and will provide service.
Feb 12 04:28:56 nfs1-cluster openais[2604]: [TOTEM] entering OPERATIONAL state.
Feb 12 04:28:56 nfs1-cluster openais[2604]: [CLM ] got nodejoin message 140.90.91.240
Feb 12 04:28:56 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2
Feb 12 04:28:56 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2 3
Feb 12 04:28:56 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 4
Feb 12 04:28:56 nfs1-cluster openais[2604]: [CLM ] got nodejoin message 140.90.91.242
Feb 12 04:28:56 nfs1-cluster openais[2604]: [CPG ] got joinlist message from node 1
Feb 12 04:28:56 nfs1-cluster openais[2604]: [CPG ] got joinlist message from node 2
Feb 12 04:53:45 nfs1-cluster openais[2604]: [TOTEM] The token was lost in the OPERATIONAL state.
Feb 12 04:53:45 nfs1-cluster openais[2604]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes).
Feb 12 04:53:45 nfs1-cluster openais[2604]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes).
Feb 12 04:53:45 nfs1-cluster openais[2604]: [TOTEM] entering GATHER state from 2.
Feb 12 04:53:45 nfs1-cluster openais[2604]: [TOTEM] Creating commit token because I am the rep.
Feb 12 04:53:45 nfs1-cluster openais[2604]: [TOTEM] Saving state aru 1e high seq received 1e
Feb 12 04:53:45 nfs1-cluster openais[2604]: [TOTEM] Storing new sequence id for ring 10c4
Feb 12 04:53:45 nfs1-cluster openais[2604]: [TOTEM] entering COMMIT state.
Feb 12 04:53:45 nfs1-cluster openais[2604]: [TOTEM] entering RECOVERY state.
Feb 12 04:53:45 nfs1-cluster openais[2604]: [TOTEM] position [0] member 140.90.91.240:
Feb 12 04:53:45 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4288 rep 140.90.91.240
Feb 12 04:53:45 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 04:53:45 nfs1-cluster openais[2604]: [TOTEM] position [1] member 140.90.91.242:
Feb 12 04:53:45 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4288 rep 140.90.91.240
Feb 12 04:53:45 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 04:53:45 nfs1-cluster openais[2604]: [TOTEM] Did not need to originate any messages in recovery.
Feb 12 04:53:45 nfs1-cluster openais[2604]: [TOTEM] Sending initial ORF token
Feb 12 04:53:45 nfs1-cluster openais[2604]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 04:53:45 nfs1-cluster openais[2604]: [CLM ] New Configuration:
Feb 12 04:53:45 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 04:53:45 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 04:53:45 nfs1-cluster openais[2604]: [CLM ] Members Left:
Feb 12 04:53:45 nfs1-cluster openais[2604]: [CLM ] Members Joined:
Feb 12 04:53:45 nfs1-cluster openais[2604]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 04:53:45 nfs1-cluster openais[2604]: [CLM ] New Configuration:
Feb 12 04:53:45 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 04:53:45 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 04:53:45 nfs1-cluster openais[2604]: [CLM ] Members Left:
Feb 12 04:53:45 nfs1-cluster openais[2604]: [CLM ] Members Joined:
Feb 12 04:53:45 nfs1-cluster openais[2604]: [SYNC ] This node is within the primary component and will provide service.
Feb 12 04:53:45 nfs1-cluster openais[2604]: [TOTEM] entering OPERATIONAL state.
Feb 12 04:53:45 nfs1-cluster openais[2604]: [CLM ] got nodejoin message 140.90.91.240
Feb 12 04:53:45 nfs1-cluster openais[2604]: [CLM ] got nodejoin message 140.90.91.242
Feb 12 04:53:45 nfs1-cluster openais[2604]: [CPG ] got joinlist message from node 1
Feb 12 04:53:45 nfs1-cluster openais[2604]: [CPG ] got joinlist message from node 2
Feb 12 05:05:39 nfs1-cluster openais[2604]: [TOTEM] The token was lost in the OPERATIONAL state.
Feb 12 05:05:39 nfs1-cluster openais[2604]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes).
Feb 12 05:05:39 nfs1-cluster openais[2604]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes).
Feb 12 05:05:39 nfs1-cluster openais[2604]: [TOTEM] entering GATHER state from 2.
Feb 12 05:05:39 nfs1-cluster openais[2604]: [TOTEM] Creating commit token because I am the rep.
Feb 12 05:05:39 nfs1-cluster openais[2604]: [TOTEM] Saving state aru 1e high seq received 1e
Feb 12 05:05:39 nfs1-cluster openais[2604]: [TOTEM] Storing new sequence id for ring 10c8
Feb 12 05:05:39 nfs1-cluster openais[2604]: [TOTEM] entering COMMIT state.
Feb 12 05:05:39 nfs1-cluster openais[2604]: [TOTEM] entering RECOVERY state.
Feb 12 05:05:39 nfs1-cluster openais[2604]: [TOTEM] position [0] member 140.90.91.240:
Feb 12 05:05:39 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4292 rep 140.90.91.240
Feb 12 05:05:39 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 05:05:39 nfs1-cluster openais[2604]: [TOTEM] position [1] member 140.90.91.242:
Feb 12 05:05:39 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4292 rep 140.90.91.240
Feb 12 05:05:39 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 05:05:39 nfs1-cluster openais[2604]: [TOTEM] Did not need to originate any messages in recovery.
Feb 12 05:05:39 nfs1-cluster openais[2604]: [TOTEM] Sending initial ORF token
Feb 12 05:05:39 nfs1-cluster openais[2604]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 05:05:39 nfs1-cluster openais[2604]: [CLM ] New Configuration:
Feb 12 05:05:39 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 05:05:39 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 05:05:39 nfs1-cluster openais[2604]: [CLM ] Members Left:
Feb 12 05:05:39 nfs1-cluster openais[2604]: [CLM ] Members Joined:
Feb 12 05:05:39 nfs1-cluster openais[2604]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 05:05:39 nfs1-cluster openais[2604]: [CLM ] New Configuration:
Feb 12 05:05:39 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 05:05:39 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 05:05:39 nfs1-cluster openais[2604]: [CLM ] Members Left:
Feb 12 05:05:39 nfs1-cluster openais[2604]: [CLM ] Members Joined:
Feb 12 05:05:39 nfs1-cluster openais[2604]: [SYNC ] This node is within the primary component and will provide service.
Feb 12 05:05:39 nfs1-cluster openais[2604]: [TOTEM] entering OPERATIONAL state.
Feb 12 05:05:39 nfs1-cluster openais[2604]: [CLM ] got nodejoin message 140.90.91.240
Feb 12 05:05:39 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2
Feb 12 05:05:39 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2 3
Feb 12 05:05:39 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2 3 4
Feb 12 05:05:39 nfs1-cluster openais[2604]: [CLM ] got nodejoin message 140.90.91.242
Feb 12 05:05:39 nfs1-cluster openais[2604]: [CPG ] got joinlist message from node 1
Feb 12 05:05:39 nfs1-cluster openais[2604]: [CPG ] got joinlist message from node 2
Feb 12 05:09:55 nfs1-cluster openais[2604]: [TOTEM] The token was lost in the OPERATIONAL state.
Feb 12 05:09:55 nfs1-cluster openais[2604]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes).
Feb 12 05:09:55 nfs1-cluster openais[2604]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes).
Feb 12 05:09:55 nfs1-cluster openais[2604]: [TOTEM] entering GATHER state from 2.
Feb 12 05:09:55 nfs1-cluster openais[2604]: [TOTEM] Creating commit token because I am the rep.
Feb 12 05:09:55 nfs1-cluster openais[2604]: [TOTEM] Saving state aru 1e high seq received 1e
Feb 12 05:09:55 nfs1-cluster openais[2604]: [TOTEM] Storing new sequence id for ring 10cc
Feb 12 05:09:55 nfs1-cluster openais[2604]: [TOTEM] entering COMMIT state.
Feb 12 05:09:58 nfs1-cluster openais[2604]: [TOTEM] entering RECOVERY state.
Feb 12 05:09:58 nfs1-cluster openais[2604]: [TOTEM] position [0] member 140.90.91.240:
Feb 12 05:09:58 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4296 rep 140.90.91.240
Feb 12 05:09:58 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 05:09:58 nfs1-cluster openais[2604]: [TOTEM] position [1] member 140.90.91.242:
Feb 12 05:09:58 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4296 rep 140.90.91.240
Feb 12 05:09:58 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 05:09:58 nfs1-cluster openais[2604]: [TOTEM] Did not need to originate any messages in recovery.
Feb 12 05:09:58 nfs1-cluster openais[2604]: [TOTEM] Sending initial ORF token
Feb 12 05:09:58 nfs1-cluster openais[2604]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 05:09:58 nfs1-cluster openais[2604]: [CLM ] New Configuration:
Feb 12 05:09:58 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 05:09:58 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 05:09:58 nfs1-cluster openais[2604]: [CLM ] Members Left:
Feb 12 05:09:58 nfs1-cluster openais[2604]: [CLM ] Members Joined:
Feb 12 05:09:58 nfs1-cluster openais[2604]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 05:09:58 nfs1-cluster openais[2604]: [CLM ] New Configuration:
Feb 12 05:09:58 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 05:09:58 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 05:09:58 nfs1-cluster openais[2604]: [CLM ] Members Left:
Feb 12 05:09:58 nfs1-cluster openais[2604]: [CLM ] Members Joined:
Feb 12 05:09:58 nfs1-cluster openais[2604]: [SYNC ] This node is within the primary component and will provide service.
Feb 12 05:09:58 nfs1-cluster openais[2604]: [TOTEM] entering OPERATIONAL state.
Feb 12 05:09:58 nfs1-cluster openais[2604]: [CLM ] got nodejoin message 140.90.91.240
Feb 12 05:09:58 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2
Feb 12 05:09:58 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2 3
Feb 12 05:09:58 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2 3 4
Feb 12 05:09:58 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2 3 4
Feb 12 05:09:58 nfs1-cluster openais[2604]: [CLM ] got nodejoin message 140.90.91.242
Feb 12 05:09:58 nfs1-cluster openais[2604]: [CPG ] got joinlist message from node 1
Feb 12 05:09:58 nfs1-cluster openais[2604]: [CPG ] got joinlist message from node 2
Feb 12 05:48:14 nfs1-cluster openais[2604]: [TOTEM] The token was lost in the OPERATIONAL state.
Feb 12 05:48:14 nfs1-cluster openais[2604]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes).
Feb 12 05:48:14 nfs1-cluster openais[2604]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes).
Feb 12 05:48:14 nfs1-cluster openais[2604]: [TOTEM] entering GATHER state from 2.
Feb 12 05:48:14 nfs1-cluster openais[2604]: [TOTEM] Creating commit token because I am the rep.
Feb 12 05:48:14 nfs1-cluster openais[2604]: [TOTEM] Saving state aru 1e high seq received 1e
Feb 12 05:48:14 nfs1-cluster openais[2604]: [TOTEM] Storing new sequence id for ring 10d0
Feb 12 05:48:14 nfs1-cluster openais[2604]: [TOTEM] entering COMMIT state.
Feb 12 05:48:14 nfs1-cluster openais[2604]: [TOTEM] entering RECOVERY state.
Feb 12 05:48:14 nfs1-cluster openais[2604]: [TOTEM] position [0] member 140.90.91.240:
Feb 12 05:48:14 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4300 rep 140.90.91.240
Feb 12 05:48:14 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 05:48:14 nfs1-cluster openais[2604]: [TOTEM] position [1] member 140.90.91.242:
Feb 12 05:48:14 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4300 rep 140.90.91.240
Feb 12 05:48:14 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 05:48:14 nfs1-cluster openais[2604]: [TOTEM] Did not need to originate any messages in recovery.
Feb 12 05:48:14 nfs1-cluster openais[2604]: [TOTEM] Sending initial ORF token
Feb 12 05:48:14 nfs1-cluster openais[2604]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 05:48:14 nfs1-cluster openais[2604]: [CLM ] New Configuration:
Feb 12 05:48:14 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 05:48:14 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 05:48:14 nfs1-cluster openais[2604]: [CLM ] Members Left:
Feb 12 05:48:14 nfs1-cluster openais[2604]: [CLM ] Members Joined:
Feb 12 05:48:14 nfs1-cluster openais[2604]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 05:48:14 nfs1-cluster openais[2604]: [CLM ] New Configuration:
Feb 12 05:48:14 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 05:48:14 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 05:48:14 nfs1-cluster openais[2604]: [CLM ] Members Left:
Feb 12 05:48:14 nfs1-cluster openais[2604]: [CLM ] Members Joined:
Feb 12 05:48:14 nfs1-cluster openais[2604]: [SYNC ] This node is within the primary component and will provide service.
Feb 12 05:48:14 nfs1-cluster openais[2604]: [TOTEM] entering OPERATIONAL state.
Feb 12 05:48:14 nfs1-cluster openais[2604]: [CLM ] got nodejoin message 140.90.91.240
Feb 12 05:48:14 nfs1-cluster openais[2604]: [CLM ] got nodejoin message 140.90.91.242
Feb 12 05:48:14 nfs1-cluster openais[2604]: [CPG ] got joinlist message from node 1
Feb 12 05:48:14 nfs1-cluster openais[2604]: [CPG ] got joinlist message from node 2
Feb 12 05:55:34 nfs1-cluster openais[2604]: [TOTEM] The token was lost in the OPERATIONAL state.
Feb 12 05:55:34 nfs1-cluster openais[2604]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes).
Feb 12 05:55:34 nfs1-cluster openais[2604]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes).
Feb 12 05:55:34 nfs1-cluster openais[2604]: [TOTEM] entering GATHER state from 2.
Feb 12 05:55:34 nfs1-cluster openais[2604]: [TOTEM] Creating commit token because I am the rep.
Feb 12 05:55:34 nfs1-cluster openais[2604]: [TOTEM] Saving state aru 1e high seq received 1e
Feb 12 05:55:34 nfs1-cluster openais[2604]: [TOTEM] Storing new sequence id for ring 10d4
Feb 12 05:55:34 nfs1-cluster openais[2604]: [TOTEM] entering COMMIT state.
Feb 12 05:55:34 nfs1-cluster openais[2604]: [TOTEM] entering RECOVERY state.
Feb 12 05:55:34 nfs1-cluster openais[2604]: [TOTEM] position [0] member 140.90.91.240:
Feb 12 05:55:34 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4304 rep 140.90.91.240
Feb 12 05:55:34 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 05:55:34 nfs1-cluster openais[2604]: [TOTEM] position [1] member 140.90.91.242:
Feb 12 05:55:34 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4304 rep 140.90.91.240
Feb 12 05:55:34 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 05:55:34 nfs1-cluster openais[2604]: [TOTEM] Did not need to originate any messages in recovery.
Feb 12 05:55:34 nfs1-cluster openais[2604]: [TOTEM] Sending initial ORF token
Feb 12 05:55:34 nfs1-cluster openais[2604]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 05:55:34 nfs1-cluster openais[2604]: [CLM ] New Configuration:
Feb 12 05:55:34 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 05:55:34 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 05:55:34 nfs1-cluster openais[2604]: [CLM ] Members Left:
Feb 12 05:55:34 nfs1-cluster openais[2604]: [CLM ] Members Joined:
Feb 12 05:55:34 nfs1-cluster openais[2604]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 05:55:34 nfs1-cluster openais[2604]: [CLM ] New Configuration:
Feb 12 05:55:34 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 05:55:34 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 05:55:34 nfs1-cluster openais[2604]: [CLM ] Members Left:
Feb 12 05:55:34 nfs1-cluster openais[2604]: [CLM ] Members Joined:
Feb 12 05:55:34 nfs1-cluster openais[2604]: [SYNC ] This node is within the primary component and will provide service.
Feb 12 05:55:34 nfs1-cluster openais[2604]: [TOTEM] entering OPERATIONAL state.
Feb 12 05:55:34 nfs1-cluster openais[2604]: [CLM ] got nodejoin message 140.90.91.240
Feb 12 05:55:34 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2
Feb 12 05:55:34 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2 3
Feb 12 05:55:34 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2 3 4
Feb 12 05:55:35 nfs1-cluster openais[2604]: [CLM ] got nodejoin message 140.90.91.242
Feb 12 05:55:35 nfs1-cluster openais[2604]: [CPG ] got joinlist message from node 1
Feb 12 05:55:35 nfs1-cluster openais[2604]: [CPG ] got joinlist message from node 2
Feb 12 05:58:37 nfs1-cluster openais[2604]: [TOTEM] The token was lost in the OPERATIONAL state.
Feb 12 05:58:37 nfs1-cluster openais[2604]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes).
Feb 12 05:58:37 nfs1-cluster openais[2604]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes).
Feb 12 05:58:37 nfs1-cluster openais[2604]: [TOTEM] entering GATHER state from 2.
Feb 12 05:58:37 nfs1-cluster openais[2604]: [TOTEM] Creating commit token because I am the rep.
Feb 12 05:58:37 nfs1-cluster openais[2604]: [TOTEM] Saving state aru 1e high seq received 1e
Feb 12 05:58:37 nfs1-cluster openais[2604]: [TOTEM] Storing new sequence id for ring 10d8
Feb 12 05:58:37 nfs1-cluster openais[2604]: [TOTEM] entering COMMIT state.
Feb 12 05:58:37 nfs1-cluster openais[2604]: [TOTEM] entering RECOVERY state.
Feb 12 05:58:37 nfs1-cluster openais[2604]: [TOTEM] position [0] member 140.90.91.240:
Feb 12 05:58:37 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4308 rep 140.90.91.240
Feb 12 05:58:37 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 05:58:37 nfs1-cluster openais[2604]: [TOTEM] position [1] member 140.90.91.242:
Feb 12 05:58:37 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4308 rep 140.90.91.240
Feb 12 05:58:37 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 05:58:37 nfs1-cluster openais[2604]: [TOTEM] Did not need to originate any messages in recovery.
Feb 12 05:58:37 nfs1-cluster openais[2604]: [TOTEM] Sending initial ORF token
Feb 12 05:58:37 nfs1-cluster openais[2604]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 05:58:37 nfs1-cluster openais[2604]: [CLM ] New Configuration:
Feb 12 05:58:37 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 05:58:37 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 05:58:37 nfs1-cluster openais[2604]: [CLM ] Members Left:
Feb 12 05:58:37 nfs1-cluster openais[2604]: [CLM ] Members Joined:
Feb 12 05:58:37 nfs1-cluster openais[2604]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 05:58:37 nfs1-cluster openais[2604]: [CLM ] New Configuration:
Feb 12 05:58:37 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 05:58:37 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 05:58:37 nfs1-cluster openais[2604]: [CLM ] Members Left:
Feb 12 05:58:37 nfs1-cluster openais[2604]: [CLM ] Members Joined:
Feb 12 05:58:37 nfs1-cluster openais[2604]: [SYNC ] This node is within the primary component and will provide service.
Feb 12 05:58:37 nfs1-cluster openais[2604]: [TOTEM] entering OPERATIONAL state.
Feb 12 05:58:37 nfs1-cluster openais[2604]: [CLM ] got nodejoin message 140.90.91.240
Feb 12 05:58:37 nfs1-cluster openais[2604]: [CLM ] got nodejoin message 140.90.91.242
Feb 12 05:58:37 nfs1-cluster openais[2604]: [CPG ] got joinlist message from node 1
Feb 12 05:58:37 nfs1-cluster openais[2604]: [CPG ] got joinlist message from node 2
Feb 12 06:28:45 nfs1-cluster openais[2604]: [TOTEM] The token was lost in the OPERATIONAL state.
Feb 12 06:28:45 nfs1-cluster openais[2604]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes).
Feb 12 06:28:45 nfs1-cluster openais[2604]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes).
Feb 12 06:28:45 nfs1-cluster openais[2604]: [TOTEM] entering GATHER state from 2.
Feb 12 06:28:45 nfs1-cluster openais[2604]: [TOTEM] Creating commit token because I am the rep.
Feb 12 06:28:45 nfs1-cluster openais[2604]: [TOTEM] Saving state aru 1e high seq received 1e
Feb 12 06:28:45 nfs1-cluster openais[2604]: [TOTEM] Storing new sequence id for ring 10dc
Feb 12 06:28:45 nfs1-cluster openais[2604]: [TOTEM] entering COMMIT state.
Feb 12 06:28:46 nfs1-cluster openais[2604]: [TOTEM] entering RECOVERY state.
Feb 12 06:28:46 nfs1-cluster openais[2604]: [TOTEM] position [0] member 140.90.91.240:
Feb 12 06:28:46 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4312 rep 140.90.91.240
Feb 12 06:28:46 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 06:28:46 nfs1-cluster openais[2604]: [TOTEM] position [1] member 140.90.91.242:
Feb 12 06:28:46 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4312 rep 140.90.91.240
Feb 12 06:28:46 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 06:28:46 nfs1-cluster openais[2604]: [TOTEM] Did not need to originate any messages in recovery.
Feb 12 06:28:46 nfs1-cluster openais[2604]: [TOTEM] Sending initial ORF token
Feb 12 06:28:46 nfs1-cluster openais[2604]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 06:28:46 nfs1-cluster openais[2604]: [CLM ] New Configuration:
Feb 12 06:28:46 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 06:28:46 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 06:28:46 nfs1-cluster openais[2604]: [CLM ] Members Left:
Feb 12 06:28:46 nfs1-cluster openais[2604]: [CLM ] Members Joined:
Feb 12 06:28:46 nfs1-cluster openais[2604]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 06:28:46 nfs1-cluster openais[2604]: [CLM ] New Configuration:
Feb 12 06:28:46 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 06:28:46 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 06:28:46 nfs1-cluster openais[2604]: [CLM ] Members Left:
Feb 12 06:28:46 nfs1-cluster openais[2604]: [CLM ] Members Joined:
Feb 12 06:28:46 nfs1-cluster openais[2604]: [SYNC ] This node is within the primary component and will provide service.
Feb 12 06:28:46 nfs1-cluster openais[2604]: [TOTEM] entering OPERATIONAL state.
Feb 12 06:28:46 nfs1-cluster openais[2604]: [CLM ] got nodejoin message 140.90.91.240
Feb 12 06:28:46 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2
Feb 12 06:28:46 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2 3
Feb 12 06:28:46 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2 3 4
Feb 12 06:28:46 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2 3 4
Feb 12 06:28:46 nfs1-cluster openais[2604]: [CLM ] got nodejoin message 140.90.91.242
Feb 12 06:28:46 nfs1-cluster openais[2604]: [CPG ] got joinlist message from node 1
Feb 12 06:28:46 nfs1-cluster openais[2604]: [CPG ] got joinlist message from node 2
Feb 12 06:42:10 nfs1-cluster openais[2604]: [TOTEM] The token was lost in the OPERATIONAL state.
Feb 12 06:42:10 nfs1-cluster openais[2604]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes).
Feb 12 06:42:10 nfs1-cluster openais[2604]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes).
Feb 12 06:42:10 nfs1-cluster openais[2604]: [TOTEM] entering GATHER state from 2.
Feb 12 06:42:10 nfs1-cluster openais[2604]: [TOTEM] Creating commit token because I am the rep.
Feb 12 06:42:10 nfs1-cluster openais[2604]: [TOTEM] Saving state aru 1e high seq received 1e
Feb 12 06:42:10 nfs1-cluster openais[2604]: [TOTEM] Storing new sequence id for ring 10e0
Feb 12 06:42:10 nfs1-cluster openais[2604]: [TOTEM] entering COMMIT state.
Feb 12 06:42:12 nfs1-cluster openais[2604]: [TOTEM] entering RECOVERY state.
Feb 12 06:42:12 nfs1-cluster openais[2604]: [TOTEM] position [0] member 140.90.91.240:
Feb 12 06:42:12 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4316 rep 140.90.91.240
Feb 12 06:42:12 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 06:42:12 nfs1-cluster openais[2604]: [TOTEM] position [1] member 140.90.91.242:
Feb 12 06:42:12 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4316 rep 140.90.91.240
Feb 12 06:42:12 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 06:42:12 nfs1-cluster openais[2604]: [TOTEM] Did not need to originate any messages in recovery.
Feb 12 06:42:12 nfs1-cluster openais[2604]: [TOTEM] Sending initial ORF token
Feb 12 06:42:12 nfs1-cluster openais[2604]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 06:42:12 nfs1-cluster openais[2604]: [CLM ] New Configuration:
Feb 12 06:42:12 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 06:42:12 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 06:42:12 nfs1-cluster openais[2604]: [CLM ] Members Left:
Feb 12 06:42:12 nfs1-cluster openais[2604]: [CLM ] Members Joined:
Feb 12 06:42:12 nfs1-cluster openais[2604]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 06:42:12 nfs1-cluster openais[2604]: [CLM ] New Configuration:
Feb 12 06:42:12 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 06:42:12 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 06:42:12 nfs1-cluster openais[2604]: [CLM ] Members Left:
Feb 12 06:42:12 nfs1-cluster openais[2604]: [CLM ] Members Joined:
Feb 12 06:42:12 nfs1-cluster openais[2604]: [SYNC ] This node is within the primary component and will provide service.
Feb 12 06:42:12 nfs1-cluster openais[2604]: [TOTEM] entering OPERATIONAL state.
Feb 12 06:42:12 nfs1-cluster openais[2604]: [CLM ] got nodejoin message 140.90.91.240
Feb 12 06:42:12 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2
Feb 12 06:42:12 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2 3
Feb 12 06:42:12 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2 3 4
Feb 12 06:42:12 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2 3 4
Feb 12 06:42:12 nfs1-cluster openais[2604]: [CLM ] got nodejoin message 140.90.91.242
Feb 12 06:42:12 nfs1-cluster openais[2604]: [CPG ] got joinlist message from node 1
Feb 12 06:42:12 nfs1-cluster openais[2604]: [CPG ] got joinlist message from node 2
Feb 12 07:38:26 nfs1-cluster openais[2604]: [TOTEM] The token was lost in the OPERATIONAL state.
Feb 12 07:38:26 nfs1-cluster openais[2604]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes).
Feb 12 07:38:26 nfs1-cluster openais[2604]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes).
Feb 12 07:38:26 nfs1-cluster openais[2604]: [TOTEM] entering GATHER state from 2.
Feb 12 07:38:26 nfs1-cluster openais[2604]: [TOTEM] Creating commit token because I am the rep.
Feb 12 07:38:26 nfs1-cluster openais[2604]: [TOTEM] Saving state aru 1e high seq received 1e
Feb 12 07:38:26 nfs1-cluster openais[2604]: [TOTEM] Storing new sequence id for ring 10e4
Feb 12 07:38:26 nfs1-cluster openais[2604]: [TOTEM] entering COMMIT state.
Feb 12 07:38:27 nfs1-cluster openais[2604]: [TOTEM] entering RECOVERY state.
Feb 12 07:38:27 nfs1-cluster openais[2604]: [TOTEM] position [0] member 140.90.91.240:
Feb 12 07:38:27 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4320 rep 140.90.91.240
Feb 12 07:38:27 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 07:38:27 nfs1-cluster openais[2604]: [TOTEM] position [1] member 140.90.91.242:
Feb 12 07:38:27 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4320 rep 140.90.91.240
Feb 12 07:38:27 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 07:38:27 nfs1-cluster openais[2604]: [TOTEM] Did not need to originate any messages in recovery.
Feb 12 07:38:27 nfs1-cluster openais[2604]: [TOTEM] Sending initial ORF token
Feb 12 07:38:27 nfs1-cluster openais[2604]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 07:38:27 nfs1-cluster openais[2604]: [CLM ] New Configuration:
Feb 12 07:38:27 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 07:38:27 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 07:38:27 nfs1-cluster openais[2604]: [CLM ] Members Left:
Feb 12 07:38:27 nfs1-cluster openais[2604]: [CLM ] Members Joined:
Feb 12 07:38:27 nfs1-cluster openais[2604]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 07:38:27 nfs1-cluster openais[2604]: [CLM ] New Configuration:
Feb 12 07:38:27 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 07:38:27 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 07:38:27 nfs1-cluster openais[2604]: [CLM ] Members Left:
Feb 12 07:38:27 nfs1-cluster openais[2604]: [CLM ] Members Joined:
Feb 12 07:38:27 nfs1-cluster openais[2604]: [SYNC ] This node is within the primary component and will provide service.
Feb 12 07:38:27 nfs1-cluster openais[2604]: [TOTEM] entering OPERATIONAL state.
Feb 12 07:38:27 nfs1-cluster openais[2604]: [CLM ] got nodejoin message 140.90.91.240
Feb 12 07:38:27 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2
Feb 12 07:38:27 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2 3
Feb 12 07:38:27 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2 3 4
Feb 12 07:38:27 nfs1-cluster openais[2604]: [CLM ] got nodejoin message 140.90.91.242
Feb 12 07:38:27 nfs1-cluster openais[2604]: [CPG ] got joinlist message from node 1
Feb 12 07:38:27 nfs1-cluster openais[2604]: [CPG ] got joinlist message from node 2
Feb 12 07:43:29 nfs1-cluster openais[2604]: [TOTEM] The token was lost in the OPERATIONAL state.
Feb 12 07:43:29 nfs1-cluster openais[2604]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes).
Feb 12 07:43:29 nfs1-cluster openais[2604]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes).
Feb 12 07:43:29 nfs1-cluster openais[2604]: [TOTEM] entering GATHER state from 2.
Feb 12 07:43:29 nfs1-cluster openais[2604]: [TOTEM] Creating commit token because I am the rep.
Feb 12 07:43:29 nfs1-cluster openais[2604]: [TOTEM] Saving state aru 1e high seq received 1e
Feb 12 07:43:29 nfs1-cluster openais[2604]: [TOTEM] Storing new sequence id for ring 10e8
Feb 12 07:43:29 nfs1-cluster openais[2604]: [TOTEM] entering COMMIT state.
Feb 12 07:43:31 nfs1-cluster openais[2604]: [TOTEM] entering RECOVERY state.
Feb 12 07:43:31 nfs1-cluster openais[2604]: [TOTEM] position [0] member 140.90.91.240:
Feb 12 07:43:31 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4324 rep 140.90.91.240
Feb 12 07:43:31 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 07:43:31 nfs1-cluster openais[2604]: [TOTEM] position [1] member 140.90.91.242:
Feb 12 07:43:31 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4324 rep 140.90.91.240
Feb 12 07:43:31 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 07:43:31 nfs1-cluster openais[2604]: [TOTEM] Did not need to originate any messages in recovery.
Feb 12 07:43:31 nfs1-cluster openais[2604]: [TOTEM] Sending initial ORF token
Feb 12 07:43:31 nfs1-cluster openais[2604]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 07:43:31 nfs1-cluster openais[2604]: [CLM ] New Configuration:
Feb 12 07:43:31 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 07:43:31 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 07:43:31 nfs1-cluster openais[2604]: [CLM ] Members Left:
Feb 12 07:43:31 nfs1-cluster openais[2604]: [CLM ] Members Joined:
Feb 12 07:43:31 nfs1-cluster openais[2604]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 07:43:31 nfs1-cluster openais[2604]: [CLM ] New Configuration:
Feb 12 07:43:31 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 07:43:31 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 07:43:31 nfs1-cluster openais[2604]: [CLM ] Members Left:
Feb 12 07:43:31 nfs1-cluster openais[2604]: [CLM ] Members Joined:
Feb 12 07:43:31 nfs1-cluster openais[2604]: [SYNC ] This node is within the primary component and will provide service.
Feb 12 07:43:31 nfs1-cluster openais[2604]: [TOTEM] entering OPERATIONAL state.
Feb 12 07:43:31 nfs1-cluster openais[2604]: [CLM ] got nodejoin message 140.90.91.240
Feb 12 07:43:31 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2
Feb 12 07:43:31 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2 3
Feb 12 07:43:31 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2 3 4
Feb 12 07:43:31 nfs1-cluster openais[2604]: [CLM ] got nodejoin message 140.90.91.242
Feb 12 07:43:31 nfs1-cluster openais[2604]: [CPG ] got joinlist message from node 1
Feb 12 07:43:31 nfs1-cluster openais[2604]: [CPG ] got joinlist message from node 2
Feb 12 07:43:46 nfs1-cluster openais[2604]: [TOTEM] The token was lost in the OPERATIONAL state.
Feb 12 07:43:46 nfs1-cluster openais[2604]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes).
Feb 12 07:43:46 nfs1-cluster openais[2604]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes).
Feb 12 07:43:46 nfs1-cluster openais[2604]: [TOTEM] entering GATHER state from 2.
Feb 12 07:43:46 nfs1-cluster openais[2604]: [TOTEM] Creating commit token because I am the rep.
Feb 12 07:43:46 nfs1-cluster openais[2604]: [TOTEM] Saving state aru 1e high seq received 1e
Feb 12 07:43:46 nfs1-cluster openais[2604]: [TOTEM] Storing new sequence id for ring 10ec
Feb 12 07:43:46 nfs1-cluster openais[2604]: [TOTEM] entering COMMIT state.
Feb 12 07:43:46 nfs1-cluster openais[2604]: [TOTEM] entering RECOVERY state.
Feb 12 07:43:46 nfs1-cluster openais[2604]: [TOTEM] position [0] member 140.90.91.240:
Feb 12 07:43:46 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4328 rep 140.90.91.240
Feb 12 07:43:46 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 07:43:46 nfs1-cluster openais[2604]: [TOTEM] position [1] member 140.90.91.242:
Feb 12 07:43:46 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4328 rep 140.90.91.240
Feb 12 07:43:46 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 07:43:46 nfs1-cluster openais[2604]: [TOTEM] Did not need to originate any messages in recovery.
Feb 12 07:43:46 nfs1-cluster openais[2604]: [TOTEM] Sending initial ORF token
Feb 12 07:43:46 nfs1-cluster openais[2604]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 07:43:46 nfs1-cluster openais[2604]: [CLM ] New Configuration:
Feb 12 07:43:46 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 07:43:46 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 07:43:46 nfs1-cluster openais[2604]: [CLM ] Members Left:
Feb 12 07:43:46 nfs1-cluster openais[2604]: [CLM ] Members Joined:
Feb 12 07:43:46 nfs1-cluster openais[2604]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 07:43:46 nfs1-cluster openais[2604]: [CLM ] New Configuration:
Feb 12 07:43:46 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 07:43:46 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 07:43:46 nfs1-cluster openais[2604]: [CLM ] Members Left:
Feb 12 07:43:46 nfs1-cluster openais[2604]: [CLM ] Members Joined:
Feb 12 07:43:46 nfs1-cluster openais[2604]: [SYNC ] This node is within the primary component and will provide service.
Feb 12 07:43:46 nfs1-cluster openais[2604]: [TOTEM] entering OPERATIONAL state.
Feb 12 07:43:46 nfs1-cluster openais[2604]: [CLM ] got nodejoin message 140.90.91.240
Feb 12 07:43:46 nfs1-cluster openais[2604]: [CLM ] got nodejoin message 140.90.91.242
Feb 12 07:43:46 nfs1-cluster openais[2604]: [CPG ] got joinlist message from node 1
Feb 12 07:43:46 nfs1-cluster openais[2604]: [CPG ] got joinlist message from node 2
Feb 12 08:43:03 nfs1-cluster openais[2604]: [TOTEM] The token was lost in the OPERATIONAL state.
Feb 12 08:43:03 nfs1-cluster openais[2604]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes).
Feb 12 08:43:03 nfs1-cluster openais[2604]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes).
Feb 12 08:43:03 nfs1-cluster openais[2604]: [TOTEM] entering GATHER state from 2.
Feb 12 08:43:03 nfs1-cluster openais[2604]: [TOTEM] Creating commit token because I am the rep.
Feb 12 08:43:03 nfs1-cluster openais[2604]: [TOTEM] Saving state aru 1e high seq received 1e
Feb 12 08:43:03 nfs1-cluster openais[2604]: [TOTEM] Storing new sequence id for ring 10f0
Feb 12 08:43:03 nfs1-cluster openais[2604]: [TOTEM] entering COMMIT state.
Feb 12 08:43:03 nfs1-cluster openais[2604]: [TOTEM] entering RECOVERY state.
Feb 12 08:43:03 nfs1-cluster openais[2604]: [TOTEM] position [0] member 140.90.91.240:
Feb 12 08:43:03 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4332 rep 140.90.91.240
Feb 12 08:43:03 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 08:43:03 nfs1-cluster openais[2604]: [TOTEM] position [1] member 140.90.91.242:
Feb 12 08:43:03 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4332 rep 140.90.91.240
Feb 12 08:43:03 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 08:43:03 nfs1-cluster openais[2604]: [TOTEM] Did not need to originate any messages in recovery.
Feb 12 08:43:03 nfs1-cluster openais[2604]: [TOTEM] Sending initial ORF token
Feb 12 08:43:03 nfs1-cluster openais[2604]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 08:43:03 nfs1-cluster openais[2604]: [CLM ] New Configuration:
Feb 12 08:43:03 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 08:43:03 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 08:43:03 nfs1-cluster openais[2604]: [CLM ] Members Left:
Feb 12 08:43:03 nfs1-cluster openais[2604]: [CLM ] Members Joined:
Feb 12 08:43:03 nfs1-cluster openais[2604]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 08:43:03 nfs1-cluster openais[2604]: [CLM ] New Configuration:
Feb 12 08:43:03 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 08:43:03 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 08:43:03 nfs1-cluster openais[2604]: [CLM ] Members Left:
Feb 12 08:43:03 nfs1-cluster openais[2604]: [CLM ] Members Joined:
Feb 12 08:43:03 nfs1-cluster openais[2604]: [SYNC ] This node is within the primary component and will provide service.
Feb 12 08:43:03 nfs1-cluster openais[2604]: [TOTEM] entering OPERATIONAL state.
Feb 12 08:43:03 nfs1-cluster openais[2604]: [CLM ] got nodejoin message 140.90.91.240
Feb 12 08:43:03 nfs1-cluster openais[2604]: [CLM ] got nodejoin message 140.90.91.242
Feb 12 08:43:03 nfs1-cluster openais[2604]: [CPG ] got joinlist message from node 1
Feb 12 08:43:03 nfs1-cluster openais[2604]: [CPG ] got joinlist message from node 2
Feb 12 08:52:26 nfs1-cluster openais[2604]: [TOTEM] The token was lost in the OPERATIONAL state.
Feb 12 08:52:26 nfs1-cluster openais[2604]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes).
Feb 12 08:52:26 nfs1-cluster openais[2604]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes).
Feb 12 08:52:26 nfs1-cluster openais[2604]: [TOTEM] entering GATHER state from 2.
Feb 12 08:52:26 nfs1-cluster openais[2604]: [TOTEM] Creating commit token because I am the rep.
Feb 12 08:52:26 nfs1-cluster openais[2604]: [TOTEM] Saving state aru 1e high seq received 1e
Feb 12 08:52:26 nfs1-cluster openais[2604]: [TOTEM] Storing new sequence id for ring 10f4
Feb 12 08:52:26 nfs1-cluster openais[2604]: [TOTEM] entering COMMIT state.
Feb 12 08:52:26 nfs1-cluster openais[2604]: [TOTEM] entering RECOVERY state.
Feb 12 08:52:26 nfs1-cluster openais[2604]: [TOTEM] position [0] member 140.90.91.240:
Feb 12 08:52:26 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4336 rep 140.90.91.240
Feb 12 08:52:26 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 08:52:26 nfs1-cluster openais[2604]: [TOTEM] position [1] member 140.90.91.242:
Feb 12 08:52:26 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4336 rep 140.90.91.240
Feb 12 08:52:26 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 08:52:26 nfs1-cluster openais[2604]: [TOTEM] Did not need to originate any messages in recovery.
Feb 12 08:52:26 nfs1-cluster openais[2604]: [TOTEM] Sending initial ORF token
Feb 12 08:52:26 nfs1-cluster openais[2604]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 08:52:26 nfs1-cluster openais[2604]: [CLM ] New Configuration:
Feb 12 08:52:26 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 08:52:26 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 08:52:26 nfs1-cluster openais[2604]: [CLM ] Members Left:
Feb 12 08:52:26 nfs1-cluster openais[2604]: [CLM ] Members Joined:
Feb 12 08:52:26 nfs1-cluster openais[2604]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 08:52:26 nfs1-cluster openais[2604]: [CLM ] New Configuration:
Feb 12 08:52:26 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 08:52:26 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 08:52:26 nfs1-cluster openais[2604]: [CLM ] Members Left:
Feb 12 08:52:26 nfs1-cluster openais[2604]: [CLM ] Members Joined:
Feb 12 08:52:26 nfs1-cluster openais[2604]: [SYNC ] This node is within the primary component and will provide service.
Feb 12 08:52:26 nfs1-cluster openais[2604]: [TOTEM] entering OPERATIONAL state.
Feb 12 08:52:26 nfs1-cluster openais[2604]: [CLM ] got nodejoin message 140.90.91.240
Feb 12 08:52:26 nfs1-cluster openais[2604]: [CLM ] got nodejoin message 140.90.91.242
Feb 12 08:52:26 nfs1-cluster openais[2604]: [CPG ] got joinlist message from node 1
Feb 12 08:52:26 nfs1-cluster openais[2604]: [CPG ] got joinlist message from node 2
Feb 12 13:00:28 nfs1-cluster openais[2604]: [TOTEM] The token was lost in the OPERATIONAL state.
Feb 12 13:00:28 nfs1-cluster openais[2604]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes).
Feb 12 13:00:28 nfs1-cluster openais[2604]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes).
Feb 12 13:00:28 nfs1-cluster openais[2604]: [TOTEM] entering GATHER state from 2.
Feb 12 13:00:28 nfs1-cluster openais[2604]: [TOTEM] Creating commit token because I am the rep.
Feb 12 13:00:28 nfs1-cluster openais[2604]: [TOTEM] Saving state aru 1e high seq received 1e
Feb 12 13:00:28 nfs1-cluster openais[2604]: [TOTEM] Storing new sequence id for ring 10f8
Feb 12 13:00:28 nfs1-cluster openais[2604]: [TOTEM] entering COMMIT state.
Feb 12 13:00:29 nfs1-cluster openais[2604]: [TOTEM] entering RECOVERY state.
Feb 12 13:00:29 nfs1-cluster openais[2604]: [TOTEM] position [0] member 140.90.91.240:
Feb 12 13:00:29 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4340 rep 140.90.91.240
Feb 12 13:00:29 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 13:00:29 nfs1-cluster openais[2604]: [TOTEM] position [1] member 140.90.91.242:
Feb 12 13:00:29 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4340 rep 140.90.91.240
Feb 12 13:00:29 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 13:00:29 nfs1-cluster openais[2604]: [TOTEM] Did not need to originate any messages in recovery.
Feb 12 13:00:29 nfs1-cluster openais[2604]: [TOTEM] Sending initial ORF token
Feb 12 13:00:29 nfs1-cluster openais[2604]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 13:00:29 nfs1-cluster openais[2604]: [CLM ] New Configuration:
Feb 12 13:00:29 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 13:00:29 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 13:00:29 nfs1-cluster openais[2604]: [CLM ] Members Left:
Feb 12 13:00:29 nfs1-cluster openais[2604]: [CLM ] Members Joined:
Feb 12 13:00:29 nfs1-cluster openais[2604]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 13:00:29 nfs1-cluster openais[2604]: [CLM ] New Configuration:
Feb 12 13:00:29 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 13:00:29 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 13:00:29 nfs1-cluster openais[2604]: [CLM ] Members Left:
Feb 12 13:00:29 nfs1-cluster openais[2604]: [CLM ] Members Joined:
Feb 12 13:00:29 nfs1-cluster openais[2604]: [SYNC ] This node is within the primary component and will provide service.
Feb 12 13:00:29 nfs1-cluster openais[2604]: [TOTEM] entering OPERATIONAL state.
Feb 12 13:00:29 nfs1-cluster openais[2604]: [CLM ] got nodejoin message 140.90.91.240
Feb 12 13:00:29 nfs1-cluster openais[2604]: [CLM ] got nodejoin message 140.90.91.242
Feb 12 13:00:29 nfs1-cluster openais[2604]: [CPG ] got joinlist message from node 1
Feb 12 13:00:29 nfs1-cluster openais[2604]: [CPG ] got joinlist message from node 2
Feb 12 13:24:02 nfs1-cluster openais[2604]: [TOTEM] The token was lost in the OPERATIONAL state.
Feb 12 13:24:02 nfs1-cluster openais[2604]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes).
Feb 12 13:24:02 nfs1-cluster openais[2604]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes).
Feb 12 13:24:02 nfs1-cluster openais[2604]: [TOTEM] entering GATHER state from 2.
Feb 12 13:24:02 nfs1-cluster openais[2604]: [TOTEM] Creating commit token because I am the rep.
Feb 12 13:24:02 nfs1-cluster openais[2604]: [TOTEM] Saving state aru 1e high seq received 1e
Feb 12 13:24:02 nfs1-cluster openais[2604]: [TOTEM] Storing new sequence id for ring 10fc
Feb 12 13:24:02 nfs1-cluster openais[2604]: [TOTEM] entering COMMIT state.
Feb 12 13:24:02 nfs1-cluster openais[2604]: [TOTEM] entering RECOVERY state.
Feb 12 13:24:02 nfs1-cluster openais[2604]: [TOTEM] position [0] member 140.90.91.240:
Feb 12 13:24:02 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4344 rep 140.90.91.240
Feb 12 13:24:02 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 13:24:02 nfs1-cluster openais[2604]: [TOTEM] position [1] member 140.90.91.242:
Feb 12 13:24:02 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4344 rep 140.90.91.240
Feb 12 13:24:02 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 13:24:02 nfs1-cluster openais[2604]: [TOTEM] Did not need to originate any messages in recovery.
Feb 12 13:24:02 nfs1-cluster openais[2604]: [TOTEM] Sending initial ORF token
Feb 12 13:24:02 nfs1-cluster openais[2604]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 13:24:02 nfs1-cluster openais[2604]: [CLM ] New Configuration:
Feb 12 13:24:02 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 13:24:02 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 13:24:02 nfs1-cluster openais[2604]: [CLM ] Members Left:
Feb 12 13:24:02 nfs1-cluster openais[2604]: [CLM ] Members Joined:
Feb 12 13:24:02 nfs1-cluster openais[2604]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 13:24:02 nfs1-cluster openais[2604]: [CLM ] New Configuration:
Feb 12 13:24:02 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 13:24:02 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 13:24:02 nfs1-cluster openais[2604]: [CLM ] Members Left:
Feb 12 13:24:02 nfs1-cluster openais[2604]: [CLM ] Members Joined:
Feb 12 13:24:02 nfs1-cluster openais[2604]: [SYNC ] This node is within the primary component and will provide service.
Feb 12 13:24:02 nfs1-cluster openais[2604]: [TOTEM] entering OPERATIONAL state.
Feb 12 13:24:02 nfs1-cluster openais[2604]: [CLM ] got nodejoin message 140.90.91.240
Feb 12 13:24:02 nfs1-cluster openais[2604]: [CLM ] got nodejoin message 140.90.91.242
Feb 12 13:24:02 nfs1-cluster openais[2604]: [CPG ] got joinlist message from node 1
Feb 12 13:24:02 nfs1-cluster openais[2604]: [CPG ] got joinlist message from node 2
Feb 12 13:31:38 nfs1-cluster openais[2604]: [TOTEM] The token was lost in the OPERATIONAL state.
Feb 12 13:31:38 nfs1-cluster openais[2604]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes).
Feb 12 13:31:38 nfs1-cluster openais[2604]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes).
Feb 12 13:31:38 nfs1-cluster openais[2604]: [TOTEM] entering GATHER state from 2.
Feb 12 13:31:38 nfs1-cluster openais[2604]: [TOTEM] Creating commit token because I am the rep.
Feb 12 13:31:38 nfs1-cluster openais[2604]: [TOTEM] Saving state aru 1e high seq received 1e
Feb 12 13:31:38 nfs1-cluster openais[2604]: [TOTEM] Storing new sequence id for ring 1100
Feb 12 13:31:38 nfs1-cluster openais[2604]: [TOTEM] entering COMMIT state.
Feb 12 13:31:39 nfs1-cluster openais[2604]: [TOTEM] entering RECOVERY state.
Feb 12 13:31:39 nfs1-cluster openais[2604]: [TOTEM] position [0] member 140.90.91.240:
Feb 12 13:31:39 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4348 rep 140.90.91.240
Feb 12 13:31:39 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 13:31:39 nfs1-cluster openais[2604]: [TOTEM] position [1] member 140.90.91.242:
Feb 12 13:31:39 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4348 rep 140.90.91.240
Feb 12 13:31:39 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 13:31:39 nfs1-cluster openais[2604]: [TOTEM] Did not need to originate any messages in recovery.
Feb 12 13:31:39 nfs1-cluster openais[2604]: [TOTEM] Sending initial ORF token
Feb 12 13:31:39 nfs1-cluster openais[2604]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 13:31:39 nfs1-cluster openais[2604]: [CLM ] New Configuration:
Feb 12 13:31:39 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 13:31:39 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 13:31:39 nfs1-cluster openais[2604]: [CLM ] Members Left:
Feb 12 13:31:39 nfs1-cluster openais[2604]: [CLM ] Members Joined:
Feb 12 13:31:39 nfs1-cluster openais[2604]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 13:31:39 nfs1-cluster openais[2604]: [CLM ] New Configuration:
Feb 12 13:31:39 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 13:31:39 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 13:31:39 nfs1-cluster openais[2604]: [CLM ] Members Left:
Feb 12 13:31:39 nfs1-cluster openais[2604]: [CLM ] Members Joined:
Feb 12 13:31:39 nfs1-cluster openais[2604]: [SYNC ] This node is within the primary component and will provide service.
Feb 12 13:31:39 nfs1-cluster openais[2604]: [TOTEM] entering OPERATIONAL state.
Feb 12 13:31:39 nfs1-cluster openais[2604]: [CLM ] got nodejoin message 140.90.91.240
Feb 12 13:31:39 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2
Feb 12 13:31:39 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2 3
Feb 12 13:31:39 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2 3 4
Feb 12 13:31:39 nfs1-cluster openais[2604]: [TOTEM] Retransmit List: 2 3 4
Feb 12 13:31:39 nfs1-cluster openais[2604]: [CLM ] got nodejoin message 140.90.91.242
Feb 12 13:31:39 nfs1-cluster openais[2604]: [CPG ] got joinlist message from node 1
Feb 12 13:31:39 nfs1-cluster openais[2604]: [CPG ] got joinlist message from node 2
Feb 12 13:36:58 nfs1-cluster openais[2604]: [TOTEM] The token was lost in the OPERATIONAL state.
Feb 12 13:36:58 nfs1-cluster openais[2604]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes).
Feb 12 13:36:58 nfs1-cluster openais[2604]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes).
Feb 12 13:36:58 nfs1-cluster openais[2604]: [TOTEM] entering GATHER state from 2.
Feb 12 13:36:58 nfs1-cluster openais[2604]: [TOTEM] Creating commit token because I am the rep.
Feb 12 13:36:58 nfs1-cluster openais[2604]: [TOTEM] Saving state aru 1e high seq received 1e
Feb 12 13:36:58 nfs1-cluster openais[2604]: [TOTEM] Storing new sequence id for ring 1104
Feb 12 13:36:58 nfs1-cluster openais[2604]: [TOTEM] entering COMMIT state.
Feb 12 13:36:58 nfs1-cluster openais[2604]: [TOTEM] entering RECOVERY state.
Feb 12 13:36:58 nfs1-cluster openais[2604]: [TOTEM] position [0] member 140.90.91.240:
Feb 12 13:36:58 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4352 rep 140.90.91.240
Feb 12 13:36:58 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 13:36:58 nfs1-cluster openais[2604]: [TOTEM] position [1] member 140.90.91.242:
Feb 12 13:36:58 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4352 rep 140.90.91.240
Feb 12 13:36:58 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 13:36:58 nfs1-cluster openais[2604]: [TOTEM] Did not need to originate any messages in recovery.
Feb 12 13:36:58 nfs1-cluster openais[2604]: [TOTEM] Sending initial ORF token
Feb 12 13:36:58 nfs1-cluster openais[2604]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 13:36:58 nfs1-cluster openais[2604]: [CLM ] New Configuration:
Feb 12 13:36:58 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 13:36:58 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 13:36:58 nfs1-cluster openais[2604]: [CLM ] Members Left:
Feb 12 13:36:58 nfs1-cluster openais[2604]: [CLM ] Members Joined:
Feb 12 13:36:58 nfs1-cluster openais[2604]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 13:36:58 nfs1-cluster openais[2604]: [CLM ] New Configuration:
Feb 12 13:36:58 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 13:36:58 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 13:36:58 nfs1-cluster openais[2604]: [CLM ] Members Left:
Feb 12 13:36:58 nfs1-cluster openais[2604]: [CLM ] Members Joined:
Feb 12 13:36:58 nfs1-cluster openais[2604]: [SYNC ] This node is within the primary component and will provide service.
Feb 12 13:36:58 nfs1-cluster openais[2604]: [TOTEM] entering OPERATIONAL state.
Feb 12 13:36:58 nfs1-cluster openais[2604]: [CLM ] got nodejoin message 140.90.91.240
Feb 12 13:36:58 nfs1-cluster openais[2604]: [CLM ] got nodejoin message 140.90.91.242
Feb 12 13:36:58 nfs1-cluster openais[2604]: [CPG ] got joinlist message from node 1
Feb 12 13:36:58 nfs1-cluster openais[2604]: [CPG ] got joinlist message from node 2
Feb 12 14:53:54 nfs1-cluster openais[2604]: [TOTEM] entering GATHER state from 12.
Feb 12 14:53:54 nfs1-cluster openais[2604]: [TOTEM] Creating commit token because I am the rep.
Feb 12 14:53:54 nfs1-cluster openais[2604]: [TOTEM] Saving state aru 1e high seq received 1e
Feb 12 14:53:54 nfs1-cluster openais[2604]: [TOTEM] Storing new sequence id for ring 1108
Feb 12 14:53:54 nfs1-cluster openais[2604]: [TOTEM] entering COMMIT state.
Feb 12 14:54:04 nfs1-cluster openais[2604]: [TOTEM] The token was lost in the COMMIT state.
Feb 12 14:54:04 nfs1-cluster openais[2604]: [TOTEM] entering GATHER state from 4.
Feb 12 14:54:04 nfs1-cluster openais[2604]: [TOTEM] Creating commit token because I am the rep.
Feb 12 14:54:04 nfs1-cluster openais[2604]: [TOTEM] Storing new sequence id for ring 110c
Feb 12 14:54:04 nfs1-cluster openais[2604]: [TOTEM] entering COMMIT state.
Feb 12 14:54:04 nfs1-cluster openais[2604]: [TOTEM] entering RECOVERY state.
Feb 12 14:54:04 nfs1-cluster openais[2604]: [TOTEM] position [0] member 140.90.91.240:
Feb 12 14:54:04 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4356 rep 140.90.91.240
Feb 12 14:54:04 nfs1-cluster openais[2604]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 14:54:04 nfs1-cluster openais[2604]: [TOTEM] position [1] member 140.90.91.242:
Feb 12 14:54:04 nfs1-cluster openais[2604]: [TOTEM] previous ring seq 4360 rep 140.90.91.242
Feb 12 14:54:04 nfs1-cluster openais[2604]: [TOTEM] aru 32 high delivered 32 received flag 1
Feb 12 14:54:04 nfs1-cluster openais[2604]: [TOTEM] Did not need to originate any messages in recovery.
Feb 12 14:54:04 nfs1-cluster openais[2604]: [TOTEM] Sending initial ORF token
Feb 12 14:54:04 nfs1-cluster openais[2604]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 14:54:04 nfs1-cluster openais[2604]: [CLM ] New Configuration:
Feb 12 14:54:04 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 14:54:04 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 14:54:04 nfs1-cluster openais[2604]: [CLM ] Members Left:
Feb 12 14:54:04 nfs1-cluster openais[2604]: [CLM ] Members Joined:
Feb 12 14:54:04 nfs1-cluster openais[2604]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 14:54:04 nfs1-cluster openais[2604]: [CLM ] New Configuration:
Feb 12 14:54:04 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 14:54:04 nfs1-cluster openais[2604]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 14:54:04 nfs1-cluster openais[2604]: [CLM ] Members Left:
Feb 12 14:54:04 nfs1-cluster openais[2604]: [CLM ] Members Joined:
Feb 12 14:54:04 nfs1-cluster openais[2604]: [SYNC ] This node is within the primary component and will provide service.
Feb 12 14:54:04 nfs1-cluster openais[2604]: [TOTEM] entering OPERATIONAL state.
Feb 12 14:54:04 nfs1-cluster openais[2604]: [CMAN ] cman killed by node 2 because we rejoined the cluster without a full restart
Feb 12 14:57:31 nfs1-cluster openais[2603]: [MAIN ] AIS Executive Service RELEASE 'subrev 1358 version 0.80.3'
Feb 12 14:57:31 nfs1-cluster openais[2603]: [MAIN ] Copyright (C) 2002-2006 MontaVista Software, Inc and contributors.
Feb 12 14:57:31 nfs1-cluster openais[2603]: [MAIN ] Copyright (C) 2006 Red Hat, Inc.
Feb 12 14:57:31 nfs1-cluster openais[2603]: [MAIN ] AIS Executive Service: started and ready to provide service.
Feb 12 14:57:31 nfs1-cluster openais[2603]: [MAIN ] Using default multicast address of 239.192.83.8
Feb 12 14:57:31 nfs1-cluster openais[2603]: [MAIN ] openais component openais_cpg loaded.
Feb 12 14:57:31 nfs1-cluster openais[2603]: [MAIN ] Registering service handler 'openais cluster closed process group service v1.01'
Feb 12 14:57:31 nfs1-cluster openais[2603]: [MAIN ] openais component openais_cfg loaded.
Feb 12 14:57:31 nfs1-cluster openais[2603]: [MAIN ] Registering service handler 'openais configuration service'
Feb 12 14:57:31 nfs1-cluster openais[2603]: [MAIN ] openais component openais_msg loaded.
Feb 12 14:57:31 nfs1-cluster openais[2603]: [MAIN ] Registering service handler 'openais message service B.01.01'
Feb 12 14:57:31 nfs1-cluster openais[2603]: [MAIN ] openais component openais_lck loaded.
Feb 12 14:57:31 nfs1-cluster openais[2603]: [MAIN ] Registering service handler 'openais distributed locking service B.01.01'
Feb 12 14:57:31 nfs1-cluster openais[2603]: [MAIN ] openais component openais_evt loaded.
Feb 12 14:57:31 nfs1-cluster openais[2603]: [MAIN ] Registering service handler 'openais event service B.01.01'
Feb 12 14:57:31 nfs1-cluster openais[2603]: [MAIN ] openais component openais_ckpt loaded.
Feb 12 14:57:31 nfs1-cluster openais[2603]: [MAIN ] Registering service handler 'openais checkpoint service B.01.01'
Feb 12 14:57:31 nfs1-cluster openais[2603]: [MAIN ] openais component openais_amf loaded.
Feb 12 14:57:31 nfs1-cluster openais[2603]: [MAIN ] Registering service handler 'openais availability management framework B.01.01'
Feb 12 14:57:31 nfs1-cluster openais[2603]: [MAIN ] openais component openais_clm loaded.
Feb 12 14:57:31 nfs1-cluster openais[2603]: [MAIN ] Registering service handler 'openais cluster membership service B.01.01'
Feb 12 14:57:31 nfs1-cluster openais[2603]: [MAIN ] openais component openais_evs loaded.
Feb 12 14:57:31 nfs1-cluster openais[2603]: [MAIN ] Registering service handler 'openais extended virtual synchrony service'
Feb 12 14:57:31 nfs1-cluster openais[2603]: [MAIN ] openais component openais_cman loaded.
Feb 12 14:57:31 nfs1-cluster openais[2603]: [MAIN ] Registering service handler 'openais CMAN membership service 2.01'
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] Token Timeout (10000 ms) retransmit timeout (495 ms)
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] token hold (386 ms) retransmits before loss (20 retrans)
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] join (60 ms) send_join (0 ms) consensus (4800 ms) merge (200 ms)
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] downcheck (1000 ms) fail to recv const (50 msgs)
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] seqno unchanged const (30 rotations) Maximum network MTU 1500
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] window size per rotation (50 messages) maximum messages per rotation (17 messages)
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] send threads (0 threads)
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] RRP token expired timeout (495 ms)
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] RRP token problem counter (2000 ms)
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] RRP threshold (10 problem count)
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] RRP mode set to none.
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] heartbeat_failures_allowed (0)
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] max_network_delay (50 ms)
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] HeartBeat is Disabled. To enable set heartbeat_failures_allowed > 0
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] Receive multicast socket recv buffer size (262142 bytes).
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes).
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] The network interface [140.90.91.240] is now up.
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] Created or loaded sequence id 4364.140.90.91.240 for this ring.
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] entering GATHER state from 15.
Feb 12 14:57:31 nfs1-cluster openais[2603]: [SERV ] Initialising service handler 'openais extended virtual synchrony service'
Feb 12 14:57:31 nfs1-cluster openais[2603]: [SERV ] Initialising service handler 'openais cluster membership service B.01.01'
Feb 12 14:57:31 nfs1-cluster openais[2603]: [SERV ] Initialising service handler 'openais availability management framework B.01.01'
Feb 12 14:57:31 nfs1-cluster openais[2603]: [SERV ] Initialising service handler 'openais checkpoint service B.01.01'
Feb 12 14:57:31 nfs1-cluster openais[2603]: [SERV ] Initialising service handler 'openais event service B.01.01'
Feb 12 14:57:31 nfs1-cluster openais[2603]: [SERV ] Initialising service handler 'openais distributed locking service B.01.01'
Feb 12 14:57:31 nfs1-cluster openais[2603]: [SERV ] Initialising service handler 'openais message service B.01.01'
Feb 12 14:57:31 nfs1-cluster openais[2603]: [SERV ] Initialising service handler 'openais configuration service'
Feb 12 14:57:31 nfs1-cluster openais[2603]: [SERV ] Initialising service handler 'openais cluster closed process group service v1.01'
Feb 12 14:57:31 nfs1-cluster openais[2603]: [SERV ] Initialising service handler 'openais CMAN membership service 2.01'
Feb 12 14:57:31 nfs1-cluster openais[2603]: [CMAN ] CMAN 2.0.73 (built Nov 29 2007 18:41:45) started
Feb 12 14:57:31 nfs1-cluster openais[2603]: [SYNC ] Not using a virtual synchrony filter.
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] Creating commit token because I am the rep.
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] Saving state aru 0 high seq received 0
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] Storing new sequence id for ring 1110
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] entering COMMIT state.
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] entering RECOVERY state.
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] position [0] member 140.90.91.240:
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] previous ring seq 4364 rep 140.90.91.240
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] aru 0 high delivered 0 received flag 1
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] Did not need to originate any messages in recovery.
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] Sending initial ORF token
Feb 12 14:57:31 nfs1-cluster openais[2603]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 14:57:31 nfs1-cluster openais[2603]: [CLM ] New Configuration:
Feb 12 14:57:31 nfs1-cluster openais[2603]: [CLM ] Members Left:
Feb 12 14:57:31 nfs1-cluster openais[2603]: [CLM ] Members Joined:
Feb 12 14:57:31 nfs1-cluster openais[2603]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 14:57:31 nfs1-cluster openais[2603]: [CLM ] New Configuration:
Feb 12 14:57:31 nfs1-cluster openais[2603]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 14:57:31 nfs1-cluster openais[2603]: [CLM ] Members Left:
Feb 12 14:57:31 nfs1-cluster openais[2603]: [CLM ] Members Joined:
Feb 12 14:57:31 nfs1-cluster openais[2603]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 14:57:31 nfs1-cluster openais[2603]: [SYNC ] This node is within the primary component and will provide service.
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] entering OPERATIONAL state.
Feb 12 14:57:31 nfs1-cluster openais[2603]: [CMAN ] quorum regained, resuming activity
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] entering GATHER state from 11.
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] Creating commit token because I am the rep.
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] Saving state aru 3 high seq received 3
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] Storing new sequence id for ring 1114
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] entering COMMIT state.
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] entering RECOVERY state.
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] position [0] member 140.90.91.240:
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] previous ring seq 4368 rep 140.90.91.240
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] aru 3 high delivered 1 received flag 1
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] position [1] member 140.90.91.242:
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] previous ring seq 4368 rep 140.90.91.242
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] aru 42 high delivered 42 received flag 1
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] Did not need to originate any messages in recovery.
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] Sending initial ORF token
Feb 12 14:57:31 nfs1-cluster openais[2603]: [CLM ] got nodejoin message 140.90.91.240
Feb 12 14:57:31 nfs1-cluster openais[2603]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 14:57:31 nfs1-cluster openais[2603]: [CLM ] New Configuration:
Feb 12 14:57:31 nfs1-cluster openais[2603]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 14:57:31 nfs1-cluster openais[2603]: [CLM ] Members Left:
Feb 12 14:57:31 nfs1-cluster openais[2603]: [CLM ] Members Joined:
Feb 12 14:57:31 nfs1-cluster openais[2603]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 14:57:31 nfs1-cluster openais[2603]: [CLM ] New Configuration:
Feb 12 14:57:31 nfs1-cluster openais[2603]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 14:57:31 nfs1-cluster openais[2603]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 14:57:31 nfs1-cluster openais[2603]: [CLM ] Members Left:
Feb 12 14:57:31 nfs1-cluster openais[2603]: [CLM ] Members Joined:
Feb 12 14:57:31 nfs1-cluster openais[2603]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 14:57:31 nfs1-cluster openais[2603]: [SYNC ] This node is within the primary component and will provide service.
Feb 12 14:57:31 nfs1-cluster openais[2603]: [TOTEM] entering OPERATIONAL state.
Feb 12 14:57:31 nfs1-cluster openais[2603]: [CLM ] got nodejoin message 140.90.91.240
Feb 12 14:57:31 nfs1-cluster openais[2603]: [CLM ] got nodejoin message 140.90.91.242
Feb 12 14:57:31 nfs1-cluster openais[2603]: [CPG ] got joinlist message from node 2
Feb 12 14:57:32 nfs1-cluster openais[2603]: [TOTEM] Retransmit List: 1a
Feb 12 15:58:00 nfs1-cluster openais[2603]: [TOTEM] The token was lost in the OPERATIONAL state.
Feb 12 15:58:00 nfs1-cluster openais[2603]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes).
Feb 12 15:58:00 nfs1-cluster openais[2603]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes).
Feb 12 15:58:00 nfs1-cluster openais[2603]: [TOTEM] entering GATHER state from 2.
Feb 12 15:58:00 nfs1-cluster openais[2603]: [TOTEM] Creating commit token because I am the rep.
Feb 12 15:58:00 nfs1-cluster openais[2603]: [TOTEM] Saving state aru fd high seq received fd
Feb 12 15:58:00 nfs1-cluster openais[2603]: [TOTEM] Storing new sequence id for ring 1118
Feb 12 15:58:00 nfs1-cluster openais[2603]: [TOTEM] entering COMMIT state.
Feb 12 15:58:03 nfs1-cluster openais[2603]: [TOTEM] entering RECOVERY state.
Feb 12 15:58:03 nfs1-cluster openais[2603]: [TOTEM] position [0] member 140.90.91.240:
Feb 12 15:58:03 nfs1-cluster openais[2603]: [TOTEM] previous ring seq 4372 rep 140.90.91.240
Feb 12 15:58:03 nfs1-cluster openais[2603]: [TOTEM] aru fd high delivered fd received flag 1
Feb 12 15:58:03 nfs1-cluster openais[2603]: [TOTEM] position [1] member 140.90.91.242:
Feb 12 15:58:03 nfs1-cluster openais[2603]: [TOTEM] previous ring seq 4372 rep 140.90.91.240
Feb 12 15:58:03 nfs1-cluster openais[2603]: [TOTEM] aru fd high delivered fd received flag 1
Feb 12 15:58:03 nfs1-cluster openais[2603]: [TOTEM] Did not need to originate any messages in recovery.
Feb 12 15:58:03 nfs1-cluster openais[2603]: [TOTEM] Sending initial ORF token
Feb 12 15:58:03 nfs1-cluster openais[2603]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 15:58:03 nfs1-cluster openais[2603]: [CLM ] New Configuration:
Feb 12 15:58:03 nfs1-cluster openais[2603]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 15:58:03 nfs1-cluster openais[2603]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 15:58:03 nfs1-cluster openais[2603]: [CLM ] Members Left:
Feb 12 15:58:03 nfs1-cluster openais[2603]: [CLM ] Members Joined:
Feb 12 15:58:03 nfs1-cluster openais[2603]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 15:58:03 nfs1-cluster openais[2603]: [CLM ] New Configuration:
Feb 12 15:58:03 nfs1-cluster openais[2603]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 15:58:03 nfs1-cluster openais[2603]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 15:58:03 nfs1-cluster openais[2603]: [CLM ] Members Left:
Feb 12 15:58:03 nfs1-cluster openais[2603]: [CLM ] Members Joined:
Feb 12 15:58:03 nfs1-cluster openais[2603]: [SYNC ] This node is within the primary component and will provide service.
Feb 12 15:58:03 nfs1-cluster openais[2603]: [TOTEM] entering OPERATIONAL state.
Feb 12 15:58:03 nfs1-cluster openais[2603]: [CLM ] got nodejoin message 140.90.91.240
Feb 12 15:58:03 nfs1-cluster openais[2603]: [TOTEM] Retransmit List: 2
Feb 12 15:58:03 nfs1-cluster openais[2603]: [TOTEM] Retransmit List: 2 3
Feb 12 15:58:03 nfs1-cluster openais[2603]: [TOTEM] Retransmit List: 2 3 4
Feb 12 15:58:03 nfs1-cluster openais[2603]: [CLM ] got nodejoin message 140.90.91.242
Feb 12 15:58:03 nfs1-cluster openais[2603]: [CPG ] got joinlist message from node 1
Feb 12 15:58:03 nfs1-cluster openais[2603]: [CPG ] got joinlist message from node 2
Feb 12 16:01:20 nfs1-cluster openais[2603]: [TOTEM] entering GATHER state from 12.
Feb 12 16:01:20 nfs1-cluster openais[2603]: [TOTEM] Creating commit token because I am the rep.
Feb 12 16:01:20 nfs1-cluster openais[2603]: [TOTEM] Saving state aru 1e high seq received 1e
Feb 12 16:01:20 nfs1-cluster openais[2603]: [TOTEM] Storing new sequence id for ring 111c
Feb 12 16:01:20 nfs1-cluster openais[2603]: [TOTEM] entering COMMIT state.
Feb 12 16:01:20 nfs1-cluster openais[2603]: [TOTEM] entering RECOVERY state.
Feb 12 16:01:20 nfs1-cluster openais[2603]: [TOTEM] position [0] member 140.90.91.240:
Feb 12 16:01:20 nfs1-cluster openais[2603]: [TOTEM] previous ring seq 4376 rep 140.90.91.240
Feb 12 16:01:20 nfs1-cluster openais[2603]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 16:01:20 nfs1-cluster openais[2603]: [TOTEM] position [1] member 140.90.91.242:
Feb 12 16:01:20 nfs1-cluster openais[2603]: [TOTEM] previous ring seq 4376 rep 140.90.91.240
Feb 12 16:01:20 nfs1-cluster openais[2603]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 16:01:20 nfs1-cluster openais[2603]: [TOTEM] Did not need to originate any messages in recovery.
Feb 12 16:01:20 nfs1-cluster openais[2603]: [TOTEM] Sending initial ORF token
Feb 12 16:01:20 nfs1-cluster openais[2603]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 16:01:20 nfs1-cluster openais[2603]: [CLM ] New Configuration:
Feb 12 16:01:20 nfs1-cluster openais[2603]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 16:01:20 nfs1-cluster openais[2603]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 16:01:20 nfs1-cluster openais[2603]: [CLM ] Members Left:
Feb 12 16:01:20 nfs1-cluster openais[2603]: [CLM ] Members Joined:
Feb 12 16:01:20 nfs1-cluster openais[2603]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 16:01:20 nfs1-cluster openais[2603]: [CLM ] New Configuration:
Feb 12 16:01:20 nfs1-cluster openais[2603]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 16:01:20 nfs1-cluster openais[2603]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 16:01:20 nfs1-cluster openais[2603]: [CLM ] Members Left:
Feb 12 16:01:20 nfs1-cluster openais[2603]: [CLM ] Members Joined:
Feb 12 16:01:20 nfs1-cluster openais[2603]: [SYNC ] This node is within the primary component and will provide service.
Feb 12 16:01:20 nfs1-cluster openais[2603]: [TOTEM] entering OPERATIONAL state.
Feb 12 16:01:20 nfs1-cluster openais[2603]: [CLM ] got nodejoin message 140.90.91.240
Feb 12 16:01:20 nfs1-cluster openais[2603]: [CLM ] got nodejoin message 140.90.91.242
Feb 12 16:01:20 nfs1-cluster openais[2603]: [CPG ] got joinlist message from node 1
Feb 12 16:01:20 nfs1-cluster openais[2603]: [CPG ] got joinlist message from node 2
Feb 12 16:12:26 nfs1-cluster openais[2603]: [TOTEM] The token was lost in the OPERATIONAL state.
Feb 12 16:12:26 nfs1-cluster openais[2603]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes).
Feb 12 16:12:26 nfs1-cluster openais[2603]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes).
Feb 12 16:12:26 nfs1-cluster openais[2603]: [TOTEM] entering GATHER state from 2.
Feb 12 16:12:26 nfs1-cluster openais[2603]: [TOTEM] Creating commit token because I am the rep.
Feb 12 16:12:26 nfs1-cluster openais[2603]: [TOTEM] Saving state aru 1e high seq received 1e
Feb 12 16:12:26 nfs1-cluster openais[2603]: [TOTEM] Storing new sequence id for ring 1120
Feb 12 16:12:26 nfs1-cluster openais[2603]: [TOTEM] entering COMMIT state.
Feb 12 16:12:29 nfs1-cluster openais[2603]: [TOTEM] entering RECOVERY state.
Feb 12 16:12:29 nfs1-cluster openais[2603]: [TOTEM] position [0] member 140.90.91.240:
Feb 12 16:12:29 nfs1-cluster openais[2603]: [TOTEM] previous ring seq 4380 rep 140.90.91.240
Feb 12 16:12:29 nfs1-cluster openais[2603]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 16:12:29 nfs1-cluster openais[2603]: [TOTEM] position [1] member 140.90.91.242:
Feb 12 16:12:29 nfs1-cluster openais[2603]: [TOTEM] previous ring seq 4380 rep 140.90.91.240
Feb 12 16:12:29 nfs1-cluster openais[2603]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 16:12:29 nfs1-cluster openais[2603]: [TOTEM] Did not need to originate any messages in recovery.
Feb 12 16:12:29 nfs1-cluster openais[2603]: [TOTEM] Sending initial ORF token
Feb 12 16:12:29 nfs1-cluster openais[2603]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 16:12:29 nfs1-cluster openais[2603]: [CLM ] New Configuration:
Feb 12 16:12:29 nfs1-cluster openais[2603]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 16:12:29 nfs1-cluster openais[2603]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 16:12:29 nfs1-cluster openais[2603]: [CLM ] Members Left:
Feb 12 16:12:29 nfs1-cluster openais[2603]: [CLM ] Members Joined:
Feb 12 16:12:29 nfs1-cluster openais[2603]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 16:12:29 nfs1-cluster openais[2603]: [CLM ] New Configuration:
Feb 12 16:12:29 nfs1-cluster openais[2603]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 16:12:29 nfs1-cluster openais[2603]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 16:12:29 nfs1-cluster openais[2603]: [CLM ] Members Left:
Feb 12 16:12:29 nfs1-cluster openais[2603]: [CLM ] Members Joined:
Feb 12 16:12:29 nfs1-cluster openais[2603]: [SYNC ] This node is within the primary component and will provide service.
Feb 12 16:12:29 nfs1-cluster openais[2603]: [TOTEM] entering OPERATIONAL state.
Feb 12 16:12:29 nfs1-cluster openais[2603]: [CLM ] got nodejoin message 140.90.91.240
Feb 12 16:12:29 nfs1-cluster openais[2603]: [TOTEM] Retransmit List: 2
Feb 12 16:12:29 nfs1-cluster openais[2603]: [TOTEM] Retransmit List: 2 3
Feb 12 16:12:29 nfs1-cluster openais[2603]: [TOTEM] Retransmit List: 2 3 4
Feb 12 16:12:29 nfs1-cluster openais[2603]: [TOTEM] Retransmit List: 2 3 4
Feb 12 16:12:29 nfs1-cluster openais[2603]: [TOTEM] Retransmit List: 2 3
Feb 12 16:12:29 nfs1-cluster openais[2603]: [CLM ] got nodejoin message 140.90.91.242
Feb 12 16:12:29 nfs1-cluster openais[2603]: [CPG ] got joinlist message from node 1
Feb 12 16:12:29 nfs1-cluster openais[2603]: [CPG ] got joinlist message from node 2
Feb 12 16:12:56 nfs1-cluster openais[2603]: [TOTEM] The token was lost in the OPERATIONAL state.
Feb 12 16:12:56 nfs1-cluster openais[2603]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes).
Feb 12 16:12:56 nfs1-cluster openais[2603]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes).
Feb 12 16:12:56 nfs1-cluster openais[2603]: [TOTEM] entering GATHER state from 2.
Feb 12 16:12:56 nfs1-cluster openais[2603]: [TOTEM] Creating commit token because I am the rep.
Feb 12 16:12:56 nfs1-cluster openais[2603]: [TOTEM] Saving state aru 1e high seq received 1e
Feb 12 16:12:56 nfs1-cluster openais[2603]: [TOTEM] Storing new sequence id for ring 1124
Feb 12 16:12:56 nfs1-cluster openais[2603]: [TOTEM] entering COMMIT state.
Feb 12 16:12:59 nfs1-cluster openais[2603]: [TOTEM] entering RECOVERY state.
Feb 12 16:12:59 nfs1-cluster openais[2603]: [TOTEM] position [0] member 140.90.91.240:
Feb 12 16:12:59 nfs1-cluster openais[2603]: [TOTEM] previous ring seq 4384 rep 140.90.91.240
Feb 12 16:12:59 nfs1-cluster openais[2603]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 16:12:59 nfs1-cluster openais[2603]: [TOTEM] position [1] member 140.90.91.242:
Feb 12 16:12:59 nfs1-cluster openais[2603]: [TOTEM] previous ring seq 4384 rep 140.90.91.240
Feb 12 16:12:59 nfs1-cluster openais[2603]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 16:12:59 nfs1-cluster openais[2603]: [TOTEM] Did not need to originate any messages in recovery.
Feb 12 16:12:59 nfs1-cluster openais[2603]: [TOTEM] Sending initial ORF token
Feb 12 16:12:59 nfs1-cluster openais[2603]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 16:12:59 nfs1-cluster openais[2603]: [CLM ] New Configuration:
Feb 12 16:12:59 nfs1-cluster openais[2603]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 16:12:59 nfs1-cluster openais[2603]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 16:12:59 nfs1-cluster openais[2603]: [CLM ] Members Left:
Feb 12 16:12:59 nfs1-cluster openais[2603]: [CLM ] Members Joined:
Feb 12 16:12:59 nfs1-cluster openais[2603]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 16:12:59 nfs1-cluster openais[2603]: [CLM ] New Configuration:
Feb 12 16:12:59 nfs1-cluster openais[2603]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 16:12:59 nfs1-cluster openais[2603]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 16:12:59 nfs1-cluster openais[2603]: [CLM ] Members Left:
Feb 12 16:12:59 nfs1-cluster openais[2603]: [CLM ] Members Joined:
Feb 12 16:12:59 nfs1-cluster openais[2603]: [SYNC ] This node is within the primary component and will provide service.
Feb 12 16:12:59 nfs1-cluster openais[2603]: [TOTEM] entering OPERATIONAL state.
Feb 12 16:12:59 nfs1-cluster openais[2603]: [CLM ] got nodejoin message 140.90.91.240
Feb 12 16:12:59 nfs1-cluster openais[2603]: [TOTEM] Retransmit List: 2
Feb 12 16:12:59 nfs1-cluster openais[2603]: [TOTEM] Retransmit List: 2 3
Feb 12 16:12:59 nfs1-cluster openais[2603]: [TOTEM] Retransmit List: 2 3 4
Feb 12 16:12:59 nfs1-cluster openais[2603]: [TOTEM] Retransmit List: 3
Feb 12 16:12:59 nfs1-cluster openais[2603]: [CLM ] got nodejoin message 140.90.91.242
Feb 12 16:12:59 nfs1-cluster openais[2603]: [CPG ] got joinlist message from node 1
Feb 12 16:12:59 nfs1-cluster openais[2603]: [CPG ] got joinlist message from node 2
Feb 12 16:29:22 nfs1-cluster openais[2603]: [TOTEM] entering GATHER state from 12.
Feb 12 16:29:22 nfs1-cluster openais[2603]: [TOTEM] Creating commit token because I am the rep.
Feb 12 16:29:22 nfs1-cluster openais[2603]: [TOTEM] Saving state aru 1e high seq received 1e
Feb 12 16:29:22 nfs1-cluster openais[2603]: [TOTEM] Storing new sequence id for ring 1128
Feb 12 16:29:22 nfs1-cluster openais[2603]: [TOTEM] entering COMMIT state.
Feb 12 16:29:32 nfs1-cluster openais[2603]: [TOTEM] The token was lost in the COMMIT state.
Feb 12 16:29:32 nfs1-cluster openais[2603]: [TOTEM] entering GATHER state from 4.
Feb 12 16:29:32 nfs1-cluster openais[2603]: [TOTEM] Creating commit token because I am the rep.
Feb 12 16:29:32 nfs1-cluster openais[2603]: [TOTEM] Storing new sequence id for ring 112c
Feb 12 16:29:32 nfs1-cluster openais[2603]: [TOTEM] entering COMMIT state.
Feb 12 16:29:32 nfs1-cluster openais[2603]: [TOTEM] entering RECOVERY state.
Feb 12 16:29:32 nfs1-cluster openais[2603]: [TOTEM] position [0] member 140.90.91.240:
Feb 12 16:29:32 nfs1-cluster openais[2603]: [TOTEM] previous ring seq 4388 rep 140.90.91.240
Feb 12 16:29:32 nfs1-cluster openais[2603]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 16:29:32 nfs1-cluster openais[2603]: [TOTEM] position [1] member 140.90.91.242:
Feb 12 16:29:32 nfs1-cluster openais[2603]: [TOTEM] previous ring seq 4392 rep 140.90.91.242
Feb 12 16:29:32 nfs1-cluster openais[2603]: [TOTEM] aru 1c high delivered 1c received flag 1
Feb 12 16:29:32 nfs1-cluster openais[2603]: [TOTEM] Did not need to originate any messages in recovery.
Feb 12 16:29:32 nfs1-cluster openais[2603]: [TOTEM] Sending initial ORF token
Feb 12 16:29:32 nfs1-cluster openais[2603]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 16:29:32 nfs1-cluster openais[2603]: [CLM ] New Configuration:
Feb 12 16:29:32 nfs1-cluster openais[2603]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 16:29:32 nfs1-cluster openais[2603]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 16:29:32 nfs1-cluster openais[2603]: [CLM ] Members Left:
Feb 12 16:29:32 nfs1-cluster openais[2603]: [CLM ] Members Joined:
Feb 12 16:29:32 nfs1-cluster openais[2603]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 16:29:32 nfs1-cluster openais[2603]: [CLM ] New Configuration:
Feb 12 16:29:32 nfs1-cluster openais[2603]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 16:29:32 nfs1-cluster openais[2603]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 16:29:32 nfs1-cluster openais[2603]: [CLM ] Members Left:
Feb 12 16:29:32 nfs1-cluster openais[2603]: [CLM ] Members Joined:
Feb 12 16:29:32 nfs1-cluster openais[2603]: [SYNC ] This node is within the primary component and will provide service.
Feb 12 16:29:32 nfs1-cluster openais[2603]: [TOTEM] entering OPERATIONAL state.
Feb 12 16:29:32 nfs1-cluster openais[2603]: [CMAN ] cman killed by node 2 because we rejoined the cluster without a full restart
Feb 12 16:32:57 nfs1-cluster openais[2603]: [MAIN ] AIS Executive Service RELEASE 'subrev 1358 version 0.80.3'
Feb 12 16:32:57 nfs1-cluster openais[2603]: [MAIN ] Copyright (C) 2002-2006 MontaVista Software, Inc and contributors.
Feb 12 16:32:57 nfs1-cluster openais[2603]: [MAIN ] Copyright (C) 2006 Red Hat, Inc.
Feb 12 16:32:57 nfs1-cluster openais[2603]: [MAIN ] AIS Executive Service: started and ready to provide service.
Feb 12 16:32:57 nfs1-cluster openais[2603]: [MAIN ] Using default multicast address of 239.192.83.8
Feb 12 16:32:57 nfs1-cluster openais[2603]: [MAIN ] openais component openais_cpg loaded.
Feb 12 16:32:57 nfs1-cluster openais[2603]: [MAIN ] Registering service handler 'openais cluster closed process group service v1.01'
Feb 12 16:32:57 nfs1-cluster openais[2603]: [MAIN ] openais component openais_cfg loaded.
Feb 12 16:32:57 nfs1-cluster openais[2603]: [MAIN ] Registering service handler 'openais configuration service'
Feb 12 16:32:57 nfs1-cluster openais[2603]: [MAIN ] openais component openais_msg loaded.
Feb 12 16:32:57 nfs1-cluster openais[2603]: [MAIN ] Registering service handler 'openais message service B.01.01'
Feb 12 16:32:57 nfs1-cluster openais[2603]: [MAIN ] openais component openais_lck loaded.
Feb 12 16:32:57 nfs1-cluster openais[2603]: [MAIN ] Registering service handler 'openais distributed locking service B.01.01'
Feb 12 16:32:57 nfs1-cluster openais[2603]: [MAIN ] openais component openais_evt loaded.
Feb 12 16:32:57 nfs1-cluster openais[2603]: [MAIN ] Registering service handler 'openais event service B.01.01'
Feb 12 16:32:57 nfs1-cluster openais[2603]: [MAIN ] openais component openais_ckpt loaded.
Feb 12 16:32:57 nfs1-cluster openais[2603]: [MAIN ] Registering service handler 'openais checkpoint service B.01.01'
Feb 12 16:32:57 nfs1-cluster openais[2603]: [MAIN ] openais component openais_amf loaded.
Feb 12 16:32:57 nfs1-cluster openais[2603]: [MAIN ] Registering service handler 'openais availability management framework B.01.01'
Feb 12 16:32:57 nfs1-cluster openais[2603]: [MAIN ] openais component openais_clm loaded.
Feb 12 16:32:57 nfs1-cluster openais[2603]: [MAIN ] Registering service handler 'openais cluster membership service B.01.01'
Feb 12 16:32:57 nfs1-cluster openais[2603]: [MAIN ] openais component openais_evs loaded.
Feb 12 16:32:57 nfs1-cluster openais[2603]: [MAIN ] Registering service handler 'openais extended virtual synchrony service'
Feb 12 16:32:57 nfs1-cluster openais[2603]: [MAIN ] openais component openais_cman loaded.
Feb 12 16:32:57 nfs1-cluster openais[2603]: [MAIN ] Registering service handler 'openais CMAN membership service 2.01'
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] Token Timeout (10000 ms) retransmit timeout (495 ms)
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] token hold (386 ms) retransmits before loss (20 retrans)
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] join (60 ms) send_join (0 ms) consensus (4800 ms) merge (200 ms)
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] downcheck (1000 ms) fail to recv const (50 msgs)
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] seqno unchanged const (30 rotations) Maximum network MTU 1500
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] window size per rotation (50 messages) maximum messages per rotation (17 messages)
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] send threads (0 threads)
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] RRP token expired timeout (495 ms)
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] RRP token problem counter (2000 ms)
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] RRP threshold (10 problem count)
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] RRP mode set to none.
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] heartbeat_failures_allowed (0)
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] max_network_delay (50 ms)
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] HeartBeat is Disabled. To enable set heartbeat_failures_allowed > 0
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] Receive multicast socket recv buffer size (262142 bytes).
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes).
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] The network interface [140.90.91.240] is now up.
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] Created or loaded sequence id 4396.140.90.91.240 for this ring.
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] entering GATHER state from 15.
Feb 12 16:32:57 nfs1-cluster openais[2603]: [SERV ] Initialising service handler 'openais extended virtual synchrony service'
Feb 12 16:32:57 nfs1-cluster openais[2603]: [SERV ] Initialising service handler 'openais cluster membership service B.01.01'
Feb 12 16:32:57 nfs1-cluster openais[2603]: [SERV ] Initialising service handler 'openais availability management framework B.01.01'
Feb 12 16:32:57 nfs1-cluster openais[2603]: [SERV ] Initialising service handler 'openais checkpoint service B.01.01'
Feb 12 16:32:57 nfs1-cluster openais[2603]: [SERV ] Initialising service handler 'openais event service B.01.01'
Feb 12 16:32:57 nfs1-cluster openais[2603]: [SERV ] Initialising service handler 'openais distributed locking service B.01.01'
Feb 12 16:32:57 nfs1-cluster openais[2603]: [SERV ] Initialising service handler 'openais message service B.01.01'
Feb 12 16:32:57 nfs1-cluster openais[2603]: [SERV ] Initialising service handler 'openais configuration service'
Feb 12 16:32:57 nfs1-cluster openais[2603]: [SERV ] Initialising service handler 'openais cluster closed process group service v1.01'
Feb 12 16:32:57 nfs1-cluster openais[2603]: [SERV ] Initialising service handler 'openais CMAN membership service 2.01'
Feb 12 16:32:57 nfs1-cluster openais[2603]: [CMAN ] CMAN 2.0.73 (built Nov 29 2007 18:41:45) started
Feb 12 16:32:57 nfs1-cluster openais[2603]: [SYNC ] Not using a virtual synchrony filter.
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] Creating commit token because I am the rep.
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] Saving state aru 0 high seq received 0
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] Storing new sequence id for ring 1130
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] entering COMMIT state.
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] entering RECOVERY state.
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] position [0] member 140.90.91.240:
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] previous ring seq 4396 rep 140.90.91.240
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] aru 0 high delivered 0 received flag 1
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] Did not need to originate any messages in recovery.
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] Sending initial ORF token
Feb 12 16:32:57 nfs1-cluster openais[2603]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 16:32:57 nfs1-cluster openais[2603]: [CLM ] New Configuration:
Feb 12 16:32:57 nfs1-cluster openais[2603]: [CLM ] Members Left:
Feb 12 16:32:57 nfs1-cluster openais[2603]: [CLM ] Members Joined:
Feb 12 16:32:57 nfs1-cluster openais[2603]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 16:32:57 nfs1-cluster openais[2603]: [CLM ] New Configuration:
Feb 12 16:32:57 nfs1-cluster openais[2603]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 16:32:57 nfs1-cluster openais[2603]: [CLM ] Members Left:
Feb 12 16:32:57 nfs1-cluster openais[2603]: [CLM ] Members Joined:
Feb 12 16:32:57 nfs1-cluster openais[2603]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 16:32:57 nfs1-cluster openais[2603]: [SYNC ] This node is within the primary component and will provide service.
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] entering OPERATIONAL state.
Feb 12 16:32:57 nfs1-cluster openais[2603]: [CMAN ] quorum regained, resuming activity
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] entering GATHER state from 11.
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] Creating commit token because I am the rep.
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] Saving state aru 3 high seq received 3
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] Storing new sequence id for ring 1134
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] entering COMMIT state.
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] entering RECOVERY state.
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] position [0] member 140.90.91.240:
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] previous ring seq 4400 rep 140.90.91.240
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] aru 3 high delivered 1 received flag 1
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] position [1] member 140.90.91.242:
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] previous ring seq 4400 rep 140.90.91.242
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] aru 40 high delivered 40 received flag 1
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] Did not need to originate any messages in recovery.
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] Sending initial ORF token
Feb 12 16:32:57 nfs1-cluster openais[2603]: [CLM ] got nodejoin message 140.90.91.240
Feb 12 16:32:57 nfs1-cluster openais[2603]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 16:32:57 nfs1-cluster openais[2603]: [CLM ] New Configuration:
Feb 12 16:32:57 nfs1-cluster openais[2603]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 16:32:57 nfs1-cluster openais[2603]: [CLM ] Members Left:
Feb 12 16:32:57 nfs1-cluster openais[2603]: [CLM ] Members Joined:
Feb 12 16:32:57 nfs1-cluster openais[2603]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 16:32:57 nfs1-cluster openais[2603]: [CLM ] New Configuration:
Feb 12 16:32:57 nfs1-cluster openais[2603]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 16:32:57 nfs1-cluster openais[2603]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 16:32:57 nfs1-cluster openais[2603]: [CLM ] Members Left:
Feb 12 16:32:57 nfs1-cluster openais[2603]: [CLM ] Members Joined:
Feb 12 16:32:57 nfs1-cluster openais[2603]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 16:32:57 nfs1-cluster openais[2603]: [SYNC ] This node is within the primary component and will provide service.
Feb 12 16:32:57 nfs1-cluster openais[2603]: [TOTEM] entering OPERATIONAL state.
Feb 12 16:32:57 nfs1-cluster openais[2603]: [CLM ] got nodejoin message 140.90.91.240
Feb 12 16:32:57 nfs1-cluster openais[2603]: [CLM ] got nodejoin message 140.90.91.242
Feb 12 16:32:57 nfs1-cluster openais[2603]: [CPG ] got joinlist message from node 2
Feb 12 16:34:40 nfs1-cluster openais[2603]: [TOTEM] The token was lost in the OPERATIONAL state.
Feb 12 16:34:40 nfs1-cluster openais[2603]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes).
Feb 12 16:34:40 nfs1-cluster openais[2603]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes).
Feb 12 16:34:40 nfs1-cluster openais[2603]: [TOTEM] entering GATHER state from 2.
Feb 12 16:34:40 nfs1-cluster openais[2603]: [TOTEM] Creating commit token because I am the rep.
Feb 12 16:34:40 nfs1-cluster openais[2603]: [TOTEM] Saving state aru fe high seq received fe
Feb 12 16:34:40 nfs1-cluster openais[2603]: [TOTEM] Storing new sequence id for ring 1138
Feb 12 16:34:40 nfs1-cluster openais[2603]: [TOTEM] entering COMMIT state.
Feb 12 16:34:42 nfs1-cluster openais[2603]: [TOTEM] entering RECOVERY state.
Feb 12 16:34:42 nfs1-cluster openais[2603]: [TOTEM] position [0] member 140.90.91.240:
Feb 12 16:34:42 nfs1-cluster openais[2603]: [TOTEM] previous ring seq 4404 rep 140.90.91.240
Feb 12 16:34:42 nfs1-cluster openais[2603]: [TOTEM] aru fe high delivered fe received flag 1
Feb 12 16:34:42 nfs1-cluster openais[2603]: [TOTEM] position [1] member 140.90.91.242:
Feb 12 16:34:42 nfs1-cluster openais[2603]: [TOTEM] previous ring seq 4404 rep 140.90.91.240
Feb 12 16:34:42 nfs1-cluster openais[2603]: [TOTEM] aru fe high delivered fe received flag 1
Feb 12 16:34:42 nfs1-cluster openais[2603]: [TOTEM] Did not need to originate any messages in recovery.
Feb 12 16:34:42 nfs1-cluster openais[2603]: [TOTEM] Sending initial ORF token
Feb 12 16:34:42 nfs1-cluster openais[2603]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 16:34:42 nfs1-cluster openais[2603]: [CLM ] New Configuration:
Feb 12 16:34:42 nfs1-cluster openais[2603]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 16:34:42 nfs1-cluster openais[2603]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 16:34:42 nfs1-cluster openais[2603]: [CLM ] Members Left:
Feb 12 16:34:42 nfs1-cluster openais[2603]: [CLM ] Members Joined:
Feb 12 16:34:42 nfs1-cluster openais[2603]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 16:34:42 nfs1-cluster openais[2603]: [CLM ] New Configuration:
Feb 12 16:34:42 nfs1-cluster openais[2603]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 16:34:42 nfs1-cluster openais[2603]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 16:34:42 nfs1-cluster openais[2603]: [CLM ] Members Left:
Feb 12 16:34:42 nfs1-cluster openais[2603]: [CLM ] Members Joined:
Feb 12 16:34:42 nfs1-cluster openais[2603]: [SYNC ] This node is within the primary component and will provide service.
Feb 12 16:34:42 nfs1-cluster openais[2603]: [TOTEM] entering OPERATIONAL state.
Feb 12 16:34:42 nfs1-cluster openais[2603]: [CLM ] got nodejoin message 140.90.91.240
Feb 12 16:34:42 nfs1-cluster openais[2603]: [TOTEM] Retransmit List: 2
Feb 12 16:34:42 nfs1-cluster openais[2603]: [TOTEM] Retransmit List: 2 3
Feb 12 16:34:42 nfs1-cluster openais[2603]: [TOTEM] Retransmit List: 2 3 4
Feb 12 16:34:42 nfs1-cluster openais[2603]: [CLM ] got nodejoin message 140.90.91.242
Feb 12 16:34:42 nfs1-cluster openais[2603]: [CPG ] got joinlist message from node 1
Feb 12 16:34:42 nfs1-cluster openais[2603]: [CPG ] got joinlist message from node 2
Feb 12 16:40:30 nfs1-cluster openais[2603]: [TOTEM] The token was lost in the OPERATIONAL state.
Feb 12 16:40:30 nfs1-cluster openais[2603]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes).
Feb 12 16:40:30 nfs1-cluster openais[2603]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes).
Feb 12 16:40:30 nfs1-cluster openais[2603]: [TOTEM] entering GATHER state from 2.
Feb 12 16:40:30 nfs1-cluster openais[2603]: [TOTEM] Creating commit token because I am the rep.
Feb 12 16:40:30 nfs1-cluster openais[2603]: [TOTEM] Saving state aru 1e high seq received 1e
Feb 12 16:40:30 nfs1-cluster openais[2603]: [TOTEM] Storing new sequence id for ring 113c
Feb 12 16:40:30 nfs1-cluster openais[2603]: [TOTEM] entering COMMIT state.
Feb 12 16:40:31 nfs1-cluster openais[2603]: [TOTEM] entering RECOVERY state.
Feb 12 16:40:31 nfs1-cluster openais[2603]: [TOTEM] position [0] member 140.90.91.240:
Feb 12 16:40:31 nfs1-cluster openais[2603]: [TOTEM] previous ring seq 4408 rep 140.90.91.240
Feb 12 16:40:31 nfs1-cluster openais[2603]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 16:40:31 nfs1-cluster openais[2603]: [TOTEM] position [1] member 140.90.91.242:
Feb 12 16:40:31 nfs1-cluster openais[2603]: [TOTEM] previous ring seq 4408 rep 140.90.91.240
Feb 12 16:40:31 nfs1-cluster openais[2603]: [TOTEM] aru 1e high delivered 1e received flag 1
Feb 12 16:40:31 nfs1-cluster openais[2603]: [TOTEM] Did not need to originate any messages in recovery.
Feb 12 16:40:31 nfs1-cluster openais[2603]: [TOTEM] Sending initial ORF token
Feb 12 16:40:31 nfs1-cluster openais[2603]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 16:40:31 nfs1-cluster openais[2603]: [CLM ] New Configuration:
Feb 12 16:40:31 nfs1-cluster openais[2603]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 16:40:31 nfs1-cluster openais[2603]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 16:40:31 nfs1-cluster openais[2603]: [CLM ] Members Left:
Feb 12 16:40:31 nfs1-cluster openais[2603]: [CLM ] Members Joined:
Feb 12 16:40:31 nfs1-cluster openais[2603]: [CLM ] CLM CONFIGURATION CHANGE
Feb 12 16:40:31 nfs1-cluster openais[2603]: [CLM ] New Configuration:
Feb 12 16:40:31 nfs1-cluster openais[2603]: [CLM ] r(0) ip(140.90.91.240)
Feb 12 16:40:31 nfs1-cluster openais[2603]: [CLM ] r(0) ip(140.90.91.242)
Feb 12 16:40:31 nfs1-cluster openais[2603]: [CLM ] Members Left:
Feb 12 16:40:31 nfs1-cluster openais[2603]: [CLM ] Members Joined:
Feb 12 16:40:31 nfs1-cluster openais[2603]: [SYNC ] This node is within the primary component and will provide service.
Feb 12 16:40:31 nfs1-cluster openais[2603]: [TOTEM] entering OPERATIONAL state.
Feb 12 16:40:31 nfs1-cluster openais[2603]: [CLM ] got nodejoin message 140.90.91.240
Feb 12 16:40:31 nfs1-cluster openais[2603]: [TOTEM] Retransmit List: 2
Feb 12 16:40:31 nfs1-cluster openais[2603]: [TOTEM] Retransmit List: 2 3
Feb 12 16:40:31 nfs1-cluster openais[2603]: [TOTEM] Retransmit List: 2 3 4
Feb 12 16:40:31 nfs1-cluster openais[2603]: [CLM ] got nodejoin message 140.90.91.242
Feb 12 16:40:31 nfs1-cluster openais[2603]: [CPG ] got joinlist message from node 1
Feb 12 16:40:31 nfs1-cluster openais[2603]: [CPG ] got joinlist message from node 2
<?xml version="1.0"?>
<cluster alias="ohd_cluster" config_version="154" name="ohd_cluster">
<fence_daemon post_fail_delay="0" post_join_delay="60"/>
<clusternodes>
<clusternode name="nfs1-cluster.nws.noaa.gov" nodeid="1" votes="1">
<fence>
<method name="1">
<device name="nfspower" port="8" switch="1"/>
</method>
</fence>
</clusternode>
<clusternode name="nfs2-cluster.nws.noaa.gov" nodeid="2" votes="1">
<fence>
<method name="1">
<device name="nfspower" port="7" switch="1"/>
</method>
</fence>
</clusternode>
</clusternodes>
<cman expected_votes="1" two_node="1"/>
<rm>
<failoverdomains>
<failoverdomain name="nfs-failover" ordered="0" restricted="1">
<failoverdomainnode name="nfs1-cluster.nws.noaa.gov" priority="1"/>
<failoverdomainnode name="nfs2-cluster.nws.noaa.gov" priority="1"/>
</failoverdomain>
</failoverdomains>
<resources>
<ip address="140.90.91.244" monitor_link="1"/>
<clusterfs device="/dev/VolGroupFS/LogVol-shared" force_unmount="0" fsid="30647" fstype="gfs" mountpoint="/fs/shared" name="fs-shared" options="rw"/>
<nfsexport name="fs-shared-exp"/>
<nfsclient name="fs-shared-client" options="no_root_squash,rw" path="/fs/shared" target="140.90.91.0/24"/>
<ip address="140.90.91.245" monitor_link="1"/>
<clusterfs device="/dev/VolGroupFS/LogVol-brianj" force_unmount="0" fsid="39025" fstype="gfs" mountpoint="/fs/rfcdata" name="fs-rfcdata" options="rw"/>
<nfsexport name="fs-rfcdata-exp"/>
<nfsclient name="fs-rfcdata-client" options="no_root_squash,rw" path="/fs/rfcdata" target="140.90.91.0/24"/>
<clusterfs device="/dev/VolGroupFS/LogVol-hsmb2" force_unmount="0" fsid="39203" fstype="gfs" mountpoint="/fs/hsmb2" name="fs-hsmb2" options="rw"/>
<clusterfs device="/dev/VolGroupFS/LogVol-hsmb5" force_unmount="0" fsid="3200" fstype="gfs" mountpoint="/fs/hsmb5" name="fs-hsmb5" options="rw"/>
<clusterfs device="/dev/VolGroupFS/LogVol-nobk-hsmb3" force_unmount="0" fsid="47183" fstype="gfs" mountpoint="/nobk/hsmb3" name="nobk-hsmb3" options="rw"/>
<clusterfs device="/dev/VolGroupFS/LogVol-nobk-hsmb4" force_unmount="0" fsid="11756" fstype="gfs" mountpoint="/nobk/hsmb4" name="nobk-hsmb4" options="rw"/>
<nfsexport name="fs-hsmb2-exp"/>
<nfsexport name="fs-hsmb5-exp"/>
<nfsexport name="nobk-hsmb3-exp"/>
<nfsexport name="nobk-hsmb4-exp"/>
<nfsclient name="fs-hsmb2-client" options="no_root_squash,rw" path="/fs/hsmb2" target="140.90.91.0/24"/>
<nfsclient name="fs-hsmb5-client" options="no_root_squash,rw" path="/fs/hsmb5" target="140.90.91.0/24"/>
<nfsclient name="nobk-hsmb3-client" options="no_root_squash,rw" path="/nobk/hsmb3" target="140.90.91.0/24"/>
<nfsclient name="nobk-hsmb4-client" options="no_root_squash,rw" path="/nobk/hsmb4" target="140.90.91.0/24"/>
<clusterfs device="/dev/VolGroupFS/LogVol-ensembles" force_unmount="0" fsid="36988" fstype="gfs" mountpoint="/fs/ensembles" name="fs-ensembles" options="rw"/>
<nfsexport name="fs-ensembles-exp"/>
<nfsclient name="fs-ensembles-client" options="no_root_squash,rw" path="/fs/ensembles" target="140.90.91.0/24"/>
<clusterfs device="/dev/VolGroupFS/LogVol-hseb" force_unmount="0" fsid="41992" fstype="gfs" mountpoint="/fs/hseb" name="fs-hseb" options="rw"/>
<nfsexport name="fs-hseb-exp"/>
<nfsclient name="fs-hseb-client" options="no_root_squash,rw" path="/fs/hseb" target="140.90.91.0/24"/>
</resources>
<service autostart="1" domain="nfs-failover" name="nfs">
<clusterfs ref="fs-shared">
<nfsexport ref="fs-shared-exp">
<nfsclient ref="fs-shared-client"/>
</nfsexport>
</clusterfs>
<ip ref="140.90.91.244"/>
<clusterfs ref="fs-rfcdata">
<nfsexport ref="fs-rfcdata-exp">
<nfsclient ref="fs-rfcdata-client"/>
</nfsexport>
<ip ref="140.90.91.244"/>
</clusterfs>
<clusterfs ref="fs-hsmb2">
<nfsexport ref="fs-hsmb2-exp">
<nfsclient ref="fs-hsmb2-client"/>
</nfsexport>
<ip ref="140.90.91.244"/>
</clusterfs>
<clusterfs ref="fs-hsmb5">
<nfsexport ref="fs-hsmb5-exp">
<nfsclient ref="fs-hsmb5-client"/>
</nfsexport>
<ip ref="140.90.91.244"/>
</clusterfs>
<clusterfs ref="nobk-hsmb3">
<nfsexport ref="nobk-hsmb3-exp">
<nfsclient ref="nobk-hsmb3-client"/>
</nfsexport>
<ip ref="140.90.91.244"/>
</clusterfs>
<clusterfs ref="nobk-hsmb4">
<nfsexport ref="nobk-hsmb4-exp">
<nfsclient ref="nobk-hsmb4-client"/>
</nfsexport>
<ip ref="140.90.91.244"/>
</clusterfs>
<clusterfs ref="fs-ensembles">
<nfsexport ref="fs-ensembles-exp">
<nfsclient ref="fs-ensembles-client"/>
</nfsexport>
</clusterfs>
<ip ref="140.90.91.244"/>
<clusterfs ref="fs-hseb">
<nfsexport ref="fs-hseb-exp">
<nfsclient ref="fs-hseb-client"/>
</nfsexport>
<ip ref="140.90.91.244"/>
</clusterfs>
</service>
</rm>
<fencedevices>
<fencedevice agent="fence_apc" ipaddr="192.168.42.30" login="cluster" name="nfspower" passwd="Tele4m32"/>
</fencedevices>
</cluster>
begin:vcard
fn:Randy Brown
n:Brown;Randy
org:National Weather Service;Office of Hydrologic Development
adr:;;1325 East West Highway;Silver Spring;MD;20910;USA
email;internet:randy.brown@xxxxxxxx
title:Senior Systems Administrator
tel;work:301-713-1669 x110
url:http://www.nws.noaa.gov/ohd/
version:2.1
end:vcard
--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster
