John Ruemker wrote:
> Gerhard Spiegl wrote:
>> Hi all,
>>
>> I'm working on a two node cluster (RHEL 5.2 + RHCS) with one
>> XEN virtual machine per node:
>>
>> node1 => VM1
>> node2 => VM2
>>
>> When node1 takes over VM2 via the command:
>>
>> clusvcadm -M vm:VM2 -m node1
>>
>> node2 gets fenced after takeover is done, which is probably expected
>> behaviour.
>>
>
> This is not expected. The vm should migrate and both nodes should
> continue running.
>
>> As node2 comes up again it fetches his VM2 back (nofailback="0", but also
>> fences node1 (ipmilan) where VM1 is still running an therefore
>> interrupted and restartet on node2.
>> When node1 comes up the same game in the other direction begins.
>> Is there a way to avoid this fence loop?
>>
>> In other words: can a service be migrated from node1 to node2 without
>> other
>> services that run on node1 being interrupted?
>>
>
> Are both nodes successfully joined in the cluster? What does 'cman_tool
> nodes' say? Can you attach logs showing all of this happening?
>
Hi,
cman_tool node before an during the migration:
[root@ols011p ~]# cman_tool nodes
Node Sts Inc Joined Name
0 M 0 2008-07-02 12:50:31 /dev/mapper/HDS-00F9p2
1 M 1228 2008-07-02 12:50:19 ols011p.ops.ctbto.org
2 M 1232 2008-07-02 12:50:19 ols012p.ops.ctbto.org
[root@ols012p ~]# cman_tool nodes
Node Sts Inc Joined Name
0 M 0 2008-07-02 12:50:32 /dev/mapper/HDS-00F9p2
1 M 1232 2008-07-02 12:50:19 ols011p.ops.ctbto.org
2 M 1224 2008-07-02 12:49:51 ols012p.ops.ctbto.org
everything seems fine.
The logs are attached in seperate files.
thanks
Gerhard
>
> John
>
> --
> Linux-cluster mailing list
> Linux-cluster@xxxxxxxxxx
> https://www.redhat.com/mailman/listinfo/linux-cluster
Jul 2 14:00:49 ols011p syslog-ng[6692]: Log statistics; dropped='tcp(AF_INET(172.27.19.171:1514))=0', processed='center(queued)=2622', processed='center(received)=1311', processed='destination(d_messages)=1311', processed='destination(server)=1311', processed='source(s_local)=1311'
Jul 2 14:01:01 ols011p crond[24314]: (root) CMD (run-parts /etc/cron.hourly)
Jul 2 14:01:10 ols011p logger: /etc/xen/scripts/vif-bridge: online XENBUS_PATH=backend/vif/2/0
Jul 2 14:01:10 ols011p logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/2/768
Jul 2 14:01:10 ols011p logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/2/5632
Jul 2 14:01:10 ols011p logger: /etc/xen/scripts/block: Writing backend/vbd/2/5632/hotplug-status connected to xenstore.
Jul 2 14:01:10 ols011p kernel: device vif2.0 entered promiscuous mode
Jul 2 14:01:10 ols011p logger: /etc/xen/scripts/vif-bridge: Successful vif-bridge online for vif2.0, bridge xenbr0.
Jul 2 14:01:10 ols011p logger: /etc/xen/scripts/vif-bridge: Writing backend/vif/2/0/hotplug-status connected to xenstore.
Jul 2 14:01:11 ols011p logger: /etc/xen/scripts/block: Writing backend/vbd/2/768/node /dev/loop0 to xenstore.
Jul 2 14:01:11 ols011p logger: /etc/xen/scripts/block: Writing backend/vbd/2/768/physical-device 7:0 to xenstore.
Jul 2 14:01:11 ols011p logger: /etc/xen/scripts/block: Writing backend/vbd/2/768/hotplug-status connected to xenstore.
Jul 2 14:01:21 ols011p clurgmgrd[8142]: <notice> vm:fusi01-hvm is now running locally
Jul 2 14:04:59 ols011p anacron[5857]: Job `cron.weekly' terminated
Jul 2 14:04:59 ols011p anacron[5857]: Normal exit (2 jobs run)
Jul 2 14:06:40 ols011p ntpd[5747]: synchronized to 172.27.0.2, stratum 3
Jul 2 14:09:02 ols011p sshd[12611]: Accepted publickey for root from 10.10.10.2 port 53945 ssh2
Jul 2 14:09:02 ols011p sshd[12611]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jul 2 14:09:08 ols011p sshd[12611]: pam_unix(sshd:session): session closed for user root
Jul 2 14:10:01 ols011p crond[12776]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Jul 2 14:10:49 ols011p syslog-ng[6692]: Log statistics; dropped='tcp(AF_INET(172.27.19.171:1514))=0', processed='center(queued)=2662', processed='center(received)=1331', processed='destination(d_messages)=1331', processed='destination(server)=1331', processed='source(s_local)=1331'
Jul 2 14:15:01 ols011p crond[13274]: (root) CMD ( [ -f /opt/openpkg/etc/rc ] && /opt/openpkg/etc/rc all quarterly)
Jul 2 14:20:01 ols011p crond[13929]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Jul 2 14:20:49 ols011p syslog-ng[6692]: Log statistics; dropped='tcp(AF_INET(172.27.19.171:1514))=0', processed='center(queued)=2668', processed='center(received)=1334', processed='destination(d_messages)=1334', processed='destination(server)=1334', processed='source(s_local)=1334'
Jul 2 14:21:17 ols011p openais[5259]: [TOTEM] The token was lost in the OPERATIONAL state.
Jul 2 14:21:17 ols011p openais[5259]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes).
Jul 2 14:21:17 ols011p openais[5259]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes).
Jul 2 14:21:17 ols011p openais[5259]: [TOTEM] entering GATHER state from 2.
Jul 2 14:21:18 ols011p qdiskd[5347]: <notice> Writing eviction notice for node 2
Jul 2 14:21:19 ols011p qdiskd[5347]: <notice> Node 2 evicted
Jul 2 14:21:20 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:21:20 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:21:21 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:21:21 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:21:21 ols011p openais[5259]: [TOTEM] entering GATHER state from 0.
Jul 2 14:21:21 ols011p openais[5259]: [TOTEM] Creating commit token because I am the rep.
Jul 2 14:21:21 ols011p openais[5259]: [TOTEM] Saving state aru e8 high seq received e8
Jul 2 14:21:21 ols011p openais[5259]: [TOTEM] Storing new sequence id for ring 4d4
Jul 2 14:21:21 ols011p openais[5259]: [TOTEM] entering COMMIT state.
Jul 2 14:21:21 ols011p openais[5259]: [TOTEM] entering RECOVERY state.
Jul 2 14:21:21 ols011p openais[5259]: [TOTEM] position [0] member 10.10.10.1:
Jul 2 14:21:21 ols011p openais[5259]: [TOTEM] previous ring seq 1232 rep 10.10.10.1
Jul 2 14:21:21 ols011p openais[5259]: [TOTEM] aru e8 high delivered e8 received flag 1
Jul 2 14:21:21 ols011p openais[5259]: [TOTEM] Did not need to originate any messages in recovery.
Jul 2 14:21:21 ols011p openais[5259]: [TOTEM] Sending initial ORF token
Jul 2 14:21:21 ols011p openais[5259]: [CLM ] CLM CONFIGURATION CHANGE
Jul 2 14:21:21 ols011p openais[5259]: [CLM ] New Configuration:
Jul 2 14:21:21 ols011p openais[5259]: [CLM ] r(0) ip(10.10.10.1)
Jul 2 14:21:21 ols011p openais[5259]: [CLM ] Members Left:
Jul 2 14:21:21 ols011p kernel: dlm: closing connection to node 2
Jul 2 14:21:21 ols011p openais[5259]: [CLM ] r(0) ip(10.10.10.2)
Jul 2 14:21:21 ols011p openais[5259]: [CLM ] Members Joined:
Jul 2 14:21:21 ols011p openais[5259]: [CLM ] CLM CONFIGURATION CHANGE
Jul 2 14:21:21 ols011p openais[5259]: [CLM ] New Configuration:
Jul 2 14:21:21 ols011p openais[5259]: [CLM ] r(0) ip(10.10.10.1)
Jul 2 14:21:21 ols011p openais[5259]: [CLM ] Members Left:
Jul 2 14:21:21 ols011p openais[5259]: [CLM ] Members Joined:
Jul 2 14:21:21 ols011p openais[5259]: [SYNC ] This node is within the primary component and will provide service.
Jul 2 14:21:21 ols011p openais[5259]: [TOTEM] entering OPERATIONAL state.
Jul 2 14:21:21 ols011p openais[5259]: [CLM ] got nodejoin message 10.10.10.1
Jul 2 14:21:21 ols011p openais[5259]: [CPG ] got joinlist message from node 1
Jul 2 14:21:22 ols011p kernel: dlm: connect from non cluster node
Jul 2 14:21:22 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:21:22 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:21:23 ols011p openais[5259]: [TOTEM] entering GATHER state from 11.
Jul 2 14:21:23 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:21:23 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:21:24 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:21:24 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:21:24 ols011p fenced[5286]: ols012p not a cluster member after 3 sec post_fail_delay
Jul 2 14:21:24 ols011p fenced[5286]: fencing node "ols012p"
Jul 2 14:21:25 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:21:25 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:21:26 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:21:26 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:21:27 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:21:27 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:21:28 ols011p openais[5259]: [TOTEM] entering GATHER state from 0.
Jul 2 14:21:28 ols011p openais[5259]: [TOTEM] Creating commit token because I am the rep.
Jul 2 14:21:28 ols011p openais[5259]: [TOTEM] Saving state aru 20 high seq received 20
Jul 2 14:21:28 ols011p openais[5259]: [TOTEM] Storing new sequence id for ring 4d8
Jul 2 14:21:28 ols011p openais[5259]: [TOTEM] entering COMMIT state.
Jul 2 14:21:28 ols011p openais[5259]: [TOTEM] entering RECOVERY state.
Jul 2 14:21:28 ols011p openais[5259]: [TOTEM] position [0] member 10.10.10.1:
Jul 2 14:21:28 ols011p openais[5259]: [TOTEM] previous ring seq 1236 rep 10.10.10.1
Jul 2 14:21:28 ols011p openais[5259]: [TOTEM] aru 20 high delivered 20 received flag 1
Jul 2 14:21:28 ols011p openais[5259]: [TOTEM] Did not need to originate any messages in recovery.
Jul 2 14:21:28 ols011p openais[5259]: [TOTEM] Sending initial ORF token
Jul 2 14:21:28 ols011p openais[5259]: [CLM ] CLM CONFIGURATION CHANGE
Jul 2 14:21:28 ols011p openais[5259]: [CLM ] New Configuration:
Jul 2 14:21:28 ols011p openais[5259]: [CLM ] r(0) ip(10.10.10.1)
Jul 2 14:21:28 ols011p openais[5259]: [CLM ] Members Left:
Jul 2 14:21:28 ols011p openais[5259]: [CLM ] Members Joined:
Jul 2 14:21:28 ols011p openais[5259]: [CLM ] CLM CONFIGURATION CHANGE
Jul 2 14:21:28 ols011p openais[5259]: [CLM ] New Configuration:
Jul 2 14:21:28 ols011p openais[5259]: [CLM ] r(0) ip(10.10.10.1)
Jul 2 14:21:28 ols011p openais[5259]: [CLM ] Members Left:
Jul 2 14:21:28 ols011p openais[5259]: [CLM ] Members Joined:
Jul 2 14:21:28 ols011p openais[5259]: [SYNC ] This node is within the primary component and will provide service.
Jul 2 14:21:28 ols011p openais[5259]: [TOTEM] entering OPERATIONAL state.
Jul 2 14:21:28 ols011p openais[5259]: [CLM ] got nodejoin message 10.10.10.1
Jul 2 14:21:28 ols011p openais[5259]: [CPG ] got joinlist message from node 1
Jul 2 14:21:28 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:21:28 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:21:29 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:21:29 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:21:30 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:21:30 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:21:31 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:21:31 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:21:32 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:21:32 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:21:33 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:21:33 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:21:34 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:21:34 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:21:35 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:21:35 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:21:35 ols011p fenced[5286]: fence "ols012p" success
Jul 2 14:21:35 ols011p kernel: GFS2: fsid=fusi01:fusi01hvm.1: jid=0: Trying to acquire journal lock...
Jul 2 14:21:35 ols011p kernel: GFS2: fsid=fusi01:fusi01pvm.1: jid=0: Trying to acquire journal lock...
Jul 2 14:21:35 ols011p kernel: GFS2: fsid=fusi01:fusi01hvm.1: jid=0: Looking at journal...
Jul 2 14:21:35 ols011p kernel: GFS2: fsid=fusi01:fusi01pvm.1: jid=0: Looking at journal...
Jul 2 14:21:35 ols011p kernel: GFS2: fsid=fusi01:fusi01hvm.1: jid=0: Acquiring the transaction lock...
Jul 2 14:21:35 ols011p kernel: GFS2: fsid=fusi01:fusi01hvm.1: jid=0: Replaying journal...
Jul 2 14:21:35 ols011p kernel: GFS2: fsid=fusi01:fusi01hvm.1: jid=0: Replayed 1 of 1 blocks
Jul 2 14:21:35 ols011p kernel: GFS2: fsid=fusi01:fusi01hvm.1: jid=0: Found 0 revoke tags
Jul 2 14:21:35 ols011p kernel: GFS2: fsid=fusi01:fusi01hvm.1: jid=0: Journal replayed in 1s
Jul 2 14:21:35 ols011p kernel: GFS2: fsid=fusi01:fusi01hvm.1: jid=0: Done
Jul 2 14:21:36 ols011p kernel: GFS2: fsid=fusi01:fusi01pvm.1: jid=0: Acquiring the transaction lock...
Jul 2 14:21:36 ols011p kernel: GFS2: fsid=fusi01:fusi01pvm.1: jid=0: Replaying journal...
Jul 2 14:21:36 ols011p kernel: GFS2: fsid=fusi01:fusi01pvm.1: jid=0: Replayed 0 of 0 blocks
Jul 2 14:21:36 ols011p kernel: GFS2: fsid=fusi01:fusi01pvm.1: jid=0: Found 0 revoke tags
Jul 2 14:21:36 ols011p kernel: GFS2: fsid=fusi01:fusi01pvm.1: jid=0: Journal replayed in 1s
Jul 2 14:21:36 ols011p kernel: GFS2: fsid=fusi01:fusi01pvm.1: jid=0: Done
Jul 2 14:21:36 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:21:36 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:21:37 ols011p clurgmgrd[8142]: <notice> Taking over service vm:fusi01-pvm from down member ols012p
Jul 2 14:21:37 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:21:37 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:21:39 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:21:39 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:21:39 ols011p kernel: tap tap-3-51712: 2 getting info
Jul 2 14:21:39 ols011p BLKTAPCTRL[6028]: Received a poll for a new vbd
Jul 2 14:21:39 ols011p BLKTAPCTRL[6028]: Created /dev/xen/blktap1 device
Jul 2 14:21:39 ols011p BLKTAPCTRL[6028]: Received device id 1 and major 253, sent domid 3 and be_id 51712
Jul 2 14:21:39 ols011p BLKTAPCTRL[6028]: Detected handle: [aio]
Jul 2 14:21:39 ols011p BLKTAPCTRL[6028]: Process does not exist:
Jul 2 14:21:39 ols011p BLKTAPCTRL[6028]: Launching process, CMDLINE [tapdisk /dev/xen/tapctrlwrite1 /dev/xen/tapctrlread1]
Jul 2 14:21:39 ols011p BLKTAPCTRL[6028]: Write_msg called: CTLMSG_PID
Jul 2 14:21:39 ols011p BLKTAPCTRL[6028]: Received CTLMSG_PID_RSP
Jul 2 14:21:39 ols011p BLKTAPCTRL[6028]: PID: [14167]
Jul 2 14:21:39 ols011p BLKTAPCTRL[6028]: Write_msg called: CTLMSG_PARAMS, sending [aio:/fusi01pvm/fusi01-pvm_28G.img, /fusi01pvm/fusi01-pvm_28G.img]
Jul 2 14:21:39 ols011p BLKTAPCTRL[6028]: Generated cookie, 0
Jul 2 14:21:39 ols011p TAPDISK[14167]: Tapdisk: Received msg, len 10, type 9, UID 0
Jul 2 14:21:39 ols011p TAPDISK[14167]: Tapdisk: Received msg, len 40, type 1, UID 0
Jul 2 14:21:39 ols011p TAPDISK[14167]: Received CTLMSG_PARAMS: [/fusi01pvm/fusi01-pvm_28G.img]
Jul 2 14:21:39 ols011p TAPDISK[14167]: Loaded driver: name [tapdisk_aio], type [0]
Jul 2 14:21:39 ols011p TAPDISK[14167]: block-aio open('/fusi01pvm/fusi01-pvm_28G.img')
Jul 2 14:21:39 ols011p TAPDISK[14167]: open(/fusi01pvm/fusi01-pvm_28G.img) with O_DIRECT
Jul 2 14:21:39 ols011p TAPDISK[14167]: Image size: pre sector_shift [30064771072] post sector_shift [58720256]
Jul 2 14:21:39 ols011p TAPDISK[14167]: Adding fd_list_entry
Jul 2 14:21:39 ols011p TAPDISK[14167]: Entered cookie 0
Jul 2 14:21:39 ols011p BLKTAPCTRL[6028]: Received CTLMSG_IMG: 58720256, 512, 0
Jul 2 14:21:39 ols011p BLKTAPCTRL[6028]: Received a poll for a new devmap
Jul 2 14:21:39 ols011p BLKTAPCTRL[6028]: Write_msg called: CTLMSG_NEWDEV
Jul 2 14:21:39 ols011p TAPDISK[14167]: Tapdisk: Received msg, len 14, type 4, UID 0
Jul 2 14:21:39 ols011p TAPDISK[14167]: Retrieving state, cookie 0.....[OK]
Jul 2 14:21:39 ols011p BLKTAPCTRL[6028]: Received CTLMSG_NEWDEV_RSP
Jul 2 14:21:39 ols011p BLKTAPCTRL[6028]: Exiting map_new_blktapctrl
Jul 2 14:21:39 ols011p logger: /etc/xen/scripts/blktap: add XENBUS_PATH=backend/tap/3/51712
Jul 2 14:21:39 ols011p logger: /etc/xen/scripts/blktap: add XENBUS_PATH=backend/tap/3/51712
Jul 2 14:21:39 ols011p logger: /etc/xen/scripts/vif-bridge: online XENBUS_PATH=backend/vif/3/0
Jul 2 14:21:39 ols011p logger: /etc/xen/scripts/blktap: Writing backend/tap/3/51712/hotplug-status connected to xenstore.
Jul 2 14:21:39 ols011p kernel: device vif3.0 entered promiscuous mode
Jul 2 14:21:39 ols011p logger: /etc/xen/scripts/vif-bridge: Successful vif-bridge online for vif3.0, bridge xenbr0.
Jul 2 14:21:39 ols011p logger: /etc/xen/scripts/vif-bridge: Writing backend/vif/3/0/hotplug-status connected to xenstore.
Jul 2 14:21:39 ols011p clurgmgrd[8142]: <notice> Service vm:fusi01-pvm started
Jul 2 14:21:40 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:21:40 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:21:41 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:21:41 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:21:42 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:21:42 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:21:43 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:21:43 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:21:44 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:21:44 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:21:45 ols011p kernel: blktap: ring-ref 8, event-channel 29, protocol 1 (x86_64-abi)
Jul 2 14:21:45 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:21:45 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:21:46 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:21:46 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:21:47 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:21:47 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:21:48 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:21:48 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:21:49 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:21:49 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:21:50 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:21:50 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:21:51 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:21:51 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:21:52 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:21:52 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:21:53 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:21:53 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:21:54 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:21:54 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:21:55 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:21:55 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:21:56 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:21:56 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:21:57 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:21:57 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:21:58 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:21:58 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:21:59 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:21:59 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:22:00 ols011p kernel: xenbr0: port 4(vif3.0) entering learning state
Jul 2 14:22:00 ols011p kernel: xenbr0: topology change detected, propagating
Jul 2 14:22:00 ols011p kernel: xenbr0: port 4(vif3.0) entering forwarding state
Jul 2 14:22:00 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:22:00 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:22:01 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:22:01 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:22:02 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:22:02 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:22:03 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:22:03 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:22:04 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:22:04 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:22:05 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:22:05 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:22:06 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:22:06 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:22:07 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:22:07 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:22:08 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:22:08 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:22:09 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:22:09 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:22:10 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:22:10 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:22:11 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:22:11 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:22:12 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:22:12 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:22:13 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:22:13 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:22:14 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:22:14 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:22:15 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:22:15 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:22:16 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:22:16 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:22:17 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:22:17 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:22:18 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:22:18 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:22:19 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:22:19 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:22:20 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:22:20 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:22:21 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:22:21 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:22:22 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:22:22 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:22:23 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:22:23 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:22:24 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:22:24 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:22:25 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:22:25 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:22:26 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:22:26 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:22:27 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:22:27 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:22:28 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:22:28 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:22:29 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:22:29 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:22:30 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:22:30 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:22:31 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:22:31 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:22:32 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:22:32 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:22:33 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:22:33 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:22:34 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:22:34 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:22:35 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:22:35 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:22:36 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:22:36 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:22:37 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:22:37 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:22:38 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:22:38 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:22:39 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:22:39 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:22:40 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:22:40 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:22:41 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:22:41 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:22:42 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:22:42 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:22:43 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:22:43 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:22:44 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:22:44 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:22:45 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:22:45 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:22:46 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:22:46 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:22:47 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:22:47 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:22:48 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:22:48 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:22:49 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:22:49 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:22:50 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:22:50 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:22:51 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:22:51 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:22:52 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:22:52 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:22:53 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:22:53 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:22:54 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:22:54 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:22:55 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:22:55 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:22:56 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:22:56 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:22:57 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:22:57 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:22:58 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:22:58 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:22:59 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:22:59 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:23:00 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:23:00 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:23:01 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:23:01 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:23:02 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:23:02 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:23:03 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:23:03 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:23:04 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:23:04 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:23:05 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:23:05 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:23:06 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:23:06 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:23:07 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:23:07 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:23:08 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:23:08 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:23:09 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:23:09 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:23:10 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:23:10 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:23:11 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:23:11 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:23:12 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:23:12 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:23:13 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:23:13 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:23:14 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:23:14 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:23:15 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:23:15 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:23:16 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:23:16 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:23:17 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:23:17 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:23:18 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:23:18 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:23:19 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:23:19 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:23:20 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:23:20 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:23:21 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:23:21 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:23:22 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:23:22 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:23:23 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:23:23 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:23:24 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:23:24 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:23:25 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:23:25 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:23:26 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:23:26 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:23:27 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:23:27 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:23:28 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:23:28 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:23:29 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:23:29 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:23:30 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:23:30 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:23:31 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:23:31 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:23:32 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:23:32 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:23:33 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:23:33 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:23:34 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:23:34 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:23:35 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:23:35 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:23:36 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:23:36 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:23:37 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:23:37 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:23:38 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:23:38 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:23:39 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:23:39 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:23:40 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:23:40 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:23:41 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:23:41 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:23:42 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:23:42 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:23:43 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:23:43 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:23:44 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:23:44 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:23:45 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:23:45 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:23:46 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:23:46 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:23:47 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:23:47 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:23:48 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:23:48 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:23:49 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:23:49 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:23:50 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:23:50 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:23:51 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:23:51 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:23:52 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:23:52 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:23:53 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:23:53 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:23:54 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:23:54 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:23:55 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:23:55 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:23:56 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:23:56 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:23:57 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:23:57 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:23:58 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:23:58 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:23:59 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:23:59 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:24:00 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:24:00 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:24:01 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:24:01 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:24:02 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:24:02 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:24:03 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:24:03 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:24:04 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:24:04 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:24:05 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:24:05 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:24:06 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:24:06 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:24:07 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:24:07 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:24:08 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:24:08 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:24:09 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:24:09 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:24:10 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:24:10 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:24:11 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:24:11 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:24:12 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:24:12 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:24:13 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:24:13 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:24:14 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:24:14 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:24:15 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:24:15 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:24:16 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:24:16 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:24:17 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:24:17 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:24:18 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:24:18 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:24:19 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:24:19 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:24:20 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:24:20 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:24:21 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:24:21 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:24:22 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:24:22 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:24:23 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:24:23 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:24:24 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:24:24 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:24:25 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:24:25 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:24:26 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:24:26 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:24:27 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:24:27 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:24:28 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:24:28 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:24:29 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:24:29 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:24:30 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:24:30 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:24:31 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:24:31 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:24:32 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:24:32 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:24:33 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:24:33 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:24:34 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:24:34 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:24:35 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:24:35 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:24:36 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:24:36 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:24:37 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:24:37 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:24:38 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:24:38 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:24:39 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:24:39 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:24:40 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:24:40 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:24:41 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:24:41 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:24:42 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:24:42 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:24:43 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:24:43 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:24:44 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:24:44 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:24:45 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:24:45 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:24:46 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:24:46 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:24:47 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:24:47 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:24:48 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:24:48 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:24:49 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:24:49 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:24:50 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:24:50 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:24:51 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:24:51 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:24:52 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:24:52 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:24:53 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:24:53 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:24:54 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:24:54 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:24:55 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:24:55 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:24:56 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:24:56 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:24:57 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:24:57 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:24:58 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:24:58 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:24:59 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:24:59 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:25:00 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:25:00 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:25:01 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:25:01 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:25:02 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:25:02 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:25:03 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:25:03 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:25:04 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:25:04 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:25:05 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:25:05 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:25:06 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:25:06 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:25:07 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:25:07 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:25:08 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:25:08 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:25:09 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:25:09 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:25:10 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:25:10 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:25:11 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:25:11 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:25:12 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:25:12 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:25:13 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:25:13 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:25:14 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:25:14 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:25:15 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:25:15 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:25:16 ols011p openais[5259]: [TOTEM] entering GATHER state from 11.
Jul 2 14:25:16 ols011p openais[5259]: [TOTEM] Creating commit token because I am the rep.
Jul 2 14:25:16 ols011p openais[5259]: [TOTEM] Saving state aru 28 high seq received 28
Jul 2 14:25:16 ols011p openais[5259]: [TOTEM] Storing new sequence id for ring 4dc
Jul 2 14:25:16 ols011p openais[5259]: [TOTEM] entering COMMIT state.
Jul 2 14:25:16 ols011p openais[5259]: [TOTEM] entering RECOVERY state.
Jul 2 14:25:16 ols011p openais[5259]: [TOTEM] position [0] member 10.10.10.1:
Jul 2 14:25:16 ols011p openais[5259]: [TOTEM] previous ring seq 1240 rep 10.10.10.1
Jul 2 14:25:16 ols011p openais[5259]: [TOTEM] aru 28 high delivered 28 received flag 1
Jul 2 14:25:16 ols011p openais[5259]: [TOTEM] position [1] member 10.10.10.2:
Jul 2 14:25:16 ols011p openais[5259]: [TOTEM] previous ring seq 1236 rep 10.10.10.2
Jul 2 14:25:16 ols011p openais[5259]: [TOTEM] aru 9 high delivered 9 received flag 1
Jul 2 14:25:16 ols011p openais[5259]: [TOTEM] Did not need to originate any messages in recovery.
Jul 2 14:25:16 ols011p openais[5259]: [TOTEM] Sending initial ORF token
Jul 2 14:25:16 ols011p openais[5259]: [CLM ] CLM CONFIGURATION CHANGE
Jul 2 14:25:16 ols011p openais[5259]: [CLM ] New Configuration:
Jul 2 14:25:16 ols011p openais[5259]: [CLM ] r(0) ip(10.10.10.1)
Jul 2 14:25:16 ols011p openais[5259]: [CLM ] Members Left:
Jul 2 14:25:16 ols011p openais[5259]: [CLM ] Members Joined:
Jul 2 14:25:16 ols011p openais[5259]: [CLM ] CLM CONFIGURATION CHANGE
Jul 2 14:25:16 ols011p openais[5259]: [CLM ] New Configuration:
Jul 2 14:25:16 ols011p openais[5259]: [CLM ] r(0) ip(10.10.10.1)
Jul 2 14:25:16 ols011p openais[5259]: [CLM ] r(0) ip(10.10.10.2)
Jul 2 14:25:16 ols011p openais[5259]: [CLM ] Members Left:
Jul 2 14:25:16 ols011p openais[5259]: [CLM ] Members Joined:
Jul 2 14:25:16 ols011p openais[5259]: [CLM ] r(0) ip(10.10.10.2)
Jul 2 14:25:16 ols011p openais[5259]: [SYNC ] This node is within the primary component and will provide service.
Jul 2 14:25:16 ols011p openais[5259]: [TOTEM] entering OPERATIONAL state.
Jul 2 14:25:16 ols011p openais[5259]: [CLM ] got nodejoin message 10.10.10.1
Jul 2 14:25:16 ols011p openais[5259]: [CLM ] got nodejoin message 10.10.10.2
Jul 2 14:25:16 ols011p openais[5259]: [CPG ] got joinlist message from node 1
Jul 2 14:25:16 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:25:16 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:25:17 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:25:17 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:25:18 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:25:18 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:25:19 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:25:19 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:25:20 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:25:20 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:25:21 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:25:21 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:25:22 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:25:22 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:25:23 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:25:23 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:25:24 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:25:24 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:25:25 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:25:25 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:25:26 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:25:26 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:25:26 ols011p openais[5259]: [TOTEM] The token was lost in the OPERATIONAL state.
Jul 2 14:25:26 ols011p openais[5259]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes).
Jul 2 14:25:26 ols011p openais[5259]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes).
Jul 2 14:25:26 ols011p openais[5259]: [TOTEM] entering GATHER state from 2.
Jul 2 14:25:27 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:25:27 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:25:28 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:25:28 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:25:29 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:25:29 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:25:30 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:25:30 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:25:31 ols011p openais[5259]: [TOTEM] entering GATHER state from 0.
Jul 2 14:25:31 ols011p openais[5259]: [TOTEM] Creating commit token because I am the rep.
Jul 2 14:25:31 ols011p openais[5259]: [TOTEM] Saving state aru 22 high seq received 22
Jul 2 14:25:31 ols011p openais[5259]: [TOTEM] Storing new sequence id for ring 4e0
Jul 2 14:25:31 ols011p openais[5259]: [TOTEM] entering COMMIT state.
Jul 2 14:25:31 ols011p openais[5259]: [TOTEM] entering RECOVERY state.
Jul 2 14:25:31 ols011p openais[5259]: [TOTEM] position [0] member 10.10.10.1:
Jul 2 14:25:31 ols011p openais[5259]: [TOTEM] previous ring seq 1244 rep 10.10.10.1
Jul 2 14:25:31 ols011p openais[5259]: [TOTEM] aru 22 high delivered 22 received flag 1
Jul 2 14:25:31 ols011p openais[5259]: [TOTEM] Did not need to originate any messages in recovery.
Jul 2 14:25:31 ols011p openais[5259]: [TOTEM] Sending initial ORF token
Jul 2 14:25:31 ols011p openais[5259]: [CLM ] CLM CONFIGURATION CHANGE
Jul 2 14:25:31 ols011p openais[5259]: [CLM ] New Configuration:
Jul 2 14:25:31 ols011p openais[5259]: [CLM ] r(0) ip(10.10.10.1)
Jul 2 14:25:31 ols011p openais[5259]: [CLM ] Members Left:
Jul 2 14:25:31 ols011p openais[5259]: [CLM ] r(0) ip(10.10.10.2)
Jul 2 14:25:31 ols011p openais[5259]: [CLM ] Members Joined:
Jul 2 14:25:31 ols011p openais[5259]: [CLM ] CLM CONFIGURATION CHANGE
Jul 2 14:25:31 ols011p openais[5259]: [CLM ] New Configuration:
Jul 2 14:25:31 ols011p openais[5259]: [CLM ] r(0) ip(10.10.10.1)
Jul 2 14:25:31 ols011p openais[5259]: [CLM ] Members Left:
Jul 2 14:25:31 ols011p openais[5259]: [CLM ] Members Joined:
Jul 2 14:25:31 ols011p openais[5259]: [SYNC ] This node is within the primary component and will provide service.
Jul 2 14:25:31 ols011p openais[5259]: [TOTEM] entering OPERATIONAL state.
Jul 2 14:25:31 ols011p openais[5259]: [CLM ] got nodejoin message 10.10.10.1
Jul 2 14:25:31 ols011p kernel: dlm: closing connection to node 2
Jul 2 14:25:31 ols011p openais[5259]: [CPG ] got joinlist message from node 1
Jul 2 14:25:31 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:25:31 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:25:32 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:25:32 ols011p qdiskd[5347]: <alert> Writing eviction notice for node 2
Jul 2 14:25:33 ols011p qdiskd[5347]: <crit> Node 2 is undead.
Jul 2 14:25:33 ols011p qdiskd[5347]: <alert> Writing eviction n
Jul 2 14:00:43 ols012p syslog-ng[6652]: Log statistics; dropped='tcp(AF_INET(172.27.19.171:1514))=0', processed='center(queued)=3004', processed='center(received)=1502', processed='destination(d_messages)=1502', processed='destination(server)=1502', processed='source(s_local)=1502'
Jul 2 14:00:47 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:01:01 ols012p crond[26261]: (root) CMD (run-parts /etc/cron.hourly)
Jul 2 14:01:02 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:01:10 ols012p clurgmgrd[8117]: <notice> Migrating vm:fusi01-hvm to ols011p
Jul 2 14:01:17 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:01:32 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:01:47 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:02:02 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:02:17 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:02:32 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:02:47 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:03:02 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:03:17 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:03:32 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:03:47 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:04:02 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:04:17 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:04:32 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:04:47 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:04:55 ols012p anacron[5876]: Job `cron.weekly' terminated
Jul 2 14:04:55 ols012p anacron[5876]: Normal exit (2 jobs run)
Jul 2 14:05:02 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:05:17 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:05:32 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:05:47 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:06:02 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:06:17 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:06:32 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:06:47 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:07:02 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:07:17 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:07:32 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:07:47 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:08:02 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:08:17 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:08:32 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:08:47 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:09:02 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:09:17 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:09:32 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:09:47 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:10:01 ols012p crond[14315]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Jul 2 14:10:02 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:10:17 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:10:32 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:10:43 ols012p syslog-ng[6652]: Log statistics; dropped='tcp(AF_INET(172.27.19.171:1514))=0', processed='center(queued)=3096', processed='center(received)=1548', processed='destination(d_messages)=1548', processed='destination(server)=1548', processed='source(s_local)=1548'
Jul 2 14:10:47 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:11:02 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:11:17 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:11:32 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:11:47 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:12:02 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:12:17 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:12:32 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:12:47 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
^[[6~Jul 2 14:13:02 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:13:17 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:13:32 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:13:47 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:14:02 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:14:17 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:14:32 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:14:47 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:15:01 ols012p crond[14817]: (root) CMD ( [ -f /opt/openpkg/etc/rc ] && /opt/openpkg/etc/rc all quarterly)
Jul 2 14:15:02 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:15:17 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:15:32 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:15:47 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:16:02 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:16:17 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:16:32 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:16:47 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:17:02 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:17:17 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:17:32 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:17:47 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:18:02 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:18:17 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:18:32 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:18:47 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:19:02 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:19:17 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:19:32 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:19:47 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:20:01 ols012p crond[15498]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Jul 2 14:20:02 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:20:17 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:20:32 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:20:43 ols012p syslog-ng[6652]: Log statistics; dropped='tcp(AF_INET(172.27.19.171:1514))=0', processed='center(queued)=3182', processed='center(received)=1591', processed='destination(d_messages)=1591', processed='destination(server)=1591', processed='source(s_local)=1591'
Jul 2 14:20:47 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:21:02 ols012p snmpd[8092]: netsnmp_assert index == tmp failed if-mib/data_access/interface.c:469 _access_interface_entry_save_name()
Jul 2 14:21:06 ols012p kernel: xenbr0: port 3(tap0) entering disabled state
Jul 2 14:21:06 ols012p kernel: device tap0 left promiscuous mode
Jul 2 14:21:06 ols012p kernel: xenbr0: port 3(tap0) entering disabled state
Jul 2 14:21:22 ols012p kernel: BUG: soft lockup - CPU#0 stuck for 15s! [python:28225]
Jul 2 14:21:22 ols012p kernel: CPU 0:
Jul 2 14:21:22 ols012p kernel: Modules linked in: xt_physdev iptable_filter ip_tables x_tables tun blktap blkbk autofs4 hidp l2cap bluetooth lock_dlm gfs2(U) dlm configfs bridge netloop netbk sunrpc bonding loop dm_round_robin dm_multipath video sbs backlight i2c_ec button battery asus_acpi ac parport_pc lp parport joydev serial_core sr_mod pcspkr sg i2c_nforce2 i2c_core k8_edac edac_mc shpchp ide_cd k8temp hwmon cdrom serio_raw e1000 dm_snapshot dm_zero dm_mirror dm_mod usb_storage qla2xxx scsi_transport_fc mptsas mptscsih mptbase scsi_transport_sas sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
Jul 2 14:21:22 ols012p kernel: Pid: 28225, comm: python Tainted: G 2.6.18-92.1.1.el5xen #1
Jul 2 14:21:22 ols012p kernel: RIP: e030:[<ffffffff8020648a>] [<ffffffff8020648a>] hypercall_page+0x48a/0x1000
Jul 2 14:21:22 ols012p kernel: RSP: e02b:ffff880007597e50 EFLAGS: 00000282
Jul 2 14:21:22 ols012p kernel: RAX: 0000000000000000 RBX: ffff88003c0ed1c0 RCX: ffffffff8020648a
Jul 2 14:21:22 ols012p kernel: RDX: 000000000a9c5b30 RSI: 00002b46aacf4710 RDI: 0000000045922530
Jul 2 14:21:22 ols012p kernel: RBP: 0000000045922500 R08: 0000003c198a86fa R09: 0000000000000100
Jul 2 14:21:22 ols012p kernel: R10: 0000000000000001 R11: 0000000000000282 R12: 0000000000305000
Jul 2 14:21:22 ols012p kernel: R13: 0000000000000007 R14: 0000000000000007 R15: 0000003c06d4dd58
Jul 2 14:21:22 ols012p kernel: FS: 00002b431014bed0(0063) GS:ffffffff805af000(0000) knlGS:0000000000000000
Jul 2 14:21:22 ols012p kernel: CS: e033 DS: 0000 ES: 0000
Jul 2 14:21:22 ols012p kernel:
Jul 2 14:21:22 ols012p kernel: Call Trace:
Jul 2 14:21:22 ols012p kernel: [<ffffffff803a481b>] privcmd_ioctl+0x80/0x31f
Jul 2 14:21:22 ols012p kernel: [<ffffffff80243a93>] do_ioctl+0x55/0x6b
Jul 2 14:21:22 ols012p kernel: [<ffffffff8023159d>] vfs_ioctl+0x248/0x261
Jul 2 14:21:22 ols012p kernel: [<ffffffff8024df10>] sys_ioctl+0x59/0x78
Jul 2 14:21:22 ols012p kernel: [<ffffffff802602f1>] tracesys+0xa7/0xb2
Jul 2 14:21:22 ols012p kernel:
Jul 2 14:21:22 ols012p kernel: dlm: connecting to 1
Jul 2 14:21:22 ols012p openais[5272]: [TOTEM] The token was lost in the OPERATIONAL state.
Jul 2 14:21:22 ols012p kernel: md: stopping all md devices.
Jul 2 14:21:23 ols012p kernel: Synchronizing SCSI cache for disk sde:
Jul 2 14:21:23 ols012p kernel: Synchronizing SCSI cache for disk sdd:
Jul 2 14:21:23 ols012p kernel: Synchronizing SCSI cache for disk sdc:
Jul 2 14:21:23 ols012p openais[5272]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes).
Jul 2 14:21:23 ols012p openais[5272]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes).
Jul 2 14:21:23 ols012p kernel: Synchronizing SCSI cache for disk sdb:
Connection to ols012 closed.
--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster
