1. cluster.conf : Please find attached herewith.
2. Is fencing in any form mandatory for a setup as simple as this ? I am just using two redhat5 VM's within VMware fusion in my Mac. What kind of fencing is applicable to this kind of setup, that can also benefit me ? All that i want to see from this PoC is to add some services like Apache, MySQL to see how quick and reliable the cluster to pick the service failures etc.,
3. Please find attached the messages from server1 and server2 when i did "service cman stop and start" on both nodes one after the other. In server1 both stop and start went fine but clustat showed "Could not connect to CMAN: connection refused". If i do "cman_tool join" on both nodes one after the other, things look green
Also, Am i missing any rpm's that are most important ? I used yum group install "Clustering" and "ClusterStorage" to install all the packages.
-Param
On Fri, Aug 24, 2012 at 7:28 PM, Digimer <lists@xxxxxxxxxx> wrote:
A few things;
1. Please repost your cluster.conf file with line wraps in plain text.
2. Manual fencing is not supported in any way, please use real fencing,
like IPMI, iLO, etc.
3. Please stop the cluster entirely, start 'tail -f -n 0
/var/log/messages' on both nodes, then start cman, then start rgmanager.
Please share the output from the logs.
Digimer
> apache ccsd.pid ccsd.sock rgmanager.sk <http://rgmanager.sk>
On 08/24/2012 06:43 AM, PARAM KRISH wrote:
> Hi, Thanks for the help. I hope we are nearing to the problem.
>
> I enabled logging , this is how my cluster.conf looks like
>
> <?xml version="1.0"?>
> <cluster alias="newCluster" config_version="16" name="newCluster">
> <logging debug="on"/>
> <cman expected_votes="1" two_node="1"/>
> <clusternodes>
> <clusternode name="server1" nodeid="1" votes="1">
> <fence><method name="single"><device
> name="human"/></method></fence></clusternode><clusternode name="server2"
> nodeid="2" votes="1"><fence><method name="single"><device
> name="human"/></method></fence></clusternode></clusternodes><fencedevices>
>
> </fencedevices><rm><failoverdomains><failoverdomain
> name="failOver" nofailback="0" ordered="1"
> restricted="0"><failoverdomainnode name="server1"
> priority="1"/><failoverdomainnode name="server2"
> priority="2"/></failoverdomain></failoverdomains><resources><ip
> address="192.168.61.130" monitor_link="1"/><apache
> config_file="conf/httpd.conf" name="httpd" server_root="/etc/httpd"
> shutdown_wait="0"/></resources><service autostart="1" domain="failOver"
> exclusive="1" name="Apache" recovery="relocate"><ip
> address="192.168.61.130" monitor_link="1"><apache
> config_file="conf/httpd.conf" name="Apache" server_root="/etc/httpd"
> shutdown_wait="0"/></ip></service><service autostart="1"
> domain="failOver" exclusive="1" name="website" recovery="relocate"><ip
> ref="192.168.61.130"><apache
> ref="httpd"/></ip></service></rm><fence_daemon clean_start="1"
> post_fail_delay="0" post_join_delay="3"/><logging debug="on"/></cluster>
>
> There is no logging happening in /var/run/cluster/
>
> [root@server1 ~]# ls /var/run/cluster/
>
> I started resource manager in foreground and it says like ..
>
> failed acquiring lockspace: No such device
> Locks not working!
>
> What next i could do ?
>
> -Param
>
> On Fri, Aug 24, 2012 at 3:18 PM, emmanuel segura <emi2fast@xxxxxxxxx
> <mailto:emi2fast@xxxxxxxxx>> wrote:> <mailto:heiko.nardmann@xxxxxxxxxxxxx>>
>
> /etc/init.d/rgmanager start or service rgmanager start
>
>
> 2012/8/24 Heiko Nardmann <heiko.nardmann@xxxxxxxxxxxxx
>> /var/run/cluster/rgmanager.sk <http://rgmanager.sk> is missing.
> It is strange that strace shows that
>> rgmanager.sk <http://rgmanager.sk> is missing ...
> Normally it is helpful to see the complete cluster.conf. Could
> you provide that one?
>
> Also of interest is /var/log/cluster/rgmanager.log - do you have
> debug enabled inside cluster.conf?
>
> Maybe it is possible to start rgmanager in the foreground (-f)
> with strace? That might also be a way to show why the
>> Am 24.08.2012 11 <tel:24.08.2012%2011>:04, schrieb PARAM KRISH:
> Just some ideas ...
>
>
> Kind regards,
>
> Heiko
>
> path="/var/run/cluster/rgmanag__er.sk <http://rgmanager.sk>>
> All,
>
> I am trying to setup a simple two node cluster in my laptop
> using two RHEL VM's.
>
> Everything looks just fine to me but i am unable to enable a
> apache service though it works beautifully when tried with
> "rg_test test" on both the nodes.
>
> What could be the problem ? Please help. I am a novice in
> red hat cluster but learnt a bit of it in the last few days
> while trying to fix all the problems encountered.
>
> Here are the details.
>
> [root@server1 ~]# clustat
> Cluster Status for newCluster @ Thu Aug 23 00:29:32 2012
> Member Status: Quorate
>
> Member Name ID Status
> ------ ---- ---- ------
> server1 1 Online, Local
> server2 2 Online
>
> [root@server1 ~]# clustat -x
> <?xml version="1.0"?>
> <clustat version="4.1.1">
> <cluster name="newCluster" id="43188" generation="250536"/>
> <quorum quorate="1" groupmember="0"/>
> <nodes>
> <node name="server1" state="1" local="1" estranged="0"
> rgmanager="0" rgmanager_master="0" qdisk="0"
> nodeid="0x00000001"/>
> <node name="server2" state="1" local="0" estranged="0"
> rgmanager="0" rgmanager_master="0" qdisk="0"
> nodeid="0x00000002"/> </nodes>
> </clustat>
>
> [root@server2 ~]# clustat
> Cluster Status for newCluster @ Thu Aug 23 03:13:34 2012
> Member Status: Quorate
>
> Member Name ID Status
> ------ ---- ---- ------
> server1 1 Online
> server2 2 Online, Local
>
> [root@server2 ~]# clustat -x
> <?xml version="1.0"?>
> <clustat version="4.1.1">
> <cluster name="newCluster" id="43188" generation="250536"/>
> <quorum quorate="1" groupmember="0"/>
> <nodes>
> <node name="server1" state="1" local="0" estranged="0"
> rgmanager="0" rgmanager_master="0" qdisk="0"
> nodeid="0x00000001"/>
> <node name="server2" state="1" local="1" estranged="0"
> rgmanager="0" rgmanager_master="0" qdisk="0"
> nodeid="0x00000002"/>
> </nodes>
> </clustat>
>
>
> [root@server2 ~]# clusvcadm -e Apache
> Local machine trying to enable service:Apache...Could not
> connect to resource group manager
>
> strace cluvcsadm -e Apache
> ...
> stat64(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 4),
> ...}) = 0
> mmap2(NULL, 4096, PROT_READ|PROT_WRITE,
> MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7fb5000
> write(1, "Local machine trying to enable s"..., 48Local
> machine trying to enable service:Apache...) = 48
> socket(PF_FILE, SOCK_STREAM, 0) = 5
> connect(5, {sa_family=AF_FILE,
> Linux-cluster@xxxxxxxxxx <mailto:Linux-cluster@xxxxxxxxxx>> <http://rgmanager.sk>"...}, 110) = -1 ENOENT (No such file
> or directory)
>
> close(5) = 0
> write(1, "Could not connect to resource gr"..., 44Could not
> connect to resource group manager
> ) = 44
> exit_group(1) = ?
>
>
> [root@server1 ~]# hostname
> server1.localdomain
>
> [root@server1 ~]# cat /etc/hosts
> # Do not remove the following line, or various programs
> # that require network functionality will fail.
> #127.0.0.1 server1.localdomain server1
> localhost.localdomain localhost
> 192.168.61.132 server1.localdomain server1
> 192.168.61.133 server2.localdomain server2
> ::1 localhost6.localdomain6 localhost6
>
>
> Package versions :
> luci-0.12.2-24.el5
> ricci-0.12.2-24.el5
> rgmanager-2.0.52-9.el5
> modcluster-0.12.1-2.el5
> cluster-cim-0.12.1-2.el5
> system-config-cluster-1.0.57-7
> lvm2-cluster-2.02.74-3.el5
> cluster-snmp-0.12.1-2.el5
>
> [root@server1 log]# cman_tool status
> Version: 6.2.0
> Config Version: 15
> Cluster Name: newCluster
> Cluster Id: 43188
> Cluster Member: Yes
> Cluster Generation: 250536
> Membership state: Cluster-Member
> Nodes: 2
> Expected votes: 1
> Total votes: 2
> Quorum: 1
> Active subsystems: 2
> Flags: 2node
> Ports Bound: 0
> Node name: server1
> Node ID: 1
> Multicast addresses: 239.192.168.93
> Node addresses: 192.168.61.132
>
> Redhat :Red Hat Enterprise Linux Server release 5.6
> (Tikanga)2.6.18-238.el5xen
>
> [root@server1 log]# service rgmanager status
> clurgmgrd (pid 9775) is running...
>
> [root@server1 log]# netstat -na | grep 11111
> tcp 0 0 0.0.0.0:11111 <http://0.0.0.0:11111>
> <http://0.0.0.0:11111> 0.0.0.0:*
> LISTEN
>
>
> Please let me know if you can help. One thing i noticed was
> that in the "clustat" it does not show "rgmanager" against
> both the nodes but i see the service is just running fine.
>
> *Note : No iptables, no SELinux enabled.*
> *
>
> *
> Hope i have given all the details required to help me
> quickly. Thanks.
>
> -Param
>
>
>
> --
> Linux-cluster mailing list
> https://www.redhat.com/__mailman/listinfo/linux-cluster
> <https://www.redhat.com/mailman/listinfo/linux-cluster>> Linux-cluster@xxxxxxxxxx <mailto:Linux-cluster@xxxxxxxxxx>
>
>
>
>
> --
> esta es mi vida e me la vivo hasta que dios quiera
>
> --
> Linux-cluster mailing list
> https://www.redhat.com/mailman/listinfo/linux-cluster
Digimer>
>
>
>
> --
> Linux-cluster mailing list
> Linux-cluster@xxxxxxxxxx
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
--
Papers and Projects: https://alteeve.ca
Aug 23 07:22:21 server2 kernel: eth1: link down Aug 23 07:22:21 server2 kernel: eth0: link down Aug 23 07:22:22 server2 avahi-daemon[3607]: Invalid query packet. Aug 23 07:22:47 server2 last message repeated 15 times Aug 23 07:22:53 server2 kernel: eth1: link up Aug 23 07:22:53 server2 kernel: eth0: link up Aug 23 07:22:56 server2 avahi-daemon[3607]: Invalid query packet. Aug 23 07:23:39 server2 last message repeated 2 times Aug 23 07:26:10 server2 openais[16727]: [TOTEM] entering GATHER state from 11. Aug 23 07:26:10 server2 openais[16727]: [TOTEM] Saving state aru d high seq received d Aug 23 07:26:10 server2 openais[16727]: [TOTEM] Storing new sequence id for ring 3d304 Aug 23 07:26:10 server2 openais[16727]: [TOTEM] entering COMMIT state. Aug 23 07:26:10 server2 openais[16727]: [TOTEM] entering RECOVERY state. Aug 23 07:26:10 server2 openais[16727]: [TOTEM] position [0] member 192.168.61.132: Aug 23 07:26:10 server2 openais[16727]: [TOTEM] previous ring seq 250624 rep 192.168.61.132 Aug 23 07:26:10 server2 openais[16727]: [TOTEM] aru c high delivered c received flag 1 Aug 23 07:26:10 server2 openais[16727]: [TOTEM] position [1] member 192.168.61.133: Aug 23 07:26:10 server2 openais[16727]: [TOTEM] previous ring seq 250624 rep 192.168.61.133 Aug 23 07:26:10 server2 openais[16727]: [TOTEM] aru d high delivered d received flag 1 Aug 23 07:26:10 server2 openais[16727]: [TOTEM] Did not need to originate any messages in recovery. Aug 23 07:26:10 server2 openais[16727]: [CLM ] CLM CONFIGURATION CHANGE Aug 23 07:26:10 server2 openais[16727]: [CLM ] New Configuration: Aug 23 07:26:11 server2 openais[16727]: [CLM ] r(0) ip(192.168.61.133) Aug 23 07:26:11 server2 openais[16727]: [CLM ] Members Left: Aug 23 07:26:11 server2 openais[16727]: [CLM ] Members Joined: Aug 23 07:26:11 server2 openais[16727]: [CLM ] CLM CONFIGURATION CHANGE Aug 23 07:26:11 server2 openais[16727]: [CLM ] New Configuration: Aug 23 07:26:11 server2 openais[16727]: [CLM ] r(0) ip(192.168.61.132) Aug 23 07:26:11 server2 openais[16727]: [CLM ] r(0) ip(192.168.61.133) Aug 23 07:26:11 server2 openais[16727]: [CLM ] Members Left: Aug 23 07:26:11 server2 openais[16727]: [CLM ] Members Joined: Aug 23 07:26:11 server2 openais[16727]: [CLM ] r(0) ip(192.168.61.132) Aug 23 07:26:11 server2 openais[16727]: [SYNC ] This node is within the primary component and will provide service. Aug 23 07:26:11 server2 openais[16727]: [TOTEM] entering OPERATIONAL state. Aug 23 07:26:11 server2 openais[16727]: [CLM ] got nodejoin message 192.168.61.132 Aug 23 07:26:11 server2 openais[16727]: [CLM ] got nodejoin message 192.168.61.133 Aug 23 07:34:12 server2 ccsd[16559]: Stopping ccsd, SIGTERM received. Aug 23 07:34:12 server2 NAMC Aug 23 07:34:12 server2 openais[16727]: [SERV ] Unloading all openais components Aug 23 07:34:12 server2 openais[16727]: [SERV ] Unloading openais component: openais_confdb v0 (19/10) Aug 23 07:34:12 server2 openais[16727]: [SERV ] Unloading openais component: openais_cpg v0 (18/8) Aug 23 07:34:12 server2 openais[16727]: [SERV ] Unloading openais component: openais_cfg v0 (17/7) Aug 23 07:34:12 server2 openais[16727]: [SERV ] Unloading openais component: openais_msg v0 (16/6) Aug 23 07:34:12 server2 openais[16727]: [SERV ] Unloading openais component: openais_lck v0 (15/5) Aug 23 07:34:12 server2 openais[16727]: [SERV ] Unloading openais component: openais_evt v0 (14/4) Aug 23 07:34:12 server2 openais[16727]: [SERV ] Unloading openais component: openais_ckpt v0 (13/3) Aug 23 07:34:12 server2 openais[16727]: [SERV ] Unloading openais component: openais_amf v0 (12/2) Aug 23 07:34:12 server2 openais[16727]: [SERV ] Unloading openais component: openais_clm v0 (11/1) Aug 23 07:34:12 server2 openais[16727]: [SERV ] Unloading openais component: openais_evs v0 (10/0) Aug 23 07:34:12 server2 openais[16727]: [SERV ] Unloading openais component: openais_cman v0 (9/9) Aug 23 07:34:12 server2 openais[16727]: [SERV ] AIS Executive exiting (reason: CMAN requested shutdown). Aug 23 07:34:29 server2 ccsd[19167]: Starting ccsd 2.0.115: Aug 23 07:34:29 server2 ccsd[19167]: Built: Dec 10 2010 11:26:52 Aug 23 07:34:29 server2 ccsd[19167]: Copyright (C) Red Hat, Inc. 2004 All rights reserved. Aug 23 07:34:29 server2 ccsd[19167]: cluster.conf (cluster name = newCluster, version = 16) found. Aug 23 07:34:29 server2 ccsd[19167]: Unable to sendto broadcast ipv4 socket, but inet_ntop returned NULL pointer: Network is unreachable Aug 23 07:34:57 server2 last message repeated 16 times Aug 23 07:34:58 server2 ccsd[19167]: Unable to connect to cluster infrastructure after 30 seconds. Aug 23 07:34:59 server2 ccsd[19167]: Unable to sendto broadcast ipv4 socket, but inet_ntop returned NULL pointer: Network is unreachable Aug 23 07:35:09 server2 last message repeated 6 times Aug 23 07:35:09 server2 openais[19177]: [MAIN ] AIS Executive Service RELEASE 'subrev 1887 version 0.80.6' Aug 23 07:35:09 server2 openais[19177]: [MAIN ] Copyright (C) 2002-2006 MontaVista Software, Inc and contributors. Aug 23 07:35:09 server2 openais[19177]: [MAIN ] Copyright (C) 2006 Red Hat, Inc. Aug 23 07:35:09 server2 openais[19177]: [MAIN ] AIS Executive Service: started and ready to provide service. Aug 23 07:35:09 server2 openais[19177]: [MAIN ] Using default multicast address of 239.192.168.93 Aug 23 07:35:09 server2 ccsd[19167]: Initial status:: Quorate Aug 23 07:35:09 server2 groupd[19197]: found uncontrolled kernel object rgmanager in /sys/kernel/dlm Aug 23 07:35:09 server2 groupd[19197]: local node must be reset to clear 1 uncontrolled instances of gfs and/or dlm Aug 23 07:35:09 server2 fence_node[19198]: Fence of "server2" was unsuccessful Aug 23 07:35:09 server2 openais[19177]: [TOTEM] Token Timeout (10000 ms) retransmit timeout (495 ms) Aug 23 07:35:09 server2 openais[19177]: [TOTEM] token hold (386 ms) retransmits before loss (20 retrans) Aug 23 07:35:09 server2 openais[19177]: [TOTEM] join (60 ms) send_join (0 ms) consensus (2000 ms) merge (200 ms) Aug 23 07:35:09 server2 openais[19177]: [TOTEM] downcheck (1000 ms) fail to recv const (2500 msgs) Aug 23 07:35:09 server2 openais[19177]: [TOTEM] seqno unchanged const (30 rotations) Maximum network MTU 1402 Aug 23 07:35:09 server2 openais[19177]: [TOTEM] window size per rotation (50 messages) maximum messages per rotation (17 messages) Aug 23 07:35:09 server2 openais[19177]: [TOTEM] missed count const (5 messages) Aug 23 07:35:09 server2 openais[19177]: [TOTEM] send threads (0 threads) Aug 23 07:35:09 server2 openais[19177]: [TOTEM] RRP token expired timeout (495 ms) Aug 23 07:35:09 server2 openais[19177]: [TOTEM] RRP token problem counter (2000 ms) Aug 23 07:35:09 server2 openais[19177]: [TOTEM] RRP threshold (10 problem count) Aug 23 07:35:09 server2 openais[19177]: [TOTEM] RRP mode set to none. Aug 23 07:35:09 server2 openais[19177]: [TOTEM] heartbeat_failures_allowed (0) Aug 23 07:35:09 server2 openais[19177]: [TOTEM] max_network_delay (50 ms) Aug 23 07:35:09 server2 openais[19177]: [TOTEM] HeartBeat is Disabled. To enable set heartbeat_failures_allowed > 0 Aug 23 07:35:09 server2 openais[19177]: [TOTEM] Receive multicast socket recv buffer size (320000 bytes). Aug 23 07:35:09 server2 openais[19177]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). Aug 23 07:35:09 server2 openais[19177]: [TOTEM] The network interface [192.168.61.133] is now up. Aug 23 07:35:09 server2 openais[19177]: [TOTEM] Created or loaded sequence id 250628.192.168.61.133 for this ring. Aug 23 07:35:09 server2 openais[19177]: [TOTEM] entering GATHER state from 15. Aug 23 07:35:09 server2 openais[19177]: [CMAN ] CMAN 2.0.115 (built Dec 10 2010 11:26:55) started Aug 23 07:35:09 server2 openais[19177]: [MAIN ] Service initialized 'openais CMAN membership service 2.01' Aug 23 07:35:09 server2 openais[19177]: [SERV ] Service initialized 'openais extended virtual synchrony service' Aug 23 07:35:09 server2 openais[19177]: [SERV ] Service initialized 'openais cluster membership service B.01.01' Aug 23 07:35:09 server2 openais[19177]: [SERV ] Service initialized 'openais availability management framework B.01.01' Aug 23 07:35:09 server2 openais[19177]: [SERV ] Service initialized 'openais checkpoint service B.01.01' Aug 23 07:35:09 server2 openais[19177]: [SERV ] Service initialized 'openais event service B.01.01' Aug 23 07:35:09 server2 openais[19177]: [SERV ] Service initialized 'openais distributed locking service B.01.01' Aug 23 07:35:09 server2 openais[19177]: [SERV ] Service initialized 'openais message service B.01.01' Aug 23 07:35:09 server2 openais[19177]: [SERV ] Service initialized 'openais configuration service' Aug 23 07:35:09 server2 openais[19177]: [SERV ] Service initialized 'openais cluster closed process group service v1.01' Aug 23 07:35:09 server2 openais[19177]: [SERV ] Service initialized 'openais cluster config database access v1.01' Aug 23 07:35:09 server2 openais[19177]: [SYNC ] Not using a virtual synchrony filter. Aug 23 07:35:09 server2 openais[19177]: [TOTEM] Creating commit token because I am the rep. Aug 23 07:35:09 server2 openais[19177]: [TOTEM] Saving state aru 0 high seq received 0 Aug 23 07:35:09 server2 openais[19177]: [TOTEM] Storing new sequence id for ring 3d308 Aug 23 07:35:09 server2 openais[19177]: [TOTEM] entering COMMIT state. Aug 23 07:35:09 server2 openais[19177]: [TOTEM] entering RECOVERY state. Aug 23 07:35:09 server2 openais[19177]: [TOTEM] position [0] member 192.168.61.133: Aug 23 07:35:09 server2 openais[19177]: [TOTEM] previous ring seq 250628 rep 192.168.61.133 Aug 23 07:35:09 server2 openais[19177]: [TOTEM] aru 0 high delivered 0 received flag 1 Aug 23 07:35:09 server2 openais[19177]: [TOTEM] Did not need to originate any messages in recovery. Aug 23 07:35:09 server2 openais[19177]: [TOTEM] Sending initial ORF token Aug 23 07:35:09 server2 openais[19177]: [CLM ] CLM CONFIGURATION CHANGE Aug 23 07:35:09 server2 openais[19177]: [CLM ] New Configuration: Aug 23 07:35:09 server2 openais[19177]: [CLM ] Members Left: Aug 23 07:35:09 server2 openais[19177]: [CLM ] Members Joined: Aug 23 07:35:09 server2 openais[19177]: [CLM ] CLM CONFIGURATION CHANGE Aug 23 07:35:09 server2 openais[19177]: [CLM ] New Configuration: Aug 23 07:35:09 server2 openais[19177]: [CLM ] r(0) ip(192.168.61.133) Aug 23 07:35:09 server2 openais[19177]: [CLM ] Members Left: Aug 23 07:35:09 server2 openais[19177]: [CLM ] Members Joined: Aug 23 07:35:09 server2 openais[19177]: [CLM ] r(0) ip(192.168.61.133) Aug 23 07:35:09 server2 openais[19177]: [SYNC ] This node is within the primary component and will provide service. Aug 23 07:35:09 server2 openais[19177]: [TOTEM] entering OPERATIONAL state. Aug 23 07:35:09 server2 openais[19177]: [CMAN ] quorum regained, resuming activity Aug 23 07:35:09 server2 openais[19177]: [CLM ] got nodejoin message 192.168.61.133 Aug 23 07:35:09 server2 openais[19177]: [CMAN ] cman killed by node 2 because we were killed by cman_tool or other application Aug 23 07:35:09 server2 openais[19177]: [TOTEM] entering GATHER state from 11. Aug 23 07:35:09 server2 openais[19177]: [TOTEM] Saving state aru d high seq received d Aug 23 07:35:09 server2 openais[19177]: [TOTEM] Storing new sequence id for ring 3d30c Aug 23 07:35:09 server2 openais[19177]: [TOTEM] entering COMMIT state. Aug 23 07:35:09 server2 openais[19177]: [TOTEM] entering RECOVERY state. Aug 23 07:35:09 server2 openais[19177]: [TOTEM] position [0] member 192.168.61.132: Aug 23 07:35:09 server2 openais[19177]: [TOTEM] previous ring seq 250632 rep 192.168.61.132 Aug 23 07:35:09 server2 openais[19177]: [TOTEM] aru d high delivered d received flag 1 Aug 23 07:35:09 server2 openais[19177]: [TOTEM] position [1] member 192.168.61.133: Aug 23 07:35:09 server2 openais[19177]: [TOTEM] previous ring seq 250632 rep 192.168.61.133 Aug 23 07:35:09 server2 openais[19177]: [TOTEM] aru d high delivered d received flag 1 Aug 23 07:35:09 server2 openais[19177]: [TOTEM] Did not need to originate any messages in recovery. Aug 23 07:35:09 server2 openais[19177]: [CLM ] CLM CONFIGURATION CHANGE Aug 23 07:35:09 server2 openais[19177]: [CLM ] New Configuration: Aug 23 07:35:09 server2 openais[19177]: [CLM ] r(0) ip(192.168.61.133) Aug 23 07:35:09 server2 openais[19177]: [CLM ] Members Left: Aug 23 07:35:09 server2 openais[19177]: [CLM ] Members Joined: Aug 23 07:35:09 server2 openais[19177]: [CLM ] CLM CONFIGURATION CHANGE Aug 23 07:35:09 server2 openais[19177]: [CLM ] New Configuration: Aug 23 07:35:09 server2 openais[19177]: [CLM ] r(0) ip(192.168.61.132) Aug 23 07:35:09 server2 openais[19177]: [CLM ] r(0) ip(192.168.61.133) Aug 23 07:35:09 server2 openais[19177]: [CLM ] Members Left: Aug 23 07:35:09 server2 openais[19177]: [CLM ] Members Joined: Aug 23 07:35:09 server2 openais[19177]: [CLM ] r(0) ip(192.168.61.132) Aug 23 07:35:09 server2 openais[19177]: [SYNC ] This node is within the primary component and will provide service. Aug 23 07:35:09 server2 openais[19177]: [TOTEM] entering OPERATIONAL state. Aug 23 07:35:09 server2 openais[19177]: [CLM ] got nodejoin message 192.168.61.132 Aug 23 07:35:09 server2 openais[19177]: [CLM ] got nodejoin message 192.168.61.133 Aug 23 07:35:10 server2 openais[19177]: [SERV ] Unloading all openais components Aug 23 07:35:10 server2 openais[19177]: [SERV ] Unloading openais component: openais_confdb v0 (19/10) Aug 23 07:35:10 server2 openais[19177]: [SERV ] Unloading openais component: openais_cpg v0 (18/8) Aug 23 07:35:10 server2 openais[19177]: [SERV ] Unloading openais component: openais_cfg v0 (17/7) Aug 23 07:35:10 server2 openais[19177]: [SERV ] Unloading openais component: openais_msg v0 (16/6) Aug 23 07:35:10 server2 openais[19177]: [SERV ] Unloading openais component: openais_lck v0 (15/5) Aug 23 07:35:10 server2 openais[19177]: [SERV ] Unloading openais component: openais_evt v0 (14/4) Aug 23 07:35:10 server2 openais[19177]: [SERV ] Unloading openais component: openais_ckpt v0 (13/3) Aug 23 07:35:10 server2 openais[19177]: [SERV ] Unloading openais component: openais_amf v0 (12/2) Aug 23 07:35:10 server2 openais[19177]: [SERV ] Unloading openais component: openais_clm v0 (11/1) Aug 23 07:35:10 server2 openais[19177]: [SERV ] Unloading openais component: openais_evs v0 (10/0) Aug 23 07:35:10 server2 openais[19177]: [SERV ] Unloading openais component: openais_cman v0 (9/9) Aug 23 07:35:10 server2 openais[19177]: [SERV ] AIS Executive exiting (reason: CMAN kill requested, exiting). Aug 23 07:35:19 server2 dlm_controld[19210]: group_init error (nil) 111 Aug 23 07:35:19 server2 gfs_controld[19216]: group_init error (nil) 111 Aug 23 07:35:19 server2 fenced[19204]: group_init error (nil) 111 Aug 23 07:35:29 server2 ccsd[19167]: Unable to connect to cluster infrastructure after 60 seconds. Aug 23 07:35:59 server2 ccsd[19167]: Unable to connect to cluster infrastructure after 90 seconds. Aug 23 07:36:30 server2 ccsd[19167]: Unable to connect to cluster infrastructure after 120 seconds. Aug 23 07:37:00 server2 ccsd[19167]: Unable to connect to cluster infrastructure after 150 seconds. Aug 23 07:37:30 server2 ccsd[19167]: Unable to connect to cluster infrastructure after 180 seconds. Aug 23 07:38:00 server2 ccsd[19167]: Unable to connect to cluster infrastructure after 210 seconds. Aug 23 07:38:30 server2 ccsd[19167]: Unable to connect to cluster infrastructure after 240 seconds. Aug 23 07:39:00 server2 ccsd[19167]: Unable to connect to cluster infrastructure after 270 seconds. Aug 23 07:39:30 server2 ccsd[19167]: Unable to connect to cluster infrastructure after 300 seconds. Aug 23 07:40:00 server2 ccsd[19167]: Unable to connect to cluster infrastructure after 330 seconds. Aug 23 07:40:30 server2 ccsd[19167]: Unable to connect to cluster infrastructure after 360 seconds. Aug 23 07:41:01 server2 ccsd[19167]: Unable to connect to cluster infrastructure after 390 seconds. Aug 23 07:41:31 server2 ccsd[19167]: Unable to connect to cluster infrastructure after 420 seconds. Aug 23 07:42:01 server2 ccsd[19167]: Unable to connect to cluster infrastructure after 450 seconds. Aug 23 07:42:31 server2 ccsd[19167]: Unable to connect to cluster infrastructure after 480 seconds. Aug 23 07:43:01 server2 ccsd[19167]: Unable to connect to cluster infrastructure after 510 seconds. Aug 23 07:43:31 server2 ccsd[19167]: Unable to connect to cluster infrastructure after 540 seconds. Aug 23 07:44:01 server2 ccsd[19167]: Unable to connect to cluster infrastructure after 570 seconds. Aug 23 07:44:31 server2 ccsd[19167]: Unable to connect to cluster infrastructure after 600 seconds. Aug 23 07:45:02 server2 ccsd[19167]: Unable to connect to cluster infrastructure after 630 seconds. Aug 23 07:45:32 server2 ccsd[19167]: Unable to connect to cluster infrastructure after 660 seconds. Aug 23 07:46:02 server2 ccsd[19167]: Unable to connect to cluster infrastructure after 690 seconds. Aug 23 07:46:32 server2 ccsd[19167]: Unable to connect to cluster infrastructure after 720 seconds. Aug 23 07:47:02 server2 ccsd[19167]: Unable to connect to cluster infrastructure after 750 seconds. Aug 23 07:47:32 server2 ccsd[19167]: Unable to connect to cluster infrastructure after 780 seconds.
Aug 23 02:12:19 server1 avahi-daemon[3412]: Invalid query packet. Aug 23 02:14:49 server1 kernel: Kernel logging (proc) stopped. Aug 23 02:14:49 server1 kernel: Kernel log daemon terminating. Aug 23 02:14:51 server1 exiting on signal 15 Aug 23 02:14:51 server1 syslogd 1.4.1: restart. Aug 23 02:14:51 server1 kernel: klogd 1.4.1, log source = /proc/kmsg started. Aug 23 02:15:28 server1 rgmanager: [24865]: <notice> Shutting down Cluster Service Manager... Aug 23 02:21:14 server1 groupd[25130]: found uncontrolled kernel object rgmanager in /sys/kernel/dlm Aug 23 02:21:14 server1 groupd[25130]: local node must be reset to clear 1 uncontrolled instances of gfs and/or dlm Aug 23 02:21:14 server1 openais[24514]: [CMAN ] cman killed by node 1 because we were killed by cman_tool or other application Aug 23 02:21:14 server1 fence_node[25131]: Fence of "server1" was unsuccessful Aug 23 02:21:24 server1 dlm_controld[25143]: group_init error (nil) 111 Aug 23 02:21:24 server1 gfs_controld[25149]: group_init error (nil) 111 Aug 23 02:21:24 server1 fenced[25137]: group_init error (nil) 111 Aug 23 02:21:36 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 120 seconds. Aug 23 02:22:06 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 150 seconds. Aug 23 02:22:36 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 180 seconds. Aug 23 02:23:06 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 210 seconds. Aug 23 02:23:36 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 240 seconds. Aug 23 02:24:06 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 270 seconds. Aug 23 02:24:37 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 300 seconds. Aug 23 02:25:07 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 330 seconds. Aug 23 02:25:37 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 360 seconds. Aug 23 02:26:07 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 390 seconds. Aug 23 02:26:37 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 420 seconds. Aug 23 02:27:07 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 450 seconds. Aug 23 02:27:37 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 480 seconds. Aug 23 02:28:07 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 510 seconds. Aug 23 02:28:38 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 540 seconds. Aug 23 02:29:08 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 570 seconds. Aug 23 02:29:38 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 600 seconds. Aug 23 02:30:08 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 630 seconds. Aug 23 02:30:38 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 660 seconds. Aug 23 02:31:08 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 690 seconds. Aug 23 02:31:38 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 720 seconds. Aug 23 02:32:08 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 750 seconds. Aug 23 02:32:38 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 780 seconds. Aug 23 02:33:09 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 810 seconds. Aug 23 02:33:39 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 840 seconds. Aug 23 02:34:09 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 870 seconds. Aug 23 02:34:39 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 900 seconds. Aug 23 02:35:09 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 930 seconds. Aug 23 02:35:39 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 960 seconds. Aug 23 02:36:09 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 990 seconds. Aug 23 02:36:39 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 1020 seconds. Aug 23 02:37:10 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 1050 seconds. Aug 23 02:37:40 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 1080 seconds. Aug 23 02:38:10 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 1110 seconds. Aug 23 02:38:40 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 1140 seconds. Aug 23 02:39:10 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 1170 seconds. Aug 23 02:39:40 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 1200 seconds. Aug 23 02:40:10 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 1230 seconds. Aug 23 02:40:40 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 1260 seconds. Aug 23 02:41:10 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 1290 seconds. Aug 23 02:41:41 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 1320 seconds. Aug 23 02:42:11 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 1350 seconds. Aug 23 02:42:41 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 1380 seconds. Aug 23 02:43:11 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 1410 seconds. Aug 23 02:43:41 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 1440 seconds. Aug 23 02:44:11 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 1470 seconds. Aug 23 02:44:41 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 1500 seconds. Aug 23 02:45:11 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 1530 seconds. Aug 23 02:45:42 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 1560 seconds. Aug 23 02:46:12 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 1590 seconds. Aug 23 02:46:42 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 1620 seconds. Aug 23 02:47:12 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 1650 seconds. Aug 23 02:47:42 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 1680 seconds. Aug 23 02:48:12 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 1710 seconds. Aug 23 02:48:42 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 1740 seconds. Aug 23 02:49:12 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 1770 seconds. Aug 23 02:49:42 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 1800 seconds. Aug 23 02:50:13 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 1830 seconds. Aug 23 02:50:43 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 1860 seconds. Aug 23 02:51:13 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 1890 seconds. Aug 23 02:51:43 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 1920 seconds. Aug 23 02:52:13 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 1950 seconds. Aug 23 02:52:43 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 1980 seconds. Aug 23 02:53:13 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 2010 seconds. Aug 23 02:53:43 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 2040 seconds. Aug 23 02:54:14 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 2070 seconds. Aug 23 02:54:44 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 2100 seconds. Aug 23 02:55:14 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 2130 seconds. Aug 23 02:55:44 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 2160 seconds. Aug 23 02:56:14 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 2190 seconds. Aug 23 02:56:44 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 2220 seconds. Aug 23 02:57:14 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 2250 seconds. Aug 23 02:57:44 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 2280 seconds. Aug 23 02:58:14 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 2310 seconds. Aug 23 02:58:45 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 2340 seconds. Aug 23 02:59:15 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 2370 seconds. Aug 23 02:59:45 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 2400 seconds. Aug 23 03:00:15 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 2430 seconds. Aug 23 03:00:45 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 2460 seconds. Aug 23 03:01:15 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 2490 seconds. Aug 23 03:01:45 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 2520 seconds. Aug 23 03:02:15 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 2550 seconds. Aug 23 03:02:46 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 2580 seconds. Aug 23 03:03:16 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 2610 seconds. Aug 23 03:03:46 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 2640 seconds. Aug 23 03:04:16 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 2670 seconds. Aug 23 03:04:46 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 2700 seconds. Aug 23 03:05:16 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 2730 seconds. Aug 23 03:05:46 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 2760 seconds. Aug 23 03:06:16 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 2790 seconds. Aug 23 03:06:46 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 2820 seconds. Aug 23 03:07:17 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 2850 seconds. Aug 23 03:07:47 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 2880 seconds. Aug 23 03:08:17 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 2910 seconds. Aug 23 03:08:47 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 2940 seconds. Aug 23 03:09:17 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 2970 seconds. Aug 23 03:09:47 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 3000 seconds. Aug 23 03:10:17 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 3030 seconds. Aug 23 03:10:47 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 3060 seconds. Aug 23 03:11:18 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 3090 seconds. Aug 23 03:11:48 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 3120 seconds. Aug 23 03:12:18 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 3150 seconds. Aug 23 03:12:48 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 3180 seconds. Aug 23 03:13:18 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 3210 seconds. Aug 23 03:13:48 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 3240 seconds. Aug 23 03:14:18 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 3270 seconds. Aug 23 03:14:48 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 3300 seconds. Aug 23 03:15:18 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 3330 seconds. Aug 23 03:15:49 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 3360 seconds. Aug 23 03:16:19 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 3390 seconds. Aug 23 03:16:49 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 3420 seconds. Aug 23 03:17:19 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 3450 seconds. Aug 23 03:17:49 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 3480 seconds. Aug 23 03:18:19 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 3510 seconds. Aug 23 03:18:49 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 3540 seconds. Aug 23 03:19:19 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 3570 seconds. Aug 23 03:19:50 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 3600 seconds. Aug 23 03:20:20 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 3630 seconds. Aug 23 03:20:50 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 3660 seconds. Aug 23 03:21:20 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 3690 seconds. Aug 23 03:21:50 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 3720 seconds. Aug 23 03:22:20 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 3750 seconds. Aug 23 03:22:50 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 3780 seconds. Aug 23 03:23:20 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 3810 seconds. Aug 23 03:23:51 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 3840 seconds. Aug 23 03:24:21 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 3870 seconds. Aug 23 03:24:51 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 3900 seconds. Aug 23 03:25:21 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 3930 seconds. Aug 23 03:25:51 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 3960 seconds. Aug 23 03:26:21 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 3990 seconds. Aug 23 03:26:51 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 4020 seconds. Aug 23 03:27:21 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 4050 seconds. Aug 23 03:27:51 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 4080 seconds. Aug 23 03:28:22 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 4110 seconds. Aug 23 03:28:52 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 4140 seconds. Aug 23 03:29:22 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 4170 seconds. Aug 23 03:29:52 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 4200 seconds. Aug 23 03:30:22 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 4230 seconds. Aug 23 03:30:52 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 4260 seconds. Aug 23 03:31:22 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 4290 seconds. Aug 23 03:31:52 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 4320 seconds. Aug 23 03:32:23 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 4350 seconds. Aug 23 03:32:53 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 4380 seconds. Aug 23 03:33:23 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 4410 seconds. Aug 23 03:33:53 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 4440 seconds. Aug 23 03:34:23 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 4470 seconds. Aug 23 03:34:53 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 4500 seconds. Aug 23 03:35:23 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 4530 seconds. Aug 23 03:35:53 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 4560 seconds. Aug 23 03:36:23 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 4590 seconds. Aug 23 03:36:54 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 4620 seconds. Aug 23 03:37:24 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 4650 seconds. Aug 23 03:37:54 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 4680 seconds. Aug 23 03:38:24 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 4710 seconds. Aug 23 03:38:54 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 4740 seconds. Aug 23 03:39:24 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 4770 seconds. Aug 23 03:39:54 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 4800 seconds. Aug 23 03:40:24 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 4830 seconds. Aug 23 03:40:55 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 4860 seconds. Aug 23 03:41:25 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 4890 seconds. Aug 23 03:41:55 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 4920 seconds. Aug 23 03:42:25 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 4950 seconds. Aug 23 03:42:55 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 4980 seconds. Aug 23 03:43:25 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 5010 seconds. Aug 23 03:43:55 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 5040 seconds. Aug 23 03:44:25 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 5070 seconds. Aug 23 03:44:55 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 5100 seconds. Aug 23 03:45:26 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 5130 seconds. Aug 23 03:45:56 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 5160 seconds. Aug 23 03:46:26 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 5190 seconds. Aug 23 03:46:56 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 5220 seconds. Aug 23 03:47:26 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 5250 seconds. Aug 23 03:47:56 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 5280 seconds. Aug 23 03:48:26 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 5310 seconds. Aug 23 03:48:56 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 5340 seconds. Aug 23 03:49:26 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 5370 seconds. Aug 23 03:49:57 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 5400 seconds. Aug 23 03:50:27 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 5430 seconds. Aug 23 03:50:57 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 5460 seconds. Aug 23 03:51:27 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 5490 seconds. Aug 23 03:51:57 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 5520 seconds. Aug 23 03:52:27 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 5550 seconds. Aug 23 03:52:57 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 5580 seconds. Aug 23 03:53:27 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 5610 seconds. Aug 23 03:53:58 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 5640 seconds. Aug 23 03:54:28 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 5670 seconds. Aug 23 03:54:58 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 5700 seconds. Aug 23 03:55:28 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 5730 seconds. Aug 23 03:55:58 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 5760 seconds. Aug 23 03:56:28 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 5790 seconds. Aug 23 03:56:58 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 5820 seconds. Aug 23 03:57:28 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 5850 seconds. Aug 23 03:57:58 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 5880 seconds. Aug 23 03:58:29 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 5910 seconds. Aug 23 03:58:59 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 5940 seconds. Aug 23 03:59:29 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 5970 seconds. Aug 23 03:59:59 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 6000 seconds. Aug 23 04:00:29 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 6030 seconds. Aug 23 04:00:59 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 6060 seconds. Aug 23 04:01:29 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 6090 seconds. Aug 23 04:01:59 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 6120 seconds. Aug 23 04:02:30 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 6150 seconds. Aug 23 04:03:00 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 6180 seconds. Aug 23 04:03:30 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 6210 seconds. Aug 23 04:04:00 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 6240 seconds. Aug 23 04:04:30 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 6270 seconds. Aug 23 04:05:00 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 6300 seconds. Aug 23 04:05:30 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 6330 seconds. Aug 23 04:06:00 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 6360 seconds. Aug 23 04:06:30 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 6390 seconds. Aug 23 04:07:01 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 6420 seconds. Aug 23 04:07:31 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 6450 seconds. Aug 23 04:08:01 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 6480 seconds. Aug 23 04:08:31 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 6510 seconds. Aug 23 04:09:01 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 6540 seconds. Aug 23 04:09:31 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 6570 seconds. Aug 23 04:10:01 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 6600 seconds. Aug 23 04:10:31 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 6630 seconds. Aug 23 04:11:02 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 6660 seconds. Aug 23 04:11:32 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 6690 seconds. Aug 23 04:12:02 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 6720 seconds. Aug 23 04:12:32 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 6750 seconds. Aug 23 04:13:02 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 6780 seconds. Aug 23 04:13:32 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 6810 seconds. Aug 23 04:14:02 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 6840 seconds. Aug 23 04:14:32 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 6870 seconds. Aug 23 04:15:02 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 6900 seconds. Aug 23 04:15:33 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 6930 seconds. Aug 23 04:16:03 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 6960 seconds. Aug 23 04:16:33 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 6990 seconds. Aug 23 04:17:03 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 7020 seconds. Aug 23 04:17:33 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 7050 seconds. Aug 23 04:18:03 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 7080 seconds. Aug 23 04:18:33 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 7110 seconds. Aug 23 04:19:03 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 7140 seconds. Aug 23 04:19:34 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 7170 seconds. Aug 23 04:20:04 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 7200 seconds. Aug 23 04:20:34 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 7230 seconds. Aug 23 04:21:04 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 7260 seconds. Aug 23 04:21:34 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 7290 seconds. Aug 23 04:22:04 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 7320 seconds. Aug 23 04:22:34 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 7350 seconds. Aug 23 04:23:04 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 7380 seconds. Aug 23 04:23:34 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 7410 seconds. Aug 23 04:24:05 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 7440 seconds. Aug 23 04:24:35 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 7470 seconds. Aug 23 04:25:05 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 7500 seconds. Aug 23 04:25:35 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 7530 seconds. Aug 23 04:26:05 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 7560 seconds. Aug 23 04:26:35 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 7590 seconds. Aug 23 04:27:05 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 7620 seconds. Aug 23 04:27:35 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 7650 seconds. Aug 23 04:28:06 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 7680 seconds. Aug 23 04:28:36 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 7710 seconds. Aug 23 04:29:06 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 7740 seconds. Aug 23 04:29:36 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 7770 seconds. Aug 23 04:30:06 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 7800 seconds. Aug 23 04:30:36 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 7830 seconds. Aug 23 04:31:06 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 7860 seconds. Aug 23 04:31:36 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 7890 seconds. Aug 23 04:32:06 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 7920 seconds. Aug 23 04:32:37 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 7950 seconds. Aug 23 04:33:07 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 7980 seconds. Aug 23 04:33:37 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 8010 seconds. Aug 23 04:34:07 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 8040 seconds. Aug 23 04:34:37 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 8070 seconds. Aug 23 04:35:07 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 8100 seconds. Aug 23 04:35:37 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 8130 seconds. Aug 23 04:36:07 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 8160 seconds. Aug 23 04:36:38 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 8190 seconds. Aug 23 04:37:08 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 8220 seconds. Aug 23 04:37:38 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 8250 seconds. Aug 23 04:38:08 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 8280 seconds. Aug 23 04:38:38 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 8310 seconds. Aug 23 04:38:54 server1 kernel: eth1: link down Aug 23 04:38:54 server1 kernel: eth0: link down Aug 23 04:39:08 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 8340 seconds. Aug 23 04:39:26 server1 kernel: eth1: link up Aug 23 04:39:26 server1 kernel: eth0: link up Aug 23 04:39:38 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 8370 seconds. Aug 23 04:40:08 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 8400 seconds. Aug 23 04:40:39 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 8430 seconds. Aug 23 04:41:09 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 8460 seconds. Aug 23 04:41:39 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 8490 seconds. Aug 23 04:42:09 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 8520 seconds. Aug 23 04:42:39 server1 ccsd[24441]: Unable to connect to cluster infrastructure after 8550 seconds. Aug 23 04:42:39 server1 openais[27441]: [MAIN ] AIS Executive Service RELEASE 'subrev 1887 version 0.80.6' Aug 23 04:42:39 server1 openais[27441]: [MAIN ] Copyright (C) 2002-2006 MontaVista Software, Inc and contributors. Aug 23 04:42:39 server1 openais[27441]: [MAIN ] Copyright (C) 2006 Red Hat, Inc. Aug 23 04:42:39 server1 openais[27441]: [MAIN ] AIS Executive Service: started and ready to provide service. Aug 23 04:42:39 server1 openais[27441]: [MAIN ] Using default multicast address of 239.192.168.93 Aug 23 04:42:40 server1 openais[27441]: [TOTEM] Token Timeout (10000 ms) retransmit timeout (495 ms) Aug 23 04:42:40 server1 openais[27441]: [TOTEM] token hold (386 ms) retransmits before loss (20 retrans) Aug 23 04:42:40 server1 openais[27441]: [TOTEM] join (60 ms) send_join (0 ms) consensus (2000 ms) merge (200 ms) Aug 23 04:42:40 server1 openais[27441]: [TOTEM] downcheck (1000 ms) fail to recv const (2500 msgs) Aug 23 04:42:40 server1 openais[27441]: [TOTEM] seqno unchanged const (30 rotations) Maximum network MTU 1402 Aug 23 04:42:40 server1 openais[27441]: [TOTEM] window size per rotation (50 messages) maximum messages per rotation (17 messages) Aug 23 04:42:40 server1 openais[27441]: [TOTEM] missed count const (5 messages) Aug 23 04:42:40 server1 openais[27441]: [TOTEM] send threads (0 threads) Aug 23 04:42:40 server1 openais[27441]: [TOTEM] RRP token expired timeout (495 ms) Aug 23 04:42:40 server1 openais[27441]: [TOTEM] RRP token problem counter (2000 ms) Aug 23 04:42:40 server1 openais[27441]: [TOTEM] RRP threshold (10 problem count) Aug 23 04:42:40 server1 openais[27441]: [TOTEM] RRP mode set to none. Aug 23 04:42:40 server1 openais[27441]: [TOTEM] heartbeat_failures_allowed (0) Aug 23 04:42:40 server1 openais[27441]: [TOTEM] max_network_delay (50 ms) Aug 23 04:42:40 server1 openais[27441]: [TOTEM] HeartBeat is Disabled. To enable set heartbeat_failures_allowed > 0 Aug 23 04:42:40 server1 openais[27441]: [TOTEM] Receive multicast socket recv buffer size (320000 bytes). Aug 23 04:42:40 server1 openais[27441]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). Aug 23 04:42:40 server1 openais[27441]: [TOTEM] The network interface [192.168.61.132] is now up. Aug 23 04:42:40 server1 openais[27441]: [TOTEM] Created or loaded sequence id 250620.192.168.61.132 for this ring. Aug 23 04:42:40 server1 openais[27441]: [TOTEM] entering GATHER state from 15. Aug 23 04:42:40 server1 openais[27441]: [CMAN ] CMAN 2.0.115 (built Dec 10 2010 11:26:55) started Aug 23 04:42:40 server1 openais[27441]: [MAIN ] Service initialized 'openais CMAN membership service 2.01' Aug 23 04:42:40 server1 openais[27441]: [SERV ] Service initialized 'openais extended virtual synchrony service' Aug 23 04:42:40 server1 openais[27441]: [SERV ] Service initialized 'openais cluster membership service B.01.01' Aug 23 04:42:40 server1 openais[27441]: [SERV ] Service initialized 'openais availability management framework B.01.01' Aug 23 04:42:40 server1 openais[27441]: [SERV ] Service initialized 'openais checkpoint service B.01.01' Aug 23 04:42:40 server1 openais[27441]: [SERV ] Service initialized 'openais event service B.01.01' Aug 23 04:42:40 server1 openais[27441]: [SERV ] Service initialized 'openais distributed locking service B.01.01' Aug 23 04:42:40 server1 openais[27441]: [SERV ] Service initialized 'openais message service B.01.01' Aug 23 04:42:40 server1 openais[27441]: [SERV ] Service initialized 'openais configuration service' Aug 23 04:42:40 server1 openais[27441]: [SERV ] Service initialized 'openais cluster closed process group service v1.01' Aug 23 04:42:40 server1 openais[27441]: [SERV ] Service initialized 'openais cluster config database access v1.01' Aug 23 04:42:40 server1 openais[27441]: [SYNC ] Not using a virtual synchrony filter. Aug 23 04:42:40 server1 openais[27441]: [TOTEM] Creating commit token because I am the rep. Aug 23 04:42:40 server1 openais[27441]: [TOTEM] Saving state aru 0 high seq received 0 Aug 23 04:42:40 server1 openais[27441]: [TOTEM] Storing new sequence id for ring 3d300 Aug 23 04:42:40 server1 openais[27441]: [TOTEM] entering COMMIT state. Aug 23 04:42:40 server1 openais[27441]: [TOTEM] entering RECOVERY state. Aug 23 04:42:40 server1 openais[27441]: [TOTEM] position [0] member 192.168.61.132: Aug 23 04:42:40 server1 openais[27441]: [TOTEM] previous ring seq 250620 rep 192.168.61.132 Aug 23 04:42:40 server1 openais[27441]: [TOTEM] aru 0 high delivered 0 received flag 1 Aug 23 04:42:40 server1 openais[27441]: [TOTEM] Did not need to originate any messages in recovery. Aug 23 04:42:40 server1 openais[27441]: [TOTEM] Sending initial ORF token Aug 23 04:42:40 server1 openais[27441]: [CLM ] CLM CONFIGURATION CHANGE Aug 23 04:42:40 server1 openais[27441]: [CLM ] New Configuration: Aug 23 04:42:40 server1 openais[27441]: [CLM ] Members Left: Aug 23 04:42:40 server1 openais[27441]: [CLM ] Members Joined: Aug 23 04:42:40 server1 openais[27441]: [CLM ] CLM CONFIGURATION CHANGE Aug 23 04:42:40 server1 openais[27441]: [CLM ] New Configuration: Aug 23 04:42:40 server1 openais[27441]: [CLM ] r(0) ip(192.168.61.132) Aug 23 04:42:40 server1 openais[27441]: [CLM ] Members Left: Aug 23 04:42:40 server1 openais[27441]: [CLM ] Members Joined: Aug 23 04:42:40 server1 openais[27441]: [CLM ] r(0) ip(192.168.61.132) Aug 23 04:42:40 server1 openais[27441]: [SYNC ] This node is within the primary component and will provide service. Aug 23 04:42:40 server1 openais[27441]: [TOTEM] entering OPERATIONAL state. Aug 23 04:42:40 server1 openais[27441]: [CMAN ] quorum regained, resuming activity Aug 23 04:42:40 server1 openais[27441]: [CLM ] got nodejoin message 192.168.61.132 Aug 23 04:42:40 server1 openais[27441]: [TOTEM] entering GATHER state from 11. Aug 23 04:42:40 server1 openais[27441]: [TOTEM] Creating commit token because I am the rep. Aug 23 04:42:40 server1 openais[27441]: [TOTEM] Saving state aru c high seq received c Aug 23 04:42:40 server1 openais[27441]: [TOTEM] Storing new sequence id for ring 3d304 Aug 23 04:42:40 server1 openais[27441]: [TOTEM] entering COMMIT state. Aug 23 04:42:40 server1 openais[27441]: [TOTEM] entering RECOVERY state. Aug 23 04:42:40 server1 openais[27441]: [TOTEM] position [0] member 192.168.61.132: Aug 23 04:42:40 server1 openais[27441]: [TOTEM] previous ring seq 250624 rep 192.168.61.132 Aug 23 04:42:40 server1 openais[27441]: [TOTEM] aru c high delivered c received flag 1 Aug 23 04:42:40 server1 openais[27441]: [TOTEM] position [1] member 192.168.61.133: Aug 23 04:42:40 server1 openais[27441]: [TOTEM] previous ring seq 250624 rep 192.168.61.133 Aug 23 04:42:40 server1 openais[27441]: [TOTEM] aru d high delivered d received flag 1 Aug 23 04:42:40 server1 openais[27441]: [TOTEM] Did not need to originate any messages in recovery. Aug 23 04:42:40 server1 openais[27441]: [TOTEM] Sending initial ORF token Aug 23 04:42:40 server1 openais[27441]: [CLM ] CLM CONFIGURATION CHANGE Aug 23 04:42:40 server1 openais[27441]: [CLM ] New Configuration: Aug 23 04:42:40 server1 openais[27441]: [CLM ] r(0) ip(192.168.61.132) Aug 23 04:42:40 server1 openais[27441]: [CLM ] Members Left: Aug 23 04:42:40 server1 openais[27441]: [CLM ] Members Joined: Aug 23 04:42:40 server1 openais[27441]: [CLM ] CLM CONFIGURATION CHANGE Aug 23 04:42:40 server1 openais[27441]: [CLM ] New Configuration: Aug 23 04:42:40 server1 openais[27441]: [CLM ] r(0) ip(192.168.61.132) Aug 23 04:42:40 server1 openais[27441]: [CLM ] r(0) ip(192.168.61.133) Aug 23 04:42:40 server1 openais[27441]: [CLM ] Members Left: Aug 23 04:42:40 server1 openais[27441]: [CLM ] Members Joined: Aug 23 04:42:40 server1 openais[27441]: [CLM ] r(0) ip(192.168.61.133) Aug 23 04:42:40 server1 ccsd[24441]: Initial status:: Quorate Aug 23 04:42:40 server1 openais[27441]: [SYNC ] This node is within the primary component and will provide service. Aug 23 04:42:40 server1 openais[27441]: [TOTEM] entering OPERATIONAL state. Aug 23 04:42:40 server1 openais[27441]: [CLM ] got nodejoin message 192.168.61.132 Aug 23 04:42:40 server1 openais[27441]: [CLM ] got nodejoin message 192.168.61.133 Aug 23 04:50:53 server1 openais[27441]: [TOTEM] The token was lost in the OPERATIONAL state. Aug 23 04:50:53 server1 openais[27441]: [TOTEM] Receive multicast socket recv buffer size (320000 bytes). Aug 23 04:50:53 server1 openais[27441]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). Aug 23 04:50:53 server1 openais[27441]: [TOTEM] entering GATHER state from 2. Aug 23 04:50:55 server1 openais[27441]: [TOTEM] entering GATHER state from 0. Aug 23 04:50:55 server1 openais[27441]: [TOTEM] Creating commit token because I am the rep. Aug 23 04:50:55 server1 openais[27441]: [TOTEM] Saving state aru 1c high seq received 1c Aug 23 04:50:55 server1 openais[27441]: [TOTEM] Storing new sequence id for ring 3d308 Aug 23 04:50:55 server1 openais[27441]: [TOTEM] entering COMMIT state. Aug 23 04:50:55 server1 openais[27441]: [TOTEM] entering RECOVERY state. Aug 23 04:50:55 server1 openais[27441]: [TOTEM] position [0] member 192.168.61.132: Aug 23 04:50:55 server1 openais[27441]: [TOTEM] previous ring seq 250628 rep 192.168.61.132 Aug 23 04:50:55 server1 openais[27441]: [TOTEM] aru 1c high delivered 1c received flag 1 Aug 23 04:50:55 server1 openais[27441]: [TOTEM] Did not need to originate any messages in recovery. Aug 23 04:50:55 server1 openais[27441]: [TOTEM] Sending initial ORF token Aug 23 04:50:55 server1 openais[27441]: [CLM ] CLM CONFIGURATION CHANGE Aug 23 04:50:55 server1 openais[27441]: [CLM ] New Configuration: Aug 23 04:50:55 server1 openais[27441]: [CLM ] r(0) ip(192.168.61.132) Aug 23 04:50:55 server1 openais[27441]: [CLM ] Members Left: Aug 23 04:50:55 server1 openais[27441]: [CLM ] r(0) ip(192.168.61.133) Aug 23 04:50:55 server1 openais[27441]: [CLM ] Members Joined: Aug 23 04:50:55 server1 openais[27441]: [CLM ] CLM CONFIGURATION CHANGE Aug 23 04:50:55 server1 openais[27441]: [CLM ] New Configuration: Aug 23 04:50:55 server1 openais[27441]: [CLM ] r(0) ip(192.168.61.132) Aug 23 04:50:55 server1 openais[27441]: [CLM ] Members Left: Aug 23 04:50:55 server1 openais[27441]: [CLM ] Members Joined: Aug 23 04:50:55 server1 openais[27441]: [SYNC ] This node is within the primary component and will provide service. Aug 23 04:50:55 server1 openais[27441]: [TOTEM] entering OPERATIONAL state. Aug 23 04:50:55 server1 openais[27441]: [CLM ] got nodejoin message 192.168.61.132 Aug 23 04:51:39 server1 openais[27441]: [TOTEM] entering GATHER state from 11. Aug 23 04:51:39 server1 openais[27441]: [TOTEM] Creating commit token because I am the rep. Aug 23 04:51:39 server1 openais[27441]: [TOTEM] Saving state aru d high seq received d Aug 23 04:51:39 server1 openais[27441]: [TOTEM] Storing new sequence id for ring 3d30c Aug 23 04:51:39 server1 openais[27441]: [TOTEM] entering COMMIT state. Aug 23 04:51:39 server1 openais[27441]: [TOTEM] entering RECOVERY state. Aug 23 04:51:39 server1 openais[27441]: [TOTEM] position [0] member 192.168.61.132: Aug 23 04:51:39 server1 openais[27441]: [TOTEM] previous ring seq 250632 rep 192.168.61.132 Aug 23 04:51:39 server1 openais[27441]: [TOTEM] aru d high delivered d received flag 1 Aug 23 04:51:39 server1 openais[27441]: [TOTEM] position [1] member 192.168.61.133: Aug 23 04:51:39 server1 openais[27441]: [TOTEM] previous ring seq 250632 rep 192.168.61.133 Aug 23 04:51:39 server1 openais[27441]: [TOTEM] aru d high delivered d received flag 1 Aug 23 04:51:39 server1 openais[27441]: [TOTEM] Did not need to originate any messages in recovery. Aug 23 04:51:39 server1 openais[27441]: [TOTEM] Sending initial ORF token Aug 23 04:51:39 server1 openais[27441]: [CLM ] CLM CONFIGURATION CHANGE Aug 23 04:51:39 server1 openais[27441]: [CLM ] New Configuration: Aug 23 04:51:39 server1 openais[27441]: [CLM ] r(0) ip(192.168.61.132) Aug 23 04:51:39 server1 openais[27441]: [CLM ] Members Left: Aug 23 04:51:39 server1 openais[27441]: [CLM ] Members Joined: Aug 23 04:51:39 server1 openais[27441]: [CLM ] CLM CONFIGURATION CHANGE Aug 23 04:51:39 server1 openais[27441]: [CLM ] New Configuration: Aug 23 04:51:39 server1 openais[27441]: [CLM ] r(0) ip(192.168.61.132) Aug 23 04:51:39 server1 openais[27441]: [CLM ] r(0) ip(192.168.61.133) Aug 23 04:51:39 server1 openais[27441]: [CLM ] Members Left: Aug 23 04:51:39 server1 openais[27441]: [CLM ] Members Joined: Aug 23 04:51:39 server1 openais[27441]: [CLM ] r(0) ip(192.168.61.133) Aug 23 04:51:39 server1 openais[27441]: [SYNC ] This node is within the primary component and will provide service. Aug 23 04:51:39 server1 openais[27441]: [TOTEM] entering OPERATIONAL state. Aug 23 04:51:39 server1 openais[27441]: [CLM ] got nodejoin message 192.168.61.132 Aug 23 04:51:39 server1 openais[27441]: [CLM ] got nodejoin message 192.168.61.133 Aug 23 04:51:50 server1 openais[27441]: [TOTEM] The token was lost in the OPERATIONAL state. Aug 23 04:51:50 server1 openais[27441]: [TOTEM] Receive multicast socket recv buffer size (320000 bytes). Aug 23 04:51:50 server1 openais[27441]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). Aug 23 04:51:50 server1 openais[27441]: [TOTEM] entering GATHER state from 2. Aug 23 04:51:52 server1 openais[27441]: [TOTEM] entering GATHER state from 0. Aug 23 04:51:52 server1 openais[27441]: [TOTEM] Creating commit token because I am the rep. Aug 23 04:51:52 server1 openais[27441]: [TOTEM] Saving state aru 1b high seq received 1b Aug 23 04:51:52 server1 openais[27441]: [TOTEM] Storing new sequence id for ring 3d310 Aug 23 04:51:52 server1 openais[27441]: [TOTEM] entering COMMIT state. Aug 23 04:51:52 server1 openais[27441]: [TOTEM] entering RECOVERY state. Aug 23 04:51:52 server1 openais[27441]: [TOTEM] position [0] member 192.168.61.132: Aug 23 04:51:52 server1 openais[27441]: [TOTEM] previous ring seq 250636 rep 192.168.61.132 Aug 23 04:51:52 server1 openais[27441]: [TOTEM] aru 1b high delivered 1b received flag 1 Aug 23 04:51:52 server1 openais[27441]: [TOTEM] Did not need to originate any messages in recovery. Aug 23 04:51:52 server1 openais[27441]: [TOTEM] Sending initial ORF token Aug 23 04:51:52 server1 openais[27441]: [CLM ] CLM CONFIGURATION CHANGE Aug 23 04:51:52 server1 openais[27441]: [CLM ] New Configuration: Aug 23 04:51:52 server1 openais[27441]: [CLM ] r(0) ip(192.168.61.132) Aug 23 04:51:52 server1 openais[27441]: [CLM ] Members Left: Aug 23 04:51:52 server1 openais[27441]: [CLM ] r(0) ip(192.168.61.133) Aug 23 04:51:52 server1 openais[27441]: [CLM ] Members Joined: Aug 23 04:51:52 server1 openais[27441]: [CLM ] CLM CONFIGURATION CHANGE Aug 23 04:51:52 server1 openais[27441]: [CLM ] New Configuration: Aug 23 04:51:52 server1 openais[27441]: [CLM ] r(0) ip(192.168.61.132) Aug 23 04:51:52 server1 openais[27441]: [CLM ] Members Left: Aug 23 04:51:52 server1 openais[27441]: [CLM ] Members Joined: Aug 23 04:51:52 server1 openais[27441]: [SYNC ] This node is within the primary component and will provide service. Aug 23 04:51:52 server1 openais[27441]: [TOTEM] entering OPERATIONAL state. Aug 23 04:51:52 server1 openais[27441]: [CLM ] got nodejoin message 192.168.61.132 Aug 23 05:02:02 server1 ccsd[24441]: Stopping ccsd, SIGTERM received. Aug 23 05:02:02 server1 NAMC Aug 23 05:02:03 server1 openais[27441]: [SERV ] Unloading all openais components Aug 23 05:02:03 server1 openais[27441]: [SERV ] AIS Executive exiting (reason: CMAN requested shutdown). Aug 23 05:02:03 server1 openais[27441]: [SERV ] Unloading all openais components Aug 23 05:02:03 server1 openais[27441]: [SERV ] Unloading openais component: openais_confdb v0 (19/10) Aug 23 05:02:03 server1 openais[27441]: [SERV ] Unloading openais component: openais_cpg v0 (18/8) Aug 23 05:02:03 server1 openais[27441]: [SERV ] Unloading openais component: openais_cfg v0 (17/7) Aug 23 05:02:03 server1 openais[27441]: [SERV ] Unloading openais component: openais_msg v0 (16/6) Aug 23 05:02:03 server1 openais[27441]: [SERV ] Unloading openais component: openais_lck v0 (15/5) Aug 23 05:02:03 server1 openais[27441]: [SERV ] Unloading openais component: openais_evt v0 (14/4) Aug 23 05:02:03 server1 openais[27441]: [SERV ] Unloading openais component: openais_ckpt v0 (13/3) Aug 23 05:02:03 server1 openais[27441]: [SERV ] Unloading openais component: openais_amf v0 (12/2) Aug 23 05:02:03 server1 openais[27441]: [SERV ] Unloading openais component: openais_clm v0 (11/1) Aug 23 05:02:03 server1 openais[27441]: [SERV ] Unloading openais component: openais_evs v0 (10/0) Aug 23 05:02:03 server1 openais[27441]: [SERV ] Unloading openais component: openais_cman v0 (9/9) Aug 23 05:02:03 server1 openais[27441]: [SERV ] AIS Executive exiting (reason: CMAN requested shutdown). Aug 23 05:02:08 server1 ccsd[27850]: Starting ccsd 2.0.115: Aug 23 05:02:08 server1 ccsd[27850]: Built: Dec 10 2010 11:26:52 Aug 23 05:02:08 server1 ccsd[27850]: Copyright (C) Red Hat, Inc. 2004 All rights reserved. Aug 23 05:02:09 server1 ccsd[27850]: cluster.conf (cluster name = newCluster, version = 16) found. Aug 23 05:02:09 server1 ccsd[27850]: Unable to sendto broadcast ipv4 socket, but inet_ntop returned NULL pointer: Network is unreachable Aug 23 05:02:37 server1 last message repeated 16 times Aug 23 05:02:38 server1 ccsd[27850]: Unable to connect to cluster infrastructure after 30 seconds. Aug 23 05:02:39 server1 ccsd[27850]: Unable to sendto broadcast ipv4 socket, but inet_ntop returned NULL pointer: Network is unreachable Aug 23 05:02:49 server1 last message repeated 6 times Aug 23 05:02:49 server1 openais[27860]: [MAIN ] AIS Executive Service RELEASE 'subrev 1887 version 0.80.6' Aug 23 05:02:49 server1 openais[27860]: [MAIN ] Copyright (C) 2002-2006 MontaVista Software, Inc and contributors. Aug 23 05:02:49 server1 openais[27860]: [MAIN ] Copyright (C) 2006 Red Hat, Inc. Aug 23 05:02:49 server1 openais[27860]: [MAIN ] AIS Executive Service: started and ready to provide service. Aug 23 05:02:49 server1 openais[27860]: [MAIN ] Using default multicast address of 239.192.168.93 Aug 23 05:02:49 server1 ccsd[27850]: Initial status:: Quorate Aug 23 05:02:49 server1 openais[27860]: [TOTEM] Token Timeout (10000 ms) retransmit timeout (495 ms) Aug 23 05:02:49 server1 openais[27860]: [TOTEM] token hold (386 ms) retransmits before loss (20 retrans) Aug 23 05:02:49 server1 openais[27860]: [TOTEM] join (60 ms) send_join (0 ms) consensus (2000 ms) merge (200 ms) Aug 23 05:02:49 server1 openais[27860]: [TOTEM] downcheck (1000 ms) fail to recv const (2500 msgs) Aug 23 05:02:49 server1 openais[27860]: [TOTEM] seqno unchanged const (30 rotations) Maximum network MTU 1402 Aug 23 05:02:49 server1 openais[27860]: [TOTEM] window size per rotation (50 messages) maximum messages per rotation (17 messages) Aug 23 05:02:49 server1 openais[27860]: [TOTEM] missed count const (5 messages) Aug 23 05:02:49 server1 groupd[27879]: found uncontrolled kernel object rgmanager in /sys/kernel/dlm Aug 23 05:02:49 server1 groupd[27879]: local node must be reset to clear 1 uncontrolled instances of gfs and/or dlm Aug 23 05:02:49 server1 fence_node[27880]: Fence of "server1" was unsuccessful Aug 23 05:02:49 server1 openais[27860]: [TOTEM] send threads (0 threads) Aug 23 05:02:49 server1 openais[27860]: [TOTEM] RRP token expired timeout (495 ms) Aug 23 05:02:49 server1 openais[27860]: [TOTEM] RRP token problem counter (2000 ms) Aug 23 05:02:49 server1 openais[27860]: [TOTEM] RRP threshold (10 problem count) Aug 23 05:02:49 server1 openais[27860]: [TOTEM] RRP mode set to none. Aug 23 05:02:49 server1 openais[27860]: [TOTEM] heartbeat_failures_allowed (0) Aug 23 05:02:49 server1 openais[27860]: [TOTEM] max_network_delay (50 ms) Aug 23 05:02:49 server1 openais[27860]: [TOTEM] HeartBeat is Disabled. To enable set heartbeat_failures_allowed > 0 Aug 23 05:02:49 server1 openais[27860]: [TOTEM] Receive multicast socket recv buffer size (320000 bytes). Aug 23 05:02:49 server1 openais[27860]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). Aug 23 05:02:49 server1 openais[27860]: [TOTEM] The network interface [192.168.61.132] is now up. Aug 23 05:02:49 server1 openais[27860]: [TOTEM] Created or loaded sequence id 250640.192.168.61.132 for this ring. Aug 23 05:02:49 server1 openais[27860]: [TOTEM] entering GATHER state from 15. Aug 23 05:02:49 server1 openais[27860]: [CMAN ] CMAN 2.0.115 (built Dec 10 2010 11:26:55) started Aug 23 05:02:49 server1 openais[27860]: [MAIN ] Service initialized 'openais CMAN membership service 2.01' Aug 23 05:02:49 server1 openais[27860]: [SERV ] Service initialized 'openais extended virtual synchrony service' Aug 23 05:02:49 server1 openais[27860]: [SERV ] Service initialized 'openais cluster membership service B.01.01' Aug 23 05:02:49 server1 openais[27860]: [SERV ] Service initialized 'openais availability management framework B.01.01' Aug 23 05:02:49 server1 openais[27860]: [SERV ] Service initialized 'openais checkpoint service B.01.01' Aug 23 05:02:49 server1 openais[27860]: [SERV ] Service initialized 'openais event service B.01.01' Aug 23 05:02:49 server1 openais[27860]: [SERV ] Service initialized 'openais distributed locking service B.01.01' Aug 23 05:02:49 server1 openais[27860]: [SERV ] Service initialized 'openais message service B.01.01' Aug 23 05:02:49 server1 openais[27860]: [SERV ] Service initialized 'openais configuration service' Aug 23 05:02:49 server1 openais[27860]: [SERV ] Service initialized 'openais cluster closed process group service v1.01' Aug 23 05:02:49 server1 openais[27860]: [SERV ] Service initialized 'openais cluster config database access v1.01' Aug 23 05:02:49 server1 openais[27860]: [SYNC ] Not using a virtual synchrony filter. Aug 23 05:02:49 server1 openais[27860]: [TOTEM] Creating commit token because I am the rep. Aug 23 05:02:49 server1 openais[27860]: [TOTEM] Saving state aru 0 high seq received 0 Aug 23 05:02:49 server1 openais[27860]: [TOTEM] Storing new sequence id for ring 3d314 Aug 23 05:02:49 server1 openais[27860]: [TOTEM] entering COMMIT state. Aug 23 05:02:49 server1 openais[27860]: [TOTEM] entering RECOVERY state. Aug 23 05:02:49 server1 openais[27860]: [TOTEM] position [0] member 192.168.61.132: Aug 23 05:02:49 server1 openais[27860]: [TOTEM] previous ring seq 250640 rep 192.168.61.132 Aug 23 05:02:49 server1 openais[27860]: [TOTEM] aru 0 high delivered 0 received flag 1 Aug 23 05:02:49 server1 openais[27860]: [TOTEM] Did not need to originate any messages in recovery. Aug 23 05:02:49 server1 openais[27860]: [TOTEM] Sending initial ORF token Aug 23 05:02:49 server1 openais[27860]: [CLM ] CLM CONFIGURATION CHANGE Aug 23 05:02:49 server1 openais[27860]: [CLM ] New Configuration: Aug 23 05:02:49 server1 openais[27860]: [CLM ] Members Left: Aug 23 05:02:49 server1 openais[27860]: [CLM ] Members Joined: Aug 23 05:02:49 server1 openais[27860]: [CLM ] CLM CONFIGURATION CHANGE Aug 23 05:02:49 server1 openais[27860]: [CLM ] New Configuration: Aug 23 05:02:49 server1 openais[27860]: [CLM ] r(0) ip(192.168.61.132) Aug 23 05:02:49 server1 openais[27860]: [CLM ] Members Left: Aug 23 05:02:49 server1 openais[27860]: [CLM ] Members Joined: Aug 23 05:02:49 server1 openais[27860]: [CLM ] r(0) ip(192.168.61.132) Aug 23 05:02:49 server1 openais[27860]: [SYNC ] This node is within the primary component and will provide service. Aug 23 05:02:49 server1 openais[27860]: [TOTEM] entering OPERATIONAL state. Aug 23 05:02:49 server1 openais[27860]: [CMAN ] quorum regained, resuming activity Aug 23 05:02:49 server1 openais[27860]: [CLM ] got nodejoin message 192.168.61.132 Aug 23 05:02:49 server1 openais[27860]: [CMAN ] cman killed by node 1 because we were killed by cman_tool or other application Aug 23 05:02:50 server1 openais[27860]: [SERV ] Unloading all openais components Aug 23 05:02:50 server1 openais[27860]: [SERV ] Unloading openais component: openais_confdb v0 (19/10) Aug 23 05:02:50 server1 openais[27860]: [SERV ] Unloading openais component: openais_cpg v0 (18/8) Aug 23 05:02:50 server1 openais[27860]: [SERV ] Unloading openais component: openais_cfg v0 (17/7) Aug 23 05:02:50 server1 openais[27860]: [SERV ] Unloading openais component: openais_msg v0 (16/6) Aug 23 05:02:50 server1 openais[27860]: [SERV ] Unloading openais component: openais_lck v0 (15/5) Aug 23 05:02:50 server1 openais[27860]: [SERV ] Unloading openais component: openais_evt v0 (14/4) Aug 23 05:02:50 server1 openais[27860]: [SERV ] Unloading openais component: openais_ckpt v0 (13/3) Aug 23 05:02:50 server1 openais[27860]: [SERV ] Unloading openais component: openais_amf v0 (12/2) Aug 23 05:02:50 server1 openais[27860]: [SERV ] Unloading openais component: openais_clm v0 (11/1) Aug 23 05:02:50 server1 openais[27860]: [SERV ] Unloading openais component: openais_evs v0 (10/0) Aug 23 05:02:50 server1 openais[27860]: [SERV ] Unloading openais component: openais_cman v0 (9/9) Aug 23 05:02:50 server1 openais[27860]: [SERV ] AIS Executive exiting (reason: CMAN kill requested, exiting). Aug 23 05:02:59 server1 dlm_controld[27892]: group_init error (nil) 111 Aug 23 05:02:59 server1 fenced[27886]: group_init error (nil) 111 Aug 23 05:02:59 server1 gfs_controld[27898]: group_init error (nil) 111 Aug 23 05:03:09 server1 ccsd[27850]: Unable to connect to cluster infrastructure after 60 seconds. Aug 23 05:03:40 server1 ccsd[27850]: Unable to connect to cluster infrastructure after 90 seconds.
Attachment:
cluster.conf
Description: Binary data
-- Linux-cluster mailing list Linux-cluster@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/linux-cluster
