On Tue, Aug 18, 2009 at 8:26 AM, Moralejo, Alfredo<alfredo.moralejo@xxxxxxxxx> wrote: > Could you send logs in messages file including the part where the cluster dissolved messages is logged? Ok.. here are the logs . I shutdown node 3 which have 2 votes. Below are the logs from node1 and node 2. X.X.X.165 is node 1 X.X.X.172 is node 2 X.X.X.173 is node 3 Log from node1: ug 18 10:35:28 cvtst1 ntpd[3862]: synchronized to X.X.X.103, stratum 2 Aug 18 11:01:44 cvtst1 clurgmgrd[4309]: <notice> Member 3 shutting down Aug 18 11:01:57 cvtst1 kernel: peth0: received packet with own address as source address Aug 18 11:01:57 cvtst1 qdiskd[3361]: <info> Node 3 shutdown Aug 18 11:02:02 cvtst1 kernel: peth0: received packet with own address as source address Aug 18 11:02:07 cvtst1 openais[3318]: [TOTEM] The token was lost in the OPERATIONAL state. Aug 18 11:02:07 cvtst1 openais[3318]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes). Aug 18 11:02:07 cvtst1 openais[3318]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). Aug 18 11:02:07 cvtst1 openais[3318]: [TOTEM] entering GATHER state from 2. Aug 18 11:02:12 cvtst1 openais[3318]: [TOTEM] entering GATHER state from 11. Aug 18 11:02:12 cvtst1 openais[3318]: [TOTEM] Creating commit token because I am the rep. Aug 18 11:02:12 cvtst1 openais[3318]: [TOTEM] Saving state aru 9f high seq received 9f Aug 18 11:02:12 cvtst1 openais[3318]: [TOTEM] Storing new sequence id for ring 758 Aug 18 11:02:12 cvtst1 openais[3318]: [TOTEM] entering COMMIT state. Aug 18 11:02:12 cvtst1 openais[3318]: [TOTEM] entering RECOVERY state. Aug 18 11:02:12 cvtst1 openais[3318]: [TOTEM] position [0] member X.X.X.165: Aug 18 11:02:12 cvtst1 openais[3318]: [TOTEM] previous ring seq 1876 rep X.X.X.165 Aug 18 11:02:12 cvtst1 openais[3318]: [TOTEM] aru 9f high delivered 9f received flag 1 Aug 18 11:02:12 cvtst1 openais[3318]: [TOTEM] position [1] member X.X.X.172: Aug 18 11:02:12 cvtst1 openais[3318]: [TOTEM] previous ring seq 1876 rep X.X.X.165 Aug 18 11:02:12 cvtst1 openais[3318]: [TOTEM] aru 9f high delivered 9f received flag 1 Aug 18 11:02:12 cvtst1 openais[3318]: [TOTEM] Did not need to originate any messages in recovery. Aug 18 11:02:12 cvtst1 openais[3318]: [TOTEM] Sending initial ORF token Aug 18 11:02:12 cvtst1 kernel: dlm: closing connection to node 3 Aug 18 11:02:12 cvtst1 openais[3318]: [CLM ] CLM CONFIGURATION CHANGE Aug 18 11:02:12 cvtst1 openais[3318]: [CLM ] New Configuration: Aug 18 11:02:12 cvtst1 openais[3318]: [CLM ] r(0) ip(X.X.X.165) Aug 18 11:02:12 cvtst1 openais[3318]: [CLM ] r(0) ip(X.X.X.172) Aug 18 11:02:12 cvtst1 openais[3318]: [CLM ] Members Left: Aug 18 11:02:12 cvtst1 openais[3318]: [CLM ] r(0) ip(X.X.X.173) Aug 18 11:02:12 cvtst1 clurgmgrd[4309]: <emerg> #1: Quorum Dissolved Aug 18 11:02:12 cvtst1 openais[3318]: [CLM ] Members Joined: Aug 18 11:02:12 cvtst1 openais[3318]: [CLM ] CLM CONFIGURATION CHANGE Aug 18 11:02:12 cvtst1 openais[3318]: [CLM ] New Configuration: Aug 18 11:02:12 cvtst1 openais[3318]: [CLM ] r(0) ip(X.X.X.165) Aug 18 11:02:12 cvtst1 openais[3318]: [CLM ] r(0) ip(X.X.X.172) Aug 18 11:02:12 cvtst1 openais[3318]: [CLM ] Members Left: Aug 18 11:02:12 cvtst1 openais[3318]: [CLM ] Members Joined: Aug 18 11:02:12 cvtst1 openais[3318]: [SYNC ] This node is within the primary component and will provide service. Aug 18 11:02:12 cvtst1 openais[3318]: [TOTEM] entering OPERATIONAL state. Aug 18 11:02:12 cvtst1 openais[3318]: [CLM ] got nodejoin message X.X.X.165 Aug 18 11:02:12 cvtst1 openais[3318]: [CLM ] got nodejoin message X.X.X.172 Aug 18 11:02:13 cvtst1 openais[3318]: [CPG ] got joinlist message from node 2 Aug 18 11:02:13 cvtst1 openais[3318]: [CPG ] got joinlist message from node 1 Aug 18 11:02:13 cvtst1 openais[3318]: [CMAN ] lost contact with quorum device Aug 18 11:02:13 cvtst1 openais[3318]: [CMAN ] quorum lost, blocking activity Aug 18 11:02:13 cvtst1 ccsd[3274]: Cluster is not quorate. Refusing connection. Aug 18 11:02:13 cvtst1 ccsd[3274]: Error while processing connect: Connection refused Aug 18 11:02:22 cvtst1 ccsd[3274]: Cluster is not quorate. Refusing connection. Aug 18 11:02:22 cvtst1 ccsd[3274]: Error while processing connect: Connection refused Aug 18 11:02:27 cvtst1 qdiskd[3361]: <info> Node 1 is the master Aug 18 11:02:30 cvtst1 openais[3318]: [CMAN ] quorum regained, resuming activity Aug 18 11:02:36 cvtst1 kernel: xenbr0: port 3(vif1.0) entering disabled state Aug 18 11:02:36 cvtst1 kernel: device vif1.0 left promiscuous mode Aug 18 11:02:36 cvtst1 kernel: xenbr0: port 3(vif1.0) entering disabled state Aug 18 11:02:39 cvtst1 clurgmgrd[4309]: <notice> Quorum Regained Aug 18 11:02:41 cvtst1 clurgmgrd[4309]: <notice> Starting stopped service vm:guest1 Aug 18 11:02:42 cvtst1 kernel: tap tap-2-51712: 2 getting info Aug 18 11:02:43 cvtst1 kernel: device vif2.0 entered promiscuous mode Aug 18 11:02:43 cvtst1 kernel: ADDRCONF(NETDEV_UP): vif2.0: link is not ready Aug 18 11:02:43 cvtst1 clurgmgrd[4309]: <notice> Service vm:guest1 started Aug 18 11:02:47 cvtst1 kernel: blktap: ring-ref 8, event-channel 6, protocol 1 (x86_64-abi) Aug 18 11:02:56 cvtst1 kernel: xenbr0: topology change detected, propagating Aug 18 11:02:56 cvtst1 kernel: xenbr0: port 3(vif2.0) entering forwarding state Aug 18 11:02:56 cvtst1 kernel: ADDRCONF(NETDEV_CHANGE): vif2.0: link becomes ready >From node2: Aug 18 11:01:44 cvtst2 clurgmgrd[4365]: <notice> Member 3 shutting down Aug 18 11:01:55 cvtst2 qdiskd[3403]: <info> Node 3 shutdown Aug 18 11:01:57 cvtst2 kernel: peth0: received packet with own address as source address Aug 18 11:02:02 cvtst2 kernel: peth0: received packet with own address as source address Aug 18 11:02:07 cvtst2 openais[3359]: [TOTEM] entering GATHER state from 12. Aug 18 11:02:09 cvtst2 openais[3359]: [CMAN ] lost contact with quorum device Aug 18 11:02:09 cvtst2 openais[3359]: [CMAN ] quorum lost, blocking activity Aug 18 11:02:09 cvtst2 clurgmgrd[4365]: <emerg> #1: Quorum Dissolved Aug 18 11:02:09 cvtst2 kernel: dlm: closing connection to node 3 Aug 18 11:02:10 cvtst2 ccsd[3316]: Cluster is not quorate. Refusing connection. Aug 18 11:02:10 cvtst2 ccsd[3316]: Error while processing connect: Connection refused Aug 18 11:02:12 cvtst2 openais[3359]: [TOTEM] entering GATHER state from 0. Aug 18 11:02:12 cvtst2 openais[3359]: [TOTEM] Saving state aru 9f high seq received 9f Aug 18 11:02:12 cvtst2 openais[3359]: [TOTEM] Storing new sequence id for ring 758 Aug 18 11:02:12 cvtst2 openais[3359]: [TOTEM] entering COMMIT state. Aug 18 11:02:12 cvtst2 openais[3359]: [TOTEM] entering RECOVERY state. Aug 18 11:02:12 cvtst2 openais[3359]: [TOTEM] position [0] member X.X.X.165: Aug 18 11:02:12 cvtst2 openais[3359]: [TOTEM] previous ring seq 1876 rep X.X.X.165 Aug 18 11:02:12 cvtst2 openais[3359]: [TOTEM] aru 9f high delivered 9f received flag 1 Aug 18 11:02:12 cvtst2 openais[3359]: [TOTEM] position [1] member X.X.X.172: Aug 18 11:02:12 cvtst2 openais[3359]: [TOTEM] previous ring seq 1876 rep X.X.X.165 Aug 18 11:02:12 cvtst2 openais[3359]: [TOTEM] aru 9f high delivered 9f received flag 1 Aug 18 11:02:12 cvtst2 openais[3359]: [TOTEM] Did not need to originate any messages in recovery. Aug 18 11:02:12 cvtst2 openais[3359]: [CLM ] CLM CONFIGURATION CHANGE Aug 18 11:02:12 cvtst2 openais[3359]: [CLM ] New Configuration: Aug 18 11:02:12 cvtst2 openais[3359]: [CLM ] r(0) ip(X.X.X.165) Aug 18 11:02:12 cvtst2 openais[3359]: [CLM ] r(0) ip(X.X.X.172) Aug 18 11:02:12 cvtst2 openais[3359]: [CLM ] Members Left: Aug 18 11:02:12 cvtst2 openais[3359]: [CLM ] r(0) ip(X.X.X.173) Aug 18 11:02:12 cvtst2 openais[3359]: [CLM ] Members Joined: Aug 18 11:02:12 cvtst2 openais[3359]: [CMAN ] quorum regained, resuming activity Aug 18 11:02:12 cvtst2 openais[3359]: [CLM ] CLM CONFIGURATION CHANGE Aug 18 11:02:12 cvtst2 openais[3359]: [CLM ] New Configuration: Aug 18 11:02:12 cvtst2 openais[3359]: [CLM ] r(0) ip(X.X.X.165) Aug 18 11:02:12 cvtst2 openais[3359]: [CLM ] r(0) ip(X.X.X.172) Aug 18 11:02:12 cvtst2 openais[3359]: [CLM ] Members Left: Aug 18 11:02:12 cvtst2 openais[3359]: [CLM ] Members Joined: Aug 18 11:02:12 cvtst2 openais[3359]: [SYNC ] This node is within the primary component and will provide service. Aug 18 11:02:12 cvtst2 openais[3359]: [TOTEM] entering OPERATIONAL state. Aug 18 11:02:12 cvtst2 openais[3359]: [CLM ] got nodejoin message X.X.X.165 Aug 18 11:02:12 cvtst2 openais[3359]: [CLM ] got nodejoin message X.X.X.172 Aug 18 11:02:12 cvtst2 openais[3359]: [CPG ] got joinlist message from node 2 Aug 18 11:02:12 cvtst2 openais[3359]: [CPG ] got joinlist message from node 1 Aug 18 11:02:25 cvtst2 qdiskd[3403]: <info> Assuming master role Aug 18 11:02:33 cvtst2 kernel: xenbr0: port 3(vif1.0) entering disabled state Aug 18 11:02:33 cvtst2 kernel: device vif1.0 left promiscuous mode Aug 18 11:02:33 cvtst2 kernel: xenbr0: port 3(vif1.0) entering disabled state Aug 18 11:02:36 cvtst2 clurgmgrd[4365]: <notice> Quorum Regained Aug 18 11:02:39 cvtst2 clurgmgrd[4365]: <notice> Starting stopped service vm:guest2 Aug 18 11:02:41 cvtst2 kernel: tap tap-2-51712: 2 getting info Aug 18 11:02:41 cvtst2 kernel: device vif2.0 entered promiscuous mode Aug 18 11:02:41 cvtst2 kernel: ADDRCONF(NETDEV_UP): vif2.0: link is not ready Aug 18 11:02:41 cvtst2 clurgmgrd[4365]: <notice> Service vm:guest2 started Aug 18 11:02:45 cvtst2 kernel: blktap: ring-ref 8, event-channel 6, protocol 1 (x86_64-abi) Aug 18 11:02:53 cvtst2 kernel: xenbr0: topology change detected, propagating Aug 18 11:02:53 cvtst2 kernel: xenbr0: port 3(vif2.0) entering forwarding state Aug 18 11:02:53 cvtst2 kernel: ADDRCONF(NETDEV_CHANGE): vif2.0: link becomes read > > Regards, > > Alfredo > > -----Original Message----- > From: linux-cluster-bounces@xxxxxxxxxx [mailto:linux-cluster-bounces@xxxxxxxxxx] On Behalf Of Paras pradhan > Sent: Monday, August 17, 2009 11:59 PM > To: linux clustering > Subject: Cluster behavior > > I have a 3 nodes cluster. > > Node A - Vote 1 > > Node B - Vote 1 > > Node C - Votes 2 > > Qdisk - Votes 3 > > > Altogether the cluster has 7 votes. The required min quorum to run the > cluster would be in this case 4. Now if I poweroff node 1, I can see > Quorum Dissolved in the terminals of Node 2 and Node3 . This cluster > has xen virtual machines. > > Whats wrong and how to do debug the problem? > > Thanks > Paras. > > -- > Linux-cluster mailing list > Linux-cluster@xxxxxxxxxx > https://www.redhat.com/mailman/listinfo/linux-cluster > > -- > Linux-cluster mailing list > Linux-cluster@xxxxxxxxxx > https://www.redhat.com/mailman/listinfo/linux-cluster > Thanks Paras. -- Linux-cluster mailing list Linux-cluster@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/linux-cluster
