Change to 7 and i got this log Sep 29 15:33:50 cvtst1 rgmanager: [23295]: <notice> Shutting down Cluster Service Manager... Sep 29 15:33:50 cvtst1 clurgmgrd[22869]: <notice> Shutting down Sep 29 15:33:50 cvtst1 clurgmgrd[22869]: <notice> Shutting down Sep 29 15:33:50 cvtst1 clurgmgrd[22869]: <notice> Shutdown complete, exiting Sep 29 15:33:50 cvtst1 rgmanager: [23295]: <notice> Cluster Service Manager is stopped. Sep 29 15:33:51 cvtst1 clurgmgrd[23324]: <notice> Resource Group Manager Starting Sep 29 15:33:51 cvtst1 clurgmgrd[23324]: <info> Loading Service Data Sep 29 15:33:51 cvtst1 clurgmgrd[23324]: <debug> Loading Resource Rules Sep 29 15:33:52 cvtst1 clurgmgrd[23324]: <debug> 21 rules loaded Sep 29 15:33:52 cvtst1 clurgmgrd[23324]: <debug> Building Resource Trees Sep 29 15:33:52 cvtst1 clurgmgrd[23324]: <debug> 0 resources defined Sep 29 15:33:52 cvtst1 clurgmgrd[23324]: <debug> Loading Failover Domains Sep 29 15:33:52 cvtst1 clurgmgrd[23324]: <debug> 1 domains defined Sep 29 15:33:52 cvtst1 clurgmgrd[23324]: <debug> 1 events defined Sep 29 15:33:52 cvtst1 clurgmgrd[23324]: <info> Initializing Services Sep 29 15:33:52 cvtst1 clurgmgrd[23324]: <info> Services Initialized Sep 29 15:33:52 cvtst1 clurgmgrd[23324]: <debug> Event: Port Opened Sep 29 15:33:52 cvtst1 clurgmgrd[23324]: <info> State change: Local UP Sep 29 15:33:52 cvtst1 clurgmgrd[23324]: <info> State change: cvtst2 UP Sep 29 15:33:52 cvtst1 clurgmgrd[23324]: <info> State change: cvtst3 UP Sep 29 15:33:57 cvtst1 clurgmgrd[23324]: <debug> Event (1:2:1) Processed Sep 29 15:33:57 cvtst1 clurgmgrd[23324]: <debug> Event (0:1:1) Processed Sep 29 15:33:57 cvtst1 clurgmgrd[23324]: <debug> Event (0:3:1) Processed Sep 29 15:34:02 cvtst1 clurgmgrd[23324]: <debug> 3 events processed Anything unusual here? Paras. On Tue, Sep 29, 2009 at 11:51 AM, brem belguebli <brem.belguebli@xxxxxxxxx> wrote: > I use log_level=7 to have more debugging info. > > It seems 4 is not enough. > > Brem > > > 2009/9/29, Paras pradhan <pradhanparas@xxxxxxxxx>: >> Withe log_level of 3 I got only this >> >> Sep 29 10:31:31 cvtst1 rgmanager: [7170]: <notice> Shutting down >> Cluster Service Manager... >> Sep 29 10:31:31 cvtst1 clurgmgrd[6673]: <notice> Shutting down >> Sep 29 10:31:41 cvtst1 clurgmgrd[6673]: <notice> Shutdown complete, exiting >> Sep 29 10:31:41 cvtst1 rgmanager: [7170]: <notice> Cluster Service >> Manager is stopped. >> Sep 29 10:31:42 cvtst1 clurgmgrd[7224]: <notice> Resource Group >> Manager Starting >> Sep 29 10:39:06 cvtst1 rgmanager: [10327]: <notice> Shutting down >> Cluster Service Manager... >> Sep 29 10:39:16 cvtst1 rgmanager: [10327]: <notice> Cluster Service >> Manager is stopped. >> Sep 29 10:39:16 cvtst1 clurgmgrd[10380]: <notice> Resource Group >> Manager Starting >> Sep 29 10:39:52 cvtst1 clurgmgrd[10380]: <notice> Member 1 shutting down >> >> I do not know what the last line means. >> >> rgmanager version I am running is: >> rgmanager-2.0.52-1.el5.centos >> >> I don't what has gone wrong. >> >> Thanks >> Paras. >> >> >> On Mon, Sep 28, 2009 at 6:41 PM, brem belguebli >> <brem.belguebli@xxxxxxxxx> wrote: >> > you mean it stopped successfully on all the nodes but it is failing to >> > start only on node cvtst1 ? >> > >> > look at the following page to make rgmanager more verbose. It 'll >> > help debug.... >> > >> > http://sources.redhat.com/cluster/wiki/RGManager >> > >> > at Logging Configuration section >> > >> > >> > >> > >> > 2009/9/29 Paras pradhan <pradhanparas@xxxxxxxxx>: >> >> Brem, >> >> >> >> When I try to restart rgmanager on all the nodes, this time i do not >> >> see rgmanager running on the first node. But I do see on other 2 >> >> nodes. >> >> >> >> Log on the first node: >> >> >> >> Sep 28 18:13:58 cvtst1 clurgmgrd[24099]: <notice> Resource Group >> >> Manager Starting >> >> Sep 28 18:17:29 cvtst1 rgmanager: [24627]: <notice> Shutting down >> >> Cluster Service Manager... >> >> Sep 28 18:17:29 cvtst1 clurgmgrd[24099]: <notice> Shutting down >> >> Sep 28 18:17:39 cvtst1 clurgmgrd[24099]: <notice> Shutdown complete, exiting >> >> Sep 28 18:17:39 cvtst1 rgmanager: [24627]: <notice> Cluster Service >> >> Manager is stopped. >> >> Sep 28 18:17:40 cvtst1 clurgmgrd[24679]: <notice> Resource Group >> >> Manager Starting >> >> >> >> - >> >> It seems service is running , but I do not see rgmanger running using clustat >> >> >> >> >> >> Don't know what is going on. >> >> >> >> Thanks >> >> Paras. >> >> >> >> >> >> On Mon, Sep 28, 2009 at 5:46 PM, brem belguebli >> >> <brem.belguebli@xxxxxxxxx> wrote: >> >>> Paras, >> >>> >> >>> Another thing, it would have been more interesting to have a start >> >>> DEBUG not a stop. >> >>> >> >>> That's why I was asking you to first stop the vm manually on all your >> >>> nodes, stop eventually rgmanager on all the nodes to reset the >> >>> potential wrong states you may have, restart rgmanager. >> >>> >> >>> If your VM is configured to autostart, this will make it start. >> >>> >> >>> It should normally fail (as it does now). Send out your newly created >> >>> DEBUG file. >> >>> >> >>> 2009/9/29 brem belguebli <brem.belguebli@xxxxxxxxx>: >> >>>> Hi Paras, >> >>>> >> >>>> >> >>>> I don't know the xen/cluster combination well, but if I do remember >> >>>> well, I think I've read somewhere that when using xen you have to >> >>>> declare the use_virsh=0 key in the VM definition in the cluster.conf. >> >>>> >> >>>> This would make rgmanager use xm commands instead of virsh >> >>>> The DEBUG output shows clearly that you are using virsh to manage your >> >>>> VM instead of xm commands. >> >>>> Check out the RH docs about virtualization >> >>>> >> >>>> I'm not a 100% sure about that, I may be completely wrong. >> >>>> >> >>>> Brem >> >>>> >> >>>> 2009/9/28 Paras pradhan <pradhanparas@xxxxxxxxx>: >> >>>>> The only thing I noticed is the message after stopping the vm using xm >> >>>>> in all nodes and starting using clusvcadm is >> >>>>> >> >>>>> "Virtual machine guest1 is blocked" >> >>>>> >> >>>>> The whole DEBUG file is attached. >> >>>>> >> >>>>> >> >>>>> Thanks >> >>>>> Paras. >> >>>>> >> >>>>> On Fri, Sep 25, 2009 at 5:53 PM, brem belguebli >> >>>>> <brem.belguebli@xxxxxxxxx> wrote: >> >>>>>> There's a problem with the script that is called by rgmanager to start >> >>>>>> the VM, I don't know what causes it >> >>>>>> >> >>>>>> May be you should try something like : >> >>>>>> >> >>>>>> 1) stop the VM on all nodes with xm commands >> >>>>>> 2) edit the /usr/share/cluster/vm.sh script and add the following >> >>>>>> lines (after the #!/bin/bash ): >> >>>>>> exec >/tmp/DEBUG 2>&1 >> >>>>>> set -x >> >>>>>> 3) start the VM with clusvcadm -e vm:guest1 >> >>>>>> >> >>>>>> It should fail as it did before. >> >>>>>> >> >>>>>> edit the the /tmp/DEBUG file and you will be able to see where it >> >>>>>> fails (it may generate a lot of debug) >> >>>>>> >> >>>>>> 4) remove the debug lines from /usr/share/cluster/vm.sh >> >>>>>> >> >>>>>> Post the DEBUG file if you're not able to see where it fails. >> >>>>>> >> >>>>>> Brem >> >>>>>> >> >>>>>> 2009/9/26 Paras pradhan <pradhanparas@xxxxxxxxx>: >> >>>>>>> No I am not manually starting not using automatic init scripts. >> >>>>>>> >> >>>>>>> I started the vm using: clusvcadm -e vm:guest1 >> >>>>>>> >> >>>>>>> I have just stopped using clusvcadm -s vm:guest1. For few seconds it >> >>>>>>> says guest1 started . But after a while I can see the guest1 on all >> >>>>>>> three nodes. >> >>>>>>> >> >>>>>>> clustat says: >> >>>>>>> >> >>>>>>> Service Name Owner (Last) >> >>>>>>> State >> >>>>>>> ------- ---- ----- ------ >> >>>>>>> ----- >> >>>>>>> vm:guest1 (none) >> >>>>>>> stopped >> >>>>>>> >> >>>>>>> But I can see the vm from xm li. >> >>>>>>> >> >>>>>>> This is what I can see from the log: >> >>>>>>> >> >>>>>>> >> >>>>>>> Sep 25 17:19:01 cvtst1 clurgmgrd[4298]: <notice> start on vm "guest1" >> >>>>>>> returned 1 (generic error) >> >>>>>>> Sep 25 17:19:01 cvtst1 clurgmgrd[4298]: <warning> #68: Failed to start >> >>>>>>> vm:guest1; return value: 1 >> >>>>>>> Sep 25 17:19:01 cvtst1 clurgmgrd[4298]: <notice> Stopping service vm:guest1 >> >>>>>>> Sep 25 17:19:02 cvtst1 clurgmgrd[4298]: <notice> Service vm:guest1 is >> >>>>>>> recovering >> >>>>>>> Sep 25 17:19:15 cvtst1 clurgmgrd[4298]: <notice> Recovering failed >> >>>>>>> service vm:guest1 >> >>>>>>> Sep 25 17:19:16 cvtst1 clurgmgrd[4298]: <notice> start on vm "guest1" >> >>>>>>> returned 1 (generic error) >> >>>>>>> Sep 25 17:19:16 cvtst1 clurgmgrd[4298]: <warning> #68: Failed to start >> >>>>>>> vm:guest1; return value: 1 >> >>>>>>> Sep 25 17:19:16 cvtst1 clurgmgrd[4298]: <notice> Stopping service vm:guest1 >> >>>>>>> Sep 25 17:19:17 cvtst1 clurgmgrd[4298]: <notice> Service vm:guest1 is >> >>>>>>> recovering >> >>>>>>> >> >>>>>>> >> >>>>>>> Paras. >> >>>>>>> >> >>>>>>> On Fri, Sep 25, 2009 at 5:07 PM, brem belguebli >> >>>>>>> <brem.belguebli@xxxxxxxxx> wrote: >> >>>>>>>> Have you started your VM via rgmanager (clusvcadm -e vm:guest1) or >> >>>>>>>> using xm commands out of cluster control (or maybe a thru an >> >>>>>>>> automatic init script ?) >> >>>>>>>> >> >>>>>>>> When clustered, you should never be starting services (manually or >> >>>>>>>> thru automatic init script) out of cluster control >> >>>>>>>> >> >>>>>>>> The thing would be to stop your vm on all the nodes with the adequate >> >>>>>>>> xm command (not using xen myself) and try to start it with clusvcadm. >> >>>>>>>> >> >>>>>>>> Then see if it is started on all nodes (send clustat output) >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> 2009/9/25 Paras pradhan <pradhanparas@xxxxxxxxx>: >> >>>>>>>>> Ok. Please see below. my vm is running on all nodes though clustat >> >>>>>>>>> says it is stopped. >> >>>>>>>>> >> >>>>>>>>> -- >> >>>>>>>>> [root@cvtst1 ~]# clustat >> >>>>>>>>> Cluster Status for test @ Fri Sep 25 16:52:34 2009 >> >>>>>>>>> Member Status: Quorate >> >>>>>>>>> >> >>>>>>>>> Member Name ID Status >> >>>>>>>>> ------ ---- ---- ------ >> >>>>>>>>> cvtst2 1 Online, rgmanager >> >>>>>>>>> cvtst1 2 Online, >> >>>>>>>>> Local, rgmanager >> >>>>>>>>> cvtst3 3 Online, rgmanager >> >>>>>>>>> >> >>>>>>>>> Service Name Owner (Last) >> >>>>>>>>> State >> >>>>>>>>> ------- ---- ----- ------ >> >>>>>>>>> ----- >> >>>>>>>>> vm:guest1 (none) >> >>>>>>>>> stopped >> >>>>>>>>> [root@cvtst1 ~]# >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> --- >> >>>>>>>>> o/p of xm li on cvtst1 >> >>>>>>>>> >> >>>>>>>>> -- >> >>>>>>>>> [root@cvtst1 ~]# xm li >> >>>>>>>>> Name ID Mem(MiB) VCPUs State Time(s) >> >>>>>>>>> Domain-0 0 3470 2 r----- 28939.4 >> >>>>>>>>> guest1 7 511 1 -b---- 7727.8 >> >>>>>>>>> >> >>>>>>>>> o/p of xm li on cvtst2 >> >>>>>>>>> >> >>>>>>>>> -- >> >>>>>>>>> [root@cvtst2 ~]# xm li >> >>>>>>>>> Name ID Mem(MiB) VCPUs State Time(s) >> >>>>>>>>> Domain-0 0 3470 2 r----- 31558.9 >> >>>>>>>>> guest1 21 511 1 -b---- 7558.2 >> >>>>>>>>> --- >> >>>>>>>>> >> >>>>>>>>> Thanks >> >>>>>>>>> Paras. >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> On Fri, Sep 25, 2009 at 4:22 PM, brem belguebli >> >>>>>>>>> <brem.belguebli@xxxxxxxxx> wrote: >> >>>>>>>>>> It looks like no. >> >>>>>>>>>> >> >>>>>>>>>> can you send an output of clustat of when the VM is running on >> >>>>>>>>>> multiple nodes at the same time? >> >>>>>>>>>> >> >>>>>>>>>> And by the way, another one after having stopped (clusvcadm -s vm:guest1) ? >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> 2009/9/25 Paras pradhan <pradhanparas@xxxxxxxxx>: >> >>>>>>>>>>> Anyone having issue as mine? Virtual machine service is not being >> >>>>>>>>>>> properly handled by the cluster. >> >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> Thanks >> >>>>>>>>>>> Paras. >> >>>>>>>>>>> >> >>>>>>>>>>> On Mon, Sep 21, 2009 at 9:55 AM, Paras pradhan <pradhanparas@xxxxxxxxx> wrote: >> >>>>>>>>>>>> Ok.. here is my cluster.conf file >> >>>>>>>>>>>> >> >>>>>>>>>>>> -- >> >>>>>>>>>>>> [root@cvtst1 cluster]# more cluster.conf >> >>>>>>>>>>>> <?xml version="1.0"?> >> >>>>>>>>>>>> <cluster alias="test" config_version="9" name="test"> >> >>>>>>>>>>>> <fence_daemon clean_start="0" post_fail_delay="0" post_join_delay="3"/> >> >>>>>>>>>>>> <clusternodes> >> >>>>>>>>>>>> <clusternode name="cvtst2" nodeid="1" votes="1"> >> >>>>>>>>>>>> <fence/> >> >>>>>>>>>>>> </clusternode> >> >>>>>>>>>>>> <clusternode name="cvtst1" nodeid="2" votes="1"> >> >>>>>>>>>>>> <fence/> >> >>>>>>>>>>>> </clusternode> >> >>>>>>>>>>>> <clusternode name="cvtst3" nodeid="3" votes="1"> >> >>>>>>>>>>>> <fence/> >> >>>>>>>>>>>> </clusternode> >> >>>>>>>>>>>> </clusternodes> >> >>>>>>>>>>>> <cman/> >> >>>>>>>>>>>> <fencedevices/> >> >>>>>>>>>>>> <rm> >> >>>>>>>>>>>> <failoverdomains> >> >>>>>>>>>>>> <failoverdomain name="myfd1" nofailback="0" ordered="1" restricted="0"> >> >>>>>>>>>>>> <failoverdomainnode name="cvtst2" priority="3"/> >> >>>>>>>>>>>> <failoverdomainnode name="cvtst1" priority="1"/> >> >>>>>>>>>>>> <failoverdomainnode name="cvtst3" priority="2"/> >> >>>>>>>>>>>> </failoverdomain> >> >>>>>>>>>>>> </failoverdomains> >> >>>>>>>>>>>> <resources/> >> >>>>>>>>>>>> <vm autostart="1" domain="myfd1" exclusive="0" max_restarts="0" >> >>>>>>>>>>>> name="guest1" path="/vms" recovery="r >> >>>>>>>>>>>> estart" restart_expire_time="0"/> >> >>>>>>>>>>>> </rm> >> >>>>>>>>>>>> </cluster> >> >>>>>>>>>>>> [root@cvtst1 cluster]# >> >>>>>>>>>>>> ------ >> >>>>>>>>>>>> >> >>>>>>>>>>>> Thanks! >> >>>>>>>>>>>> Paras. >> >>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>>> On Sun, Sep 20, 2009 at 9:44 AM, Volker Dormeyer <volker@xxxxxxxxxxxx> wrote: >> >>>>>>>>>>>>> On Fri, Sep 18, 2009 at 05:08:57PM -0500, >> >>>>>>>>>>>>> Paras pradhan <pradhanparas@xxxxxxxxx> wrote: >> >>>>>>>>>>>>>> I am using cluster suite for HA of xen virtual machines. Now I am >> >>>>>>>>>>>>>> having another problem. When I start the my xen vm in one node, it >> >>>>>>>>>>>>>> also starts on other nodes. Which daemon controls this? >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> This is usually done bei clurgmgrd (which is part of the rgmanager >> >>>>>>>>>>>>> package). To me, this sounds like a configuration problem. Maybe, >> >>>>>>>>>>>>> you can post your cluster.conf? >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> Regards, >> >>>>>>>>>>>>> Volker >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> -- >> >>>>>>>>>>>>> Linux-cluster mailing list >> >>>>>>>>>>>>> Linux-cluster@xxxxxxxxxx >> >>>>>>>>>>>>> https://www.redhat.com/mailman/listinfo/linux-cluster >> >>>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> -- >> >>>>>>>>>>> Linux-cluster mailing list >> >>>>>>>>>>> Linux-cluster@xxxxxxxxxx >> >>>>>>>>>>> https://www.redhat.com/mailman/listinfo/linux-cluster >> >>>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> -- >> >>>>>>>>>> Linux-cluster mailing list >> >>>>>>>>>> Linux-cluster@xxxxxxxxxx >> >>>>>>>>>> https://www.redhat.com/mailman/listinfo/linux-cluster >> >>>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> -- >> >>>>>>>>> Linux-cluster mailing list >> >>>>>>>>> Linux-cluster@xxxxxxxxxx >> >>>>>>>>> https://www.redhat.com/mailman/listinfo/linux-cluster >> >>>>>>>>> >> >>>>>>>> >> >>>>>>>> -- >> >>>>>>>> Linux-cluster mailing list >> >>>>>>>> Linux-cluster@xxxxxxxxxx >> >>>>>>>> https://www.redhat.com/mailman/listinfo/linux-cluster >> >>>>>>>> >> >>>>>>> >> >>>>>>> -- >> >>>>>>> Linux-cluster mailing list >> >>>>>>> Linux-cluster@xxxxxxxxxx >> >>>>>>> https://www.redhat.com/mailman/listinfo/linux-cluster >> >>>>>>> >> >>>>>> >> >>>>>> -- >> >>>>>> Linux-cluster mailing list >> >>>>>> Linux-cluster@xxxxxxxxxx >> >>>>>> https://www.redhat.com/mailman/listinfo/linux-cluster >> >>>>>> >> >>>>> >> >>>>> -- >> >>>>> Linux-cluster mailing list >> >>>>> Linux-cluster@xxxxxxxxxx >> >>>>> https://www.redhat.com/mailman/listinfo/linux-cluster >> >>>>> >> >>>> >> >>> >> >>> -- >> >>> Linux-cluster mailing list >> >>> Linux-cluster@xxxxxxxxxx >> >>> https://www.redhat.com/mailman/listinfo/linux-cluster >> >>> >> >> >> >> -- >> >> Linux-cluster mailing list >> >> Linux-cluster@xxxxxxxxxx >> >> https://www.redhat.com/mailman/listinfo/linux-cluster >> >> >> > >> > -- >> > Linux-cluster mailing list >> > Linux-cluster@xxxxxxxxxx >> > https://www.redhat.com/mailman/listinfo/linux-cluster >> > >> >> -- >> Linux-cluster mailing list >> Linux-cluster@xxxxxxxxxx >> https://www.redhat.com/mailman/listinfo/linux-cluster >> > > -- > Linux-cluster mailing list > Linux-cluster@xxxxxxxxxx > https://www.redhat.com/mailman/listinfo/linux-cluster > -- Linux-cluster mailing list Linux-cluster@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/linux-cluster
