Hi Paras,
did you reboot all the cluster nodes? I needed a complete reboot
(actually I was angry enough to switch everything off and on again): a
restart of the cluster suite was not enough.
As far as I understood, since the cluster was not able to bring a VM in
a good running state, it decided that the VM has to migrate to another
node... and at the end all the VMs was trying to migrate from one node
to another with the result that I had all VMs starting on all the
cluster nodes.
Restarting the cluster suite didn't kill a lot of processes that was
stubbornly trying to migrate virtual machines...
Daniela
Paras pradhan wrote:
Adding use_virsh=0 works great. Now I do not have my vm starting at
all the nodes. This is a good fix. Thanks ..
The only problem left is I do not see rgmanager running on my node1
and the clustat of node2 and node3 is reporting vm as migrating
o/p
Service Name Owner (Last)
State
------- ---- ----- ------
-----
vm:guest1 cvtst1
migrating
[root@cvtst3 vms]#
Thanks
Paras.
On Tue, Oct 6, 2009 at 10:48 AM, Daniela Anzellotti
<daniela.anzellotti@xxxxxxxxxxxxx> wrote:
Hi Paras,
yes. At least it looks so...
We have a cluster of two nodes + a quorum disk (it's not configured as a
"two-node cluster")
They are running Scientific Linux 5.x, kernel 2.6.18-128.7.1.el5xen and
openais-0.80.6-8.el5.x86_64
cman-2.0.115-1.el5.x86_64
rgmanager-2.0.52-1.el5.x86_64
The XEN VMs access the disk as simple block devices.
Disks are on a SAN, configured with Clustered LVM.
xen-3.0.3-94.el5_4.1.x86_64
xen-libs-3.0.3-94.el5_4.1.x86_64
VM configuration files are as the following
name = "www1"
uuid = "3bd3e910-23c0-97ee-55ab-086260ef1e53"
memory = 1024
maxmem = 1024
vcpus = 1
bootloader = "/usr/bin/pygrub"
vfb = [ "type=vnc,vncunused=1,keymap=en-us" ]
disk = [ "phy:/dev/vg_cluster/www1.disk,xvda,w", \
"phy:/dev/vg_cluster/www1.swap,xvdb,w" ]
vif = [ "mac=00:16:3e:da:00:07,bridge=xenbr1" ]
on_poweroff = "destroy"
on_reboot = "restart"
on_crash = "restart"
extra = "xencons=tty0 console=tty0"
I changed in /etc/cluster/cluster.conf all the VM directive from
<vm autostart="1" domain="rhcs1_dom" exclusive="0" \
migrate="live" name="www1" path="/etc/xen" recovery="restart"/>
to
<vm autostart="1" use_virsh="0" domain="rhcs1_dom" exclusive="0" \
migrate="live" name="www1" path="/etc/xen" recovery="restart"/>
Rebooted the cluster nodes and it started working again...
As i said I hope I'll not have any other bad surprise (I tested a VM
migration and it is working too), but at least cluster it's working now (it
was not able to start a VM, before)!
Ciao
Daniela
Paras pradhan wrote:
So you mean your cluster is running fine with the CMAN
cman-2.0.115-1.el5.x86_64 ?
Which version of openais are you running?
Thanks
Paras.
On Mon, Oct 5, 2009 at 7:19 AM, Daniela Anzellotti
<daniela.anzellotti@xxxxxxxxxxxxx> wrote:
Hi all,
I had a problem similar to Paras's one today: yum updated the following
rpms
last week and today (I had to restart the cluster) the cluster was not
able
to start vm: services.
Oct 02 05:31:05 Updated: openais-0.80.6-8.el5.x86_64
Oct 02 05:31:07 Updated: cman-2.0.115-1.el5.x86_64
Oct 02 05:31:10 Updated: rgmanager-2.0.52-1.el5.x86_64
Oct 03 04:03:12 Updated: xen-libs-3.0.3-94.el5_4.1.x86_64
Oct 03 04:03:12 Updated: xen-libs-3.0.3-94.el5_4.1.i386
Oct 03 04:03:16 Updated: xen-3.0.3-94.el5_4.1.x86_64
So, after checked the vm.sh script, I add the declaration use_virsh="0"
in
the VM definition in the cluster.conf (as suggested by Brem, thanks!) and
everything is now working again.
BTW I didn't understand if the problem was caused by the new XEN version
or
the new openais one, thus I disabled automatic updates for both.
I hope I'll not have any other bad surprise...
Thank you,
cheers,
Daniela
Paras pradhan wrote:
Yes this is very strange. I don't know what to do now. May be re
create the cluster? But not a good solution actually.
Packages :
Kernel: kernel-xen-2.6.18-164.el5
OS: Full updated of CentOS 5.3 except CMAN downgraded to
cman-2.0.98-1.el5
Other packages related to cluster suite:
rgmanager-2.0.52-1.el5.centos
cman-2.0.98-1.el5
xen-3.0.3-80.el5_3.3
xen-libs-3.0.3-80.el5_3.3
kmod-gfs-xen-0.1.31-3.el5_3.1
kmod-gfs-xen-0.1.31-3.el5_3.1
kmod-gfs-0.1.31-3.el5_3.1
gfs-utils-0.1.18-1.el5
gfs2-utils-0.1.62-1.el5
lvm2-2.02.40-6.el5
lvm2-cluster-2.02.40-7.el5
openais-0.80.3-22.el5_3.9
Thanks!
Paras.
On Wed, Sep 30, 2009 at 10:02 AM, brem belguebli
<brem.belguebli@xxxxxxxxx> wrote:
Hi Paras,
Your cluster.conf file seems correct. If it is not a ntp issue, I
don't see anything except a bug that causes this, or some prerequisite
that is not respected.
May be you could post the versions (os, kernel, packages etc...) you
are using, someone may have hit the same issue with your versions.
Brem
2009/9/30, Paras pradhan <pradhanparas@xxxxxxxxx>:
All of the nodes are synced with ntp server. So this is not the case
with me.
Thanks
Paras.
On Tue, Sep 29, 2009 at 6:29 PM, Johannes Rußek
<johannes.russek@xxxxxxxxxxxxxxxxx> wrote:
make sure the time on the nodes is in sync, apparently when a node
has
too
much offset, you won't see rgmanager (even though the process is
running).
this happened today and setting the time fixed it for me. afaicr
there
was
no sign of this in the logs though.
johannes
Paras pradhan schrieb:
I don't see rgmanager .
Here is the o/p from clustat
[root@cvtst1 cluster]# clustat
Cluster Status for test @ Tue Sep 29 15:53:33 2009
Member Status: Quorate
Member Name ID
Status
------ ----
----
------
cvtst2 1 Online
cvtst1 2
Online,
Local
cvtst3 3 Online
Thanks
Paras.
On Tue, Sep 29, 2009 at 3:44 PM, brem belguebli
<brem.belguebli@xxxxxxxxx> wrote:
It looks correct, rgmanager seems to start on all nodes
what gives you clustat ?
If rgmanager doesn't show, check out the logs something may have
gone
wrong.
2009/9/29 Paras pradhan <pradhanparas@xxxxxxxxx>:
Change to 7 and i got this log
Sep 29 15:33:50 cvtst1 rgmanager: [23295]: <notice> Shutting down
Cluster Service Manager...
Sep 29 15:33:50 cvtst1 clurgmgrd[22869]: <notice> Shutting down
Sep 29 15:33:50 cvtst1 clurgmgrd[22869]: <notice> Shutting down
Sep 29 15:33:50 cvtst1 clurgmgrd[22869]: <notice> Shutdown
complete,
exiting
Sep 29 15:33:50 cvtst1 rgmanager: [23295]: <notice> Cluster
Service
Manager is stopped.
Sep 29 15:33:51 cvtst1 clurgmgrd[23324]: <notice> Resource Group
Manager Starting
Sep 29 15:33:51 cvtst1 clurgmgrd[23324]: <info> Loading Service
Data
Sep 29 15:33:51 cvtst1 clurgmgrd[23324]: <debug> Loading Resource
Rules
Sep 29 15:33:52 cvtst1 clurgmgrd[23324]: <debug> 21 rules loaded
Sep 29 15:33:52 cvtst1 clurgmgrd[23324]: <debug> Building Resource
Trees
Sep 29 15:33:52 cvtst1 clurgmgrd[23324]: <debug> 0 resources
defined
Sep 29 15:33:52 cvtst1 clurgmgrd[23324]: <debug> Loading Failover
Domains
Sep 29 15:33:52 cvtst1 clurgmgrd[23324]: <debug> 1 domains defined
Sep 29 15:33:52 cvtst1 clurgmgrd[23324]: <debug> 1 events defined
Sep 29 15:33:52 cvtst1 clurgmgrd[23324]: <info> Initializing
Services
Sep 29 15:33:52 cvtst1 clurgmgrd[23324]: <info> Services
Initialized
Sep 29 15:33:52 cvtst1 clurgmgrd[23324]: <debug> Event: Port
Opened
Sep 29 15:33:52 cvtst1 clurgmgrd[23324]: <info> State change:
Local
UP
Sep 29 15:33:52 cvtst1 clurgmgrd[23324]: <info> State change:
cvtst2
UP
Sep 29 15:33:52 cvtst1 clurgmgrd[23324]: <info> State change:
cvtst3
UP
Sep 29 15:33:57 cvtst1 clurgmgrd[23324]: <debug> Event (1:2:1)
Processed
Sep 29 15:33:57 cvtst1 clurgmgrd[23324]: <debug> Event (0:1:1)
Processed
Sep 29 15:33:57 cvtst1 clurgmgrd[23324]: <debug> Event (0:3:1)
Processed
Sep 29 15:34:02 cvtst1 clurgmgrd[23324]: <debug> 3 events
processed
Anything unusual here?
Paras.
On Tue, Sep 29, 2009 at 11:51 AM, brem belguebli
<brem.belguebli@xxxxxxxxx> wrote:
I use log_level=7 to have more debugging info.
It seems 4 is not enough.
Brem
2009/9/29, Paras pradhan <pradhanparas@xxxxxxxxx>:
Withe log_level of 3 I got only this
Sep 29 10:31:31 cvtst1 rgmanager: [7170]: <notice> Shutting down
Cluster Service Manager...
Sep 29 10:31:31 cvtst1 clurgmgrd[6673]: <notice> Shutting down
Sep 29 10:31:41 cvtst1 clurgmgrd[6673]: <notice> Shutdown
complete,
exiting
Sep 29 10:31:41 cvtst1 rgmanager: [7170]: <notice> Cluster
Service
Manager is stopped.
Sep 29 10:31:42 cvtst1 clurgmgrd[7224]: <notice> Resource Group
Manager Starting
Sep 29 10:39:06 cvtst1 rgmanager: [10327]: <notice> Shutting
down
Cluster Service Manager...
Sep 29 10:39:16 cvtst1 rgmanager: [10327]: <notice> Cluster
Service
Manager is stopped.
Sep 29 10:39:16 cvtst1 clurgmgrd[10380]: <notice> Resource Group
Manager Starting
Sep 29 10:39:52 cvtst1 clurgmgrd[10380]: <notice> Member 1
shutting
down
I do not know what the last line means.
rgmanager version I am running is:
rgmanager-2.0.52-1.el5.centos
I don't what has gone wrong.
Thanks
Paras.
On Mon, Sep 28, 2009 at 6:41 PM, brem belguebli
<brem.belguebli@xxxxxxxxx> wrote:
you mean it stopped successfully on all the nodes but it is
failing
to
start only on node cvtst1 ?
look at the following page to make rgmanager more verbose. It
'll
help debug....
http://sources.redhat.com/cluster/wiki/RGManager
at Logging Configuration section
2009/9/29 Paras pradhan <pradhanparas@xxxxxxxxx>:
Brem,
When I try to restart rgmanager on all the nodes, this time i
do
not
see rgmanager running on the first node. But I do see on other
2
nodes.
Log on the first node:
Sep 28 18:13:58 cvtst1 clurgmgrd[24099]: <notice> Resource
Group
Manager Starting
Sep 28 18:17:29 cvtst1 rgmanager: [24627]: <notice> Shutting
down
Cluster Service Manager...
Sep 28 18:17:29 cvtst1 clurgmgrd[24099]: <notice> Shutting
down
Sep 28 18:17:39 cvtst1 clurgmgrd[24099]: <notice> Shutdown
complete,
exiting
Sep 28 18:17:39 cvtst1 rgmanager: [24627]: <notice> Cluster
Service
Manager is stopped.
Sep 28 18:17:40 cvtst1 clurgmgrd[24679]: <notice> Resource
Group
Manager Starting
-
It seems service is running , but I do not see rgmanger
running
using clustat
Don't know what is going on.
Thanks
Paras.
On Mon, Sep 28, 2009 at 5:46 PM, brem belguebli
<brem.belguebli@xxxxxxxxx> wrote:
Paras,
Another thing, it would have been more interesting to have a
start
DEBUG not a stop.
That's why I was asking you to first stop the vm manually on
all
your
nodes, stop eventually rgmanager on all the nodes to reset
the
potential wrong states you may have, restart rgmanager.
If your VM is configured to autostart, this will make it
start.
It should normally fail (as it does now). Send out your newly
created
DEBUG file.
2009/9/29 brem belguebli <brem.belguebli@xxxxxxxxx>:
Hi Paras,
I don't know the xen/cluster combination well, but if I do
remember
well, I think I've read somewhere that when using xen you
have
to
declare the use_virsh=0 key in the VM definition in the
cluster.conf.
This would make rgmanager use xm commands instead of virsh
The DEBUG output shows clearly that you are using virsh to
manage
your
VM instead of xm commands.
Check out the RH docs about virtualization
I'm not a 100% sure about that, I may be completely wrong.
Brem
2009/9/28 Paras pradhan <pradhanparas@xxxxxxxxx>:
The only thing I noticed is the message after stopping the
vm
using xm
in all nodes and starting using clusvcadm is
"Virtual machine guest1 is blocked"
The whole DEBUG file is attached.
Thanks
Paras.
On Fri, Sep 25, 2009 at 5:53 PM, brem belguebli
<brem.belguebli@xxxxxxxxx> wrote:
There's a problem with the script that is called by
rgmanager to
start
the VM, I don't know what causes it
May be you should try something like :
1) stop the VM on all nodes with xm commands
2) edit the /usr/share/cluster/vm.sh script and add the
following
lines (after the #!/bin/bash ):
exec >/tmp/DEBUG 2>&1
set -x
3) start the VM with clusvcadm -e vm:guest1
It should fail as it did before.
edit the the /tmp/DEBUG file and you will be able to see
where
it
fails (it may generate a lot of debug)
4) remove the debug lines from /usr/share/cluster/vm.sh
Post the DEBUG file if you're not able to see where it
fails.
Brem
2009/9/26 Paras pradhan <pradhanparas@xxxxxxxxx>:
No I am not manually starting not using automatic init
scripts.
I started the vm using: clusvcadm -e vm:guest1
I have just stopped using clusvcadm -s vm:guest1. For few
seconds it
says guest1 started . But after a while I can see the
guest1 on
all
three nodes.
clustat says:
Service Name
Owner
(Last)
State
------- ----
-----
------
-----
vm:guest1
(none)
stopped
But I can see the vm from xm li.
This is what I can see from the log:
Sep 25 17:19:01 cvtst1 clurgmgrd[4298]: <notice> start on
vm
"guest1"
returned 1 (generic error)
Sep 25 17:19:01 cvtst1 clurgmgrd[4298]: <warning> #68:
Failed
to start
vm:guest1; return value: 1
Sep 25 17:19:01 cvtst1 clurgmgrd[4298]: <notice> Stopping
service vm:guest1
Sep 25 17:19:02 cvtst1 clurgmgrd[4298]: <notice> Service
vm:guest1 is
recovering
Sep 25 17:19:15 cvtst1 clurgmgrd[4298]: <notice>
Recovering
failed
service vm:guest1
Sep 25 17:19:16 cvtst1 clurgmgrd[4298]: <notice> start on
vm
"guest1"
returned 1 (generic error)
Sep 25 17:19:16 cvtst1 clurgmgrd[4298]: <warning> #68:
Failed
to start
vm:guest1; return value: 1
Sep 25 17:19:16 cvtst1 clurgmgrd[4298]: <notice> Stopping
service vm:guest1
Sep 25 17:19:17 cvtst1 clurgmgrd[4298]: <notice> Service
vm:guest1 is
recovering
Paras.
On Fri, Sep 25, 2009 at 5:07 PM, brem belguebli
<brem.belguebli@xxxxxxxxx> wrote:
Have you started your VM via rgmanager (clusvcadm -e
vm:guest1) or
using xm commands out of cluster control (or maybe a
thru
an
automatic init script ?)
When clustered, you should never be starting services
(manually or
thru automatic init script) out of cluster control
The thing would be to stop your vm on all the nodes with
the
adequate
xm command (not using xen myself) and try to start it
with
clusvcadm.
Then see if it is started on all nodes (send clustat
output)
2009/9/25 Paras pradhan <pradhanparas@xxxxxxxxx>:
Ok. Please see below. my vm is running on all nodes
though
clustat
says it is stopped.
--
[root@cvtst1 ~]# clustat
Cluster Status for test @ Fri Sep 25 16:52:34 2009
Member Status: Quorate
Member Name
ID Status
------ ----
---- ------
cvtst2
1
Online, rgmanager
cvtst1
2
Online,
Local, rgmanager
cvtst3
3
Online, rgmanager
Service Name
Owner (Last)
State
------- ----
----- ------
-----
vm:guest1
(none)
stopped
[root@cvtst1 ~]#
---
o/p of xm li on cvtst1
--
[root@cvtst1 ~]# xm li
Name ID Mem(MiB)
VCPUs
State Time(s)
Domain-0 0 3470
2
r----- 28939.4
guest1 7 511
1
-b---- 7727.8
o/p of xm li on cvtst2
--
[root@cvtst2 ~]# xm li
Name ID Mem(MiB)
VCPUs
State Time(s)
Domain-0 0 3470
2
r----- 31558.9
guest1 21 511
1
-b---- 7558.2
---
Thanks
Paras.
On Fri, Sep 25, 2009 at 4:22 PM, brem belguebli
<brem.belguebli@xxxxxxxxx> wrote:
It looks like no.
can you send an output of clustat of when the VM is
running
on
multiple nodes at the same time?
And by the way, another one after having stopped
(clusvcadm
-s vm:guest1) ?
2009/9/25 Paras pradhan <pradhanparas@xxxxxxxxx>:
Anyone having issue as mine? Virtual machine service
is
not
being
properly handled by the cluster.
Thanks
Paras.
On Mon, Sep 21, 2009 at 9:55 AM, Paras pradhan
<pradhanparas@xxxxxxxxx> wrote:
Ok.. here is my cluster.conf file
--
[root@cvtst1 cluster]# more cluster.conf
<?xml version="1.0"?>
<cluster alias="test" config_version="9"
name="test">
<fence_daemon clean_start="0"
post_fail_delay="0"
post_join_delay="3"/>
<clusternodes>
<clusternode name="cvtst2" nodeid="1"
votes="1">
<fence/>
</clusternode>
<clusternode name="cvtst1" nodeid="2"
votes="1">
<fence/>
</clusternode>
<clusternode name="cvtst3" nodeid="3"
votes="1">
<fence/>
</clusternode>
</clusternodes>
<cman/>
<fencedevices/>
<rm>
<failoverdomains>
<failoverdomain name="myfd1"
nofailback="0" ordered="1" restricted="0">
<failoverdomainnode
name="cvtst2" priority="3"/>
<failoverdomainnode
name="cvtst1" priority="1"/>
<failoverdomainnode
name="cvtst3" priority="2"/>
</failoverdomain>
</failoverdomains>
<resources/>
<vm autostart="1" domain="myfd1"
exclusive="0" max_restarts="0"
name="guest1" path="/vms" recovery="r
estart" restart_expire_time="0"/>
</rm>
</cluster>
[root@cvtst1 cluster]#
------
Thanks!
Paras.
On Sun, Sep 20, 2009 at 9:44 AM, Volker Dormeyer
<volker@xxxxxxxxxxxx> wrote:
On Fri, Sep 18, 2009 at 05:08:57PM -0500,
Paras pradhan <pradhanparas@xxxxxxxxx> wrote:
I am using cluster suite for HA of xen virtual
machines.
Now I am
having another problem. When I start the my xen vm
in
one node, it
also starts on other nodes. Which daemon controls
this?
This is usually done bei clurgmgrd (which is part
of
the
rgmanager
package). To me, this sounds like a configuration
problem. Maybe,
you can post your cluster.conf?
Regards,
Volker
--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster
--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster
--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster
--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster
--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster
--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster
--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster
--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster
--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster
--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster
--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster
--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster
--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster
--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster
--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster
--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster
--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster
--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster
--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster
--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster
--
- Daniela Anzellotti ------------------------------------
INFN Roma - tel.: +39.06.49914282 - fax: +39.06.490354
e-mail: daniela.anzellotti@xxxxxxxxxxxxx
---------------------------------------------------------
--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster
--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster
--
- Daniela Anzellotti ------------------------------------
INFN Roma - tel.: +39.06.49914282 - fax: +39.06.490354
e-mail: daniela.anzellotti@xxxxxxxxxxxxx
---------------------------------------------------------
--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster
--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster
--
- Daniela Anzellotti ------------------------------------
INFN Roma - tel.: +39.06.49914282 - fax: +39.06.490354
e-mail: daniela.anzellotti@xxxxxxxxxxxxx
---------------------------------------------------------
--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster
