Re: failover domain and service start

["YB Tan Sri Dato Sri' Adli a.k.a Dell" <white.heron@xxxxxxxxx>]

please call me 60173623661 for linux system configuration.

Sent from Yahoo Mail for iPhone

---

From: emmanuel segura <emi2fast@xxxxxxxxx>;
To: linux clustering <linux-cluster@xxxxxxxxxx>;
Subject: Re: failover domain and service start
Sent: Thu, Dec 19, 2013 9:05:18 AM

other thing, change the rgmanager loglevel, for more information you can use this link https://fedorahosted.org/cluster/wiki/RGManager

2013/12/19 emmanuel segura <emi2fast@xxxxxxxxx>

grep "/vms_c" /proc/mounts in every node

2013/12/18 Paras pradhan <pradhanparas@xxxxxxxxx>

I see this:

Dec 18 16:17:18 cvtst3 clurgmgrd[13935]: <notice> Starting stopped service vm:guest1 

Dec 18 16:17:19 cvtst3 clurgmgrd[13935]: <notice> start on vm "guest1" returned 1 (generic error) 

Dec 18 16:17:19 cvtst3 clurgmgrd[13935]: <warning> #68: Failed to start vm:guest1; return value: 1 

Dec 18 16:17:19 cvtst3 clurgmgrd[13935]: <notice> Stopping service vm:guest1 

On Wed, Dec 18, 2013 at 4:39 PM, emmanuel segura <emi2fast@xxxxxxxxx> wrote:

can you show the log?

2013/12/18 Paras pradhan <pradhanparas@xxxxxxxxx>

Added but same problem. vm does not start.

-Paras.

On Wed, Dec 18, 2013 at 1:13 PM, emmanuel segura <emi2fast@xxxxxxxxx> wrote:

from script vm.sh i saw it try to discovery the hypervisor you are using, with  hypervisor="xen" you force the script to use xen

2013/12/18 Paras pradhan <pradhanparas@xxxxxxxxx>

The only parameter I don't have is:  hypervisor="xen"

Does it matter?

This is what i have:

<vm autostart="1" domain="myfd1" exclusive="0" max_restarts="0" name="guest1" path="/vms_c" recovery="restart" restart_expire_time="0" use_virsh="0"/>

-Paras.

On Wed, Dec 18, 2013 at 12:24 PM, emmanuel segura <emi2fast@xxxxxxxxx> wrote:

<vm name="guest1" hypervisor="xen" path="/vms_c" use_virsh="0">

Incriment the config version of cluster.conf and ccs_tool update /etc/cluster/cluster.conf

2013/12/18 Paras pradhan <pradhanparas@xxxxxxxxx>

Emmauel,

With export OCF_RESKEY_name="guest1" ; export OCF_RESKEY_path="/vms_c" ; export OCF_RESKEY_use_virsh=0

, I can start the vm using : /usr/share/cluster/vm.sh

I am wondering how to make the changes to cluser.conf or other files so that we can start the vm using clucvcsadm.

-Thanks

sorry for the delay.

Paras.

On Thu, Dec 5, 2013 at 12:36 PM, Paras pradhan <pradhanparas@xxxxxxxxx> wrote:

Emmanue no.l I was busy on some other things I will test and let you know asap !

On Thu, Dec 5, 2013 at 12:26 PM, emmanuel segura <emi2fast@xxxxxxxxx> wrote:

Hello Paras

did you solved the problem?

Thanks
Emmanuel

2013/11/25 emmanuel segura <emi2fast@xxxxxxxxx>

Hello Paras

Maybe i found the solution, in function validate_all we got

       if [ -z "$OCF_RESKEY_hypervisor" ] ||
           [ "$OCF_RESKEY_hypervisor" = "auto" ]; then
                export OCF_RESKEY_hypervisor="`virsh version | grep \"Running hypervisor:\" | awk '{print $3}' | tr A-Z a-z`"
                if [ -z "$OCF_RESKEY_hypervisor" ]; then
                        ocf_log err "Could not determine Hypervisor"
                        return $OCF_ERR_ARGS
                fi
                echo Hypervisor: $OCF_RESKEY_hypervisor
        fi

        #
        # Xen hypervisor only for when use_virsh = 0.
        #
        if [ "$OCF_RESKEY_use_virsh" = "0" ]; then
                if [ "$OCF_RESKEY_hypervisor" != "xen" ]; then
                        ocf_log err "Cannot use $OCF_RESKEY_hypervisor hypervisor without using virsh"
                        return $OCF_ERR_ARGS
                fi

with this following enviroment variables, when i tested by hand the angent uses xm commands

env | grep OCF
OCF_RESKEY_hypervisor=xen
OCF_RESKEY_path=/vms_c
OCF_RESKEY_name=guest1
OCF_RESKEY_use_virsh=0

[root@client ~]# /usr/share/cluster/vm.sh status
Management tool: xm
<err>    Cannot find 'xm'; is it installed?
[vm.sh] Cannot find 'xm'; is it installed?

I don't have xen installed to test it

                if [ -n "$OCF_RESKEY_xmlfile" ]; then
                        ocf_log err "Cannot use xmlfile if use_virsh is set to 0"
                        return $OCF_ERR_ARGS
                fi

2013/11/25 emmanuel segura <emi2fast@xxxxxxxxx>

Hello paras

missing the export command in front of variables, the correct way is this

export OCF_RESKEY_name="guest1" ; export OCF_RESKEY_path="/vms_c" ; export OCF_RESKEY_use_virsh=0
[root@client ~]# env | grep OCF
OCF_RESKEY_path=/vms_c
OCF_RESKEY_name=guest1
OCF_RESKEY_use_virsh=0

2013/11/25 emmanuel segura <emi2fast@xxxxxxxxx>

Hello Paras

I have  a centos 6, i don't know if it is different on redhat 5, but i saw in the script vm.sh calls do_start function when start parameter is given

do_start()
{
        if [ "$OCF_RESKEY_use_virsh" = "1" ]; then
                do_virsh_start $*
                return $?
        fi

        do_xm_start $*
        return $?
}

i don't know why because the vm.sh uses virsh when you launch the script by hand :(

2013/11/25 Paras pradhan <pradhanparas@xxxxxxxxx>

Looks like use_virsh=0 has no effect.

--

[root@cvtst3 ~]# export OCF_RESKEY_name="guest1" ; OCF_RESKEY_path="/vms_c" ; OCF_RESKEY_use_virsh=0

[root@cvtst3 ~]# set -x

++ printf '\033]0;%s@%s:%s\007' root cvtst3 '~'

[root@cvtst3 ~]# /usr/share/cluster/vm.sh start

+ /usr/share/cluster/vm.sh start

Hypervisor: xen

Management tool: virsh

Hypervisor URI: xen:///

Migration URI format: xenmigr://target_host/

Virtual machine guest1 is error: failed to get domain 'guest1'

error: Domain not found: xenUnifiedDomainLookupByName

<debug>  virsh -c xen:/// start guest1

error: failed to get domain 'guest1'

error: Domain not found: xenUnifiedDomainLookupByName

++ printf '\033]0;%s@%s:%s\007' root cvtst3 '~'

[root@cvtst3 ~]# set +x

+ set +x

---

-Paras.

On Fri, Nov 22, 2013 at 5:22 PM, emmanuel segura <emi2fast@xxxxxxxxx> wrote:

Hellos Paras

Stop the vm and retry to start the vm with following commands and if you got some error show it

export OCF_RESKEY_name="guest1" ; OCF_RESKEY_path="/vms_c" ; OCF_RESKEY_use_virsh=0

set -x
/usr/share/cluster/vm.sh start
set +x

2013/11/22 Paras pradhan <pradhanparas@xxxxxxxxx>

I found the workaround to my issue. What i did is:

run the vm using xm and then start using cluvscadm. This works for me for the time being but I am not sure what is causing this. This is what I did

xm create /vms_c/guest1

clusvcadm -e vm: guest1 ( This detects that guest1 is up and quickly changes its status to success)

Although i used virt-install, it also create a xem format configuration file and since use_virsh=0 it should be able to use this xen format config file. This is my vm configuration:

---

name = "guest1"

maxmem = 2048

memory = 512

vcpus = 1

#cpus="1-2"

bootloader = "/usr/bin/pygrub"

_on_poweroff_ = "destroy"

_on_reboot_ = "restart"

_on_crash_ = "restart"

vfb = [  ]

disk = [ "tap:aio:/vms_c/guest1.img,xvda,w", "tap:aio:/vms_c/guest1-disk.img,xvdb,w" ]

vif = [ "rate=10MB/s,mac=00:16:3e:6b:be:71,bridge=xenbr0" ]

---

Thanks for you help Emmanuel ! Really appreciate it.

-Paras.

On Fri, Nov 22, 2013 at 11:10 AM, emmanuel segura <emi2fast@xxxxxxxxx> wrote:

ok, but your vm doesn't start on others nodes, i think, for configuration problems
================================================================

Nov 21 15:40:29 vtst3 clurgmgrd[13911]: <notice> start on vm "guest1" returned 1 (generic error) 

Nov 21 15:40:29 vtst3 clurgmgrd[13911]: <warning> #68: Failed to start vm:guest1; return value: 1 

Nov 21 15:40:29 vtst3 clurgmgrd[13911]: <notice> Stopping service vm:guest1 

Nov 21 15:40:35 vtst3 clurgmgrd[13911]: <notice> Service vm:guest1 is recovering 

Nov 21 15:40:35 vtst3 clurgmgrd[13911]: <warning> #71: Relocating failed service vm:guest1 

Nov 21 15:40:35 vtst3 clurgmgrd[13911]: <notice> Service vm:guest1 is stopped

================================================================

in few words, try in every cluster node

export OCF_RESKEY_name="guest1" ; OCF_RESKEY_path="/vms_c"

set -x
/usr/share/cluster/vm.sh start

/usr/share/cluster/vm.sh stop

after you check if your vm can start and stop on every cluster node,

/usr/share/cluster/vm.sh start
/usr/share/cluster/vm.sh migrate name_of_a_cluster_node

can you show me your vm configuration under /vms_c?

Thanks

Emmanuel

2013/11/22 Paras pradhan <pradhanparas@xxxxxxxxx>

And also to test I made use_virsh=1 , same problem. The vm does not start up if the FD domains are offline. 

-Paras.

On Fri, Nov 22, 2013 at 10:37 AM, Paras pradhan <pradhanparas@xxxxxxxxx> wrote:

Well thats seems to theoretically correct. But right now my cluser has use_virsh=0 and I don't have any issue untill my mebmers on the failover domains are offline. So wondering what is it that clusvcadm -e is looking when I don't use virsh .

On Fri, Nov 22, 2013 at 10:05 AM, emmanuel segura <emi2fast@xxxxxxxxx> wrote:

If you used virt-install, i think you need use virsh, the cluster uses xm xen command if you got use_virsh=0 and virsh if you got use_virsh=1 in your cluster config

2013/11/22 Paras pradhan <pradhanparas@xxxxxxxxx>

I use virt-install to create virtual machines. Is there a way to debug why clusvcadm -e vm:guest1 is failing? vm.sh  seems to use virsh and my cluster.conf has use_virsh=0

Thanks

Paras.

On Nov 21, 2013 5:53 PM, "emmanuel segura" <emi2fast@xxxxxxxxx> wrote:

but did you configure your vm with xen tools or using virt-manager?

2013/11/22 Paras pradhan <pradhanparas@xxxxxxxxx>

Well no i don't want to use virsh.   But as we are debugging with virsh now i found a strange issue.

I exported an xml file and imported to all nodes . Ran

---

name="guest1" path="/vms_c"

export OCF_RESKEY_name="guest1" ; OCF_RESKEY_path="/vms_c"

set -x
/usr/share/cluster/vm.sh start
set +x

--

vm starts now. BUT from a cluster service : cluvscam -e vm:guest1 , same error.

So if i populate all my domains' config files to all my cluser nodes and make use_virsh=1, then the issue is resolved. But this is a lot of work for those who have hundreds of vm.

vm.start uses virsh . Is there a way to tell him not use virsh?

Thanks

Paras.

On Thu, Nov 21, 2013 at 5:19 PM, emmanuel segura <emi2fast@xxxxxxxxx> wrote:

if you are using virsh for manage your vms, change this in your cluster.conf

from
use_virsh="0"

to
use_virsh="1"

2013/11/22 Paras pradhan <pradhanparas@xxxxxxxxx>

I think i found the problem. 

virsh list --all does not show my vm . This is because it was created on another node. and another node has it. Now I want to start the service on a different node in which it was not created or where virsh list --all does not have an entry. Is it possible to create this entry using a xen config file?Looks like this is now a Xen issue rather than a linux-cluster issue . :)

Paras.

On Thu, Nov 21, 2013 at 4:58 PM, emmanuel segura <emi2fast@xxxxxxxxx> wrote:

1:did you verify your xen livemigration configuration?

2: where you vm disk reside?

3: can you see your vm defined on every cluster node with xm list?

2013/11/21 Paras pradhan <pradhanparas@xxxxxxxxx>

This is what I get

Hypervisor: xen

Management tool: virsh

Hypervisor URI: xen:///

Migration URI format: xenmigr://target_host/

Virtual machine guest1 is error: failed to get domain 'guest1'

error: Domain not found: xenUnifiedDomainLookupByName

<debug>  virsh -c xen:/// start guest1

error: failed to get domain 'guest1'

error: Domain not found: xenUnifiedDomainLookupByName

++ printf '\033]0;%s@%s:%s\007' root vtst3 '~'

[root@cvtst3 ~]# set +x

+ set +x

--

I am wondering why it failed to get domain .

-Paras.

On Thu, Nov 21, 2013 at 4:43 PM, emmanuel segura <emi2fast@xxxxxxxxx> wrote:

yes

2013/11/21 Paras pradhan <pradhanparas@xxxxxxxxx>

Well it is guest1. Isn't it?.

<vm autostart="1" domain="myfd1" exclusive="0" max_restarts="0" name="guest1" path="/vms_c" recovery="restart" restart_expire_time="0" use_virsh="0"/>

It is a vm service if it matters.

-Paras.

On Thu, Nov 21, 2013 at 4:22 PM, emmanuel segura <emi2fast@xxxxxxxxx> wrote:

use the servicename you defined in your cluster.conf

2013/11/21 Paras pradhan <pradhanparas@xxxxxxxxx>

Says:

Running in test mode.

No resource guest1 of type service found

-Paras.

On Thu, Nov 21, 2013 at 4:07 PM, emmanuel segura <emi2fast@xxxxxxxxx> wrote:

rg_test test /etc/cluster/cluster.conf start service guest1

2013/11/21 Paras pradhan <pradhanparas@xxxxxxxxx>

Hi,

My failover domain looks like this:

<failoverdomain name="myfd1" nofailback="1" ordered="1" restricted="0">

                                <failoverdomainnode name="vtst1" priority="1"/>

                                <failoverdomainnode name="vtst3" priority="2"/>

                                <failoverdomainnode name="vtst2" priority="3"/>

                        </failoverdomain>

I have vm service that uses this failover domain. If my node vtst1 is offline, the service doesnot start on vtst3 which is 2nd in the priority.

I tried to start it with: clusvcadm -e vm:guest1   and even with -F and -m option.

All i see is this error:

Nov 21 15:40:29 vtst3 clurgmgrd[13911]: <notice> start on vm "guest1" returned 1 (generic error) 

Nov 21 15:40:29 vtst3 clurgmgrd[13911]: <warning> #68: Failed to start vm:guest1; return value: 1 

Nov 21 15:40:29 vtst3 clurgmgrd[13911]: <notice> Stopping service vm:guest1 

Nov 21 15:40:35 vtst3 clurgmgrd[13911]: <notice> Service vm:guest1 is recovering 

Nov 21 15:40:35 vtst3 clurgmgrd[13911]: <warning> #71: Relocating failed service vm:guest1 

Nov 21 15:40:35 vtst3 clurgmgrd[13911]: <notice> Service vm:guest1 is stopped 

How do I debug?

Thanks!

Paras.

--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster

--
esta es mi vida e me la vivo hasta que dios quiera

--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster

--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster

--
esta es mi vida e me la vivo hasta que dios quiera

--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster

--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster

--
esta es mi vida e me la vivo hasta que dios quiera

--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster

--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster

--
esta es mi vida e me la vivo hasta que dios quiera

--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster

--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster

--
esta es mi vida e me la vivo hasta que dios quiera

--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster

--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster

--
esta es mi vida e me la vivo hasta que dios quiera

--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster

--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster

--
esta es mi vida e me la vivo hasta que dios quiera

--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster

--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster

--
esta es mi vida e me la vivo hasta que dios quiera

--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster

--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster

--
esta es mi vida e me la vivo hasta que dios quiera

--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster

--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster

--
esta es mi vida e me la vivo hasta que dios quiera

--
esta es mi vida e me la vivo hasta que dios quiera

--
esta es mi vida e me la vivo hasta que dios quiera

--
esta es mi vida e me la vivo hasta que dios quiera

--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster

--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster

--
esta es mi vida e me la vivo hasta que dios quiera

--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster

--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster

--
esta es mi vida e me la vivo hasta que dios quiera

--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster

--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster

--
esta es mi vida e me la vivo hasta que dios quiera

--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster

--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster

--
esta es mi vida e me la vivo hasta que dios quiera

--
esta es mi vida e me la vivo hasta que dios quiera

-- 
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster