Totem Process pause detected

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hello,

I'm running a centos 6.7 cluster of 2 nodes on a Hyper-V hypervisor. 
Every day at 11PM a snapshot job save both servers.
The snapshotting process seems to cause a loss of connectivity between the two nodes which results in the cluster partitioning and pacemaker to start services on both nodes.
Then once the snapshotting is done, the two halves of the cluster are able to see each other again and pacemaker chooses one on which to run the services.
Unfortunately that means that our DRBD partition has been mounted on both, so it now goes into «  split brain mode » .   


When I was running corosync 1.4, I used to adjust the « token » variable in the configuration file so that both nodes would wait longer before detecting a loss of the other.

Now that I have upgraded to corosync 2 (2.3.5 to be more precise) the problem is back with a vengeance.

I have tried the configuration below, with a a very high totem value, and that resulted in the following errors (I have since reverted that change):

Dec 21 08:59:13 [16696] node1 corosync notice  [TOTEM ] totemsrp.c:783 Process pause detected for 3464149 ms, flush
ing membership messages.
Dec 21 08:59:13 [16696] node1 corosync notice  [TOTEM ] totemsrp.c:783 Process pause detected for 3464149 ms, flush
ing membership messages.
Dec 21 08:59:13 [16696] node1 corosync notice  [TOTEM ] totemsrp.c:783 Process pause detected for 3464199 ms, flush
ing membership messages.


What can I do to prevent the cluster splitting apart during those nightly snapshots? 
How do I manually set a long totem timeout without breaking everything else?




======================================================================

Software version:
2.6.32-573.7.1.el6.x86_64

corosync-2.3.5-1.el6.x86_64
corosynclib-2.3.5-1.el6.x86_64

pacemaker-cluster-libs-1.1.13-1.el6.x86_64
pacemaker-cli-1.1.13-1.el6.x86_64

kmod-microsoft-hyper-v-4.0.11-20150728.x86_64
microsoft-hyper-v-4.0.11-20150728.x86_64

Configuration:

totem {
    version: 2

    crypto_cipher: none
    crypto_hash: none
    clear_node_high_bit: yes
    cluster_name: cluster
    transport: udpu
    token: 150000

    interface {
        ringnumber: 0
        bindnetaddr: 10.200.0.2
        mcastport: 5405
        ttl: 1
    }
}

nodelist {
    node {
        ring0_addr:  10.200.0.2
    }

    node {
        ring0_addr:  10.200.0.3
    }
}

logging {
    fileline: on
    to_stderr: no
    to_logfile: yes
    logfile: /var/log/cluster/corosync.log
    to_syslog: yes
    debug: off
    timestamp: on
    logger_subsys {
        subsys: QUORUM
        debug: off
    }
}


quorum {
    provider: corosync_votequorum
    two_node: 1
}



Thank you for your help,
— 
Ludovic Zammit






_______________________________________________
discuss mailing list
discuss@xxxxxxxxxxxx
http://lists.corosync.org/mailman/listinfo/discuss

[Index of Archives]     [Linux Clusters]     [Corosync Project]     [Linux USB Devel]     [Linux Audio Users]     [Photo]     [Yosemite News]    [Yosemite Photos]    [Linux Kernel]     [Linux SCSI]     [X.Org]

  Powered by Linux