Hi Christine,
thanks for your input. I have checked and in the configuration with only one network I have debugging turned on as well (same corosync.conf files).
These messages are repeating every 1-2 seconds and the reason why I think there is something wrong is that if I do operation on a sqlite3 db on the GFS2 filesystem the operations are much slower when I have the secondary network as well (and the extra logging)
If I try to strace the sqlite3 command, it is stuck for few seconds (very similar to the period of the logging repeating) in a fcntl system call needed to lock the db file
________________________________________
From: linux-cluster-bounces@xxxxxxxxxx <linux-cluster-bounces@xxxxxxxxxx> on behalf of Christine Caulfield <ccaulfie@xxxxxxxxxx>
Sent: Tuesday, April 12, 2016 2:28 PM
To: linux-cluster@xxxxxxxxxx
Subject: Re: Help with corosync and GFS2 on multi network setup
On 12/04/16 13:45, Stefano Panella wrote:
> Hi everybody,
>
> we have been using corosync directly to provide clustering for GFS2 on our centos 7.2 pools with only one network interface and all has been working great so far!
>
> We now have a new set-up with two network interfaces for every host in the cluster:
> A -> 1 Gbit (the one we would like corosync to use, 10.220.88.X)
> B -> 10 Gbit (used for iscsi connection to storage, 10.220.246.X)
>
> when we run corosync in this mode we get the logs continuously spammed by messages like these:
>
> [12880] cl15-02 corosyncdebug [TOTEM ] entering GATHER state from 0(consensus timeout).
> [12880] cl15-02 corosyncdebug [TOTEM ] Creating commit token because I am the rep.
> [12880] cl15-02 corosyncdebug [TOTEM ] Saving state aru 10 high seq received 10
> [12880] cl15-02 corosyncdebug [MAIN ] Storing new sequence id for ring 5750
> [12880] cl15-02 corosyncdebug [TOTEM ] entering COMMIT state.
> [12880] cl15-02 corosyncdebug [TOTEM ] got commit token
> [12880] cl15-02 corosyncdebug [TOTEM ] entering RECOVERY state.
> [12880] cl15-02 corosyncdebug [TOTEM ] TRANS [0] member 10.220.88.41:
> [12880] cl15-02 corosyncdebug [TOTEM ] TRANS [1] member 10.220.88.47:
> [12880] cl15-02 corosyncdebug [TOTEM ] position [0] member 10.220.88.41:
> [12880] cl15-02 corosyncdebug [TOTEM ] previous ring seq 574c rep 10.220.88.41
> [12880] cl15-02 corosyncdebug [TOTEM ] aru 10 high delivered 10 received flag 1
> [12880] cl15-02 corosyncdebug [TOTEM ] position [1] member 10.220.88.47:
> [12880] cl15-02 corosyncdebug [TOTEM ] previous ring seq 574c rep 10.220.88.41
> [12880] cl15-02 corosyncdebug [TOTEM ] aru 10 high delivered 10 received flag 1
>
> [12880] cl15-02 corosyncdebug [TOTEM ] Did not need to originate any messages in recovery.
> [12880] cl15-02 corosyncdebug [TOTEM ] got commit token
> [12880] cl15-02 corosyncdebug [TOTEM ] Sending initial ORF token
> [12880] cl15-02 corosyncdebug [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 0, aru 0
> [12880] cl15-02 corosyncdebug [TOTEM ] install seq 0 aru 0 high seq received 0
> [12880] cl15-02 corosyncdebug [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 1, aru 0
> [12880] cl15-02 corosyncdebug [TOTEM ] install seq 0 aru 0 high seq received 0
> [12880] cl15-02 corosyncdebug [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 2, aru 0
> [12880] cl15-02 corosyncdebug [TOTEM ] install seq 0 aru 0 high seq received 0
> [12880] cl15-02 corosyncdebug [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 3, aru 0
> [12880] cl15-02 corosyncdebug [TOTEM ] install seq 0 aru 0 high seq received 0
> [12880] cl15-02 corosyncdebug [TOTEM ] retrans flag count 4 token aru 0 install seq 0 aru 0 0
> [12880] cl15-02 corosyncdebug [TOTEM ] Resetting old ring state
> [12880] cl15-02 corosyncdebug [TOTEM ] recovery to regular 1-0
> [12880] cl15-02 corosyncdebug [TOTEM ] waiting_trans_ack changed to 1
> Apr 11 16:19:54 [13372] cl15-02 pacemakerd: info: pcmk_quorum_notification: Membership 22352: quorum retained (2)
> Apr 11 16:19:54 [13378] cl15-02 crmd: info: pcmk_quorum_notification: Membership 22352: quorum retained (2)
> [12880] cl15-02 corosyncdebug [TOTEM ] entering OPERATIONAL state.
> [12880] cl15-02 corosyncnotice [TOTEM ] A new membership (10.220.88.41:22352) was formed. Members
> [12880] cl15-02 corosyncdebug [SYNC ] Committing synchronization for corosync configuration map access
> Apr 11 16:19:54 [13373] cl15-02 cib: info: cib_process_request: Forwarding cib_modify operation for section nodes to master (origin=local/crmd/27157)
> [12880] cl15-02 corosyncdebug [CMAP ] Not first sync -> no action
> Apr 11 16:19:54 [13373] cl15-02 cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/27158)
> [12880] cl15-02 corosyncdebug [CPG ] got joinlist message from node 0x2
> [12880] cl15-02 corosyncdebug [CPG ] comparing: sender r(0) ip(10.220.88.41) ; members(old:2 left:0)
> [12880] cl15-02 corosyncdebug [CPG ] comparing: sender r(0) ip(10.220.88.47) ; members(old:2 left:0)
> [12880] cl15-02 corosyncdebug [CPG ] chosen downlist: sender r(0) ip(10.220.88.41) ; members(old:2 left:0)
> [12880] cl15-02 corosyncdebug [CPG ] got joinlist message from node 0x1
> [12880] cl15-02 corosyncdebug [SYNC ] Committing synchronization for corosync cluster closed process group service v1.01
> Apr 11 16:19:54 [13373] cl15-02 cib: info: cib_process_request: Completed cib_modify operation for section nodes: OK (rc=0, origin=cl15-02/crmd/27157, version=0.18.22)
> [12880] cl15-02 corosyncdebug [CPG ] joinlist_messages[0] group:clvmd, ip:r(0) ip(10.220.88.41) , pid:35677
> Apr 11 16:19:54 [13373] cl15-02 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=cl15-02/crmd/27158, version=0.18.22)
> [12880] cl15-02 corosyncdebug [CPG ] joinlist_messages[1] group:dlm:ls:clvmd\x00, ip:r(0) ip(10.220.88.41) , pid:34995
> [12880] cl15-02 corosyncdebug [CPG ] joinlist_messages[2] group:dlm:controld\x00, ip:r(0) ip(10.220.88.41) , pid:34995
> [12880] cl15-02 corosyncdebug [CPG ] joinlist_messages[3] group:crmd\x00, ip:r(0) ip(10.220.88.41) , pid:13378
> [12880] cl15-02 corosyncdebug [CPG ] joinlist_messages[4] group:attrd\x00, ip:r(0) ip(10.220.88.41) , pid:13376
> [12880] cl15-02 corosyncdebug [CPG ] joinlist_messages[5] group:stonith-ng\x00, ip:r(0) ip(10.220.88.41) , pid:13374
> [12880] cl15-02 corosyncdebug [CPG ] joinlist_messages[6] group:cib\x00, ip:r(0) ip(10.220.88.41) , pid:13373
> [12880] cl15-02 corosyncdebug [CPG ] joinlist_messages[7] group:pacemakerd\x00, ip:r(0) ip(10.220.88.41) , pid:13372
> [12880] cl15-02 corosyncdebug [CPG ] joinlist_messages[8] group:crmd\x00, ip:r(0) ip(10.220.88.47) , pid:12879
> [12880] cl15-02 corosyncdebug [CPG ] joinlist_messages[9] group:attrd\x00, ip:r(0) ip(10.220.88.47) , pid:12877
> [12880] cl15-02 corosyncdebug [CPG ] joinlist_messages[10] group:stonith-ng\x00, ip:r(0) ip(10.220.88.47) , pid:12875
> [12880] cl15-02 corosyncdebug [CPG ] joinlist_messages[11] group:cib\x00, ip:r(0) ip(10.220.88.47) , pid:12874
> [12880] cl15-02 corosyncdebug [CPG ] joinlist_messages[12] group:pacemakerd\x00, ip:r(0) ip(10.220.88.47) , pid:12873
> [12880] cl15-02 corosyncdebug [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
> [12880] cl15-02 corosyncdebug [VOTEQ ] got nodeinfo message from cluster node 1
> [12880] cl15-02 corosyncdebug [VOTEQ ] nodeinfo message[1]: votes: 1, expected: 3 flags: 1
> [12880] cl15-02 corosyncdebug [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
> [12880] cl15-02 corosyncdebug [VOTEQ ] total_votes=2, expected_votes=3
> [12880] cl15-02 corosyncdebug [VOTEQ ] node 1 state=1, votes=1, expected=3
> [12880] cl15-02 corosyncdebug [VOTEQ ] node 2 state=1, votes=1, expected=3
> [12880] cl15-02 corosyncdebug [VOTEQ ] node 3 state=2, votes=1, expected=3
> [12880] cl15-02 corosyncdebug [VOTEQ ] lowest node id: 1 us: 1
> [12880] cl15-02 corosyncdebug [VOTEQ ] highest node id: 2 us: 1
> [12880] cl15-02 corosyncdebug [VOTEQ ] got nodeinfo message from cluster node 1
> [12880] cl15-02 corosyncdebug [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
> [12880] cl15-02 corosyncdebug [VOTEQ ] got nodeinfo message from cluster node 2
> [12880] cl15-02 corosyncdebug [VOTEQ ] nodeinfo message[2]: votes: 1, expected: 3 flags: 1
> [12880] cl15-02 corosyncdebug [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
> [12880] cl15-02 corosyncdebug [VOTEQ ] got nodeinfo message from cluster node 2
> [12880] cl15-02 corosyncdebug [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
> [12880] cl15-02 corosyncdebug [SYNC ] Committing synchronization for corosync vote quorum service v1.0
> [12880] cl15-02 corosyncdebug [VOTEQ ] total_votes=2, expected_votes=3
> [12880] cl15-02 corosyncdebug [VOTEQ ] node 1 state=1, votes=1, expected=3
> [12880] cl15-02 corosyncdebug [VOTEQ ] node 2 state=1, votes=1, expected=3
> [12880] cl15-02 corosyncdebug [VOTEQ ] node 3 state=2, votes=1, expected=3
> [12880] cl15-02 corosyncdebug [VOTEQ ] lowest node id: 1 us: 1
> [12880] cl15-02 corosyncdebug [VOTEQ ] highest node id: 2 us: 1
> [12880] cl15-02 corosyncnotice [QUORUM] Members[2]: 1 2
> [12880] cl15-02 corosyncdebug [QUORUM] sending quorum notification to (nil), length = 56
> [12880] cl15-02 corosyncnotice [MAIN ] Completed service synchronization, ready to provide service.
> [12880] cl15-02 corosyncdebug [TOTEM ] waiting_trans_ack changed to 0
> [12880] cl15-02 corosyncdebug [QUORUM] got quorate request on 0x7f5a907749a0
> [12880] cl15-02 corosyncdebug [TOTEM ] entering GATHER state from 11(merge during join).
>
>
> and we do not get them when there is only a single network interface in the systems.
>
> --------------------------------------------------------------------------------------
> These are the network configurations on the three hosts:
>
> [root@cl15-02 ~]# ifconfig | grep inet
> inet 10.220.88.41 netmask 255.255.248.0 broadcast 10.220.95.255
> inet 10.220.246.50 netmask 255.255.255.0 broadcast 10.220.246.255
> inet 127.0.0.1 netmask 255.0.0.0
>
> [root@cl15-08 ~]# ifconfig | grep inet
> inet 10.220.88.47 netmask 255.255.248.0 broadcast 10.220.95.255
> inet 10.220.246.51 netmask 255.255.255.0 broadcast 10.220.246.255
> inet 127.0.0.1 netmask 255.0.0.0
>
> [root@cl15-09 ~]# ifconfig | grep inet
> inet 10.220.88.48 netmask 255.255.248.0 broadcast 10.220.95.255
> inet 10.220.246.59 netmask 255.255.255.0 broadcast 10.220.246.255
> inet 127.0.0.1 netmask 255.0.0.0
>
> -----------------------------------------------------------------------------------
> corosync-quorumtool output:
>
> [root@cl15-02 ~]# corosync-quorumtool
> Quorum information
> ------------------
> Date: Mon Apr 11 15:46:26 2016
> Quorum provider: corosync_votequorum
> Nodes: 3
> Node ID: 1
> Ring ID: 18952
> Quorate: Yes
>
> Votequorum information
> ----------------------
> Expected votes: 3
> Highest expected: 3
> Total votes: 3
> Quorum: 2
> Flags: Quorate
>
> Membership information
> ----------------------
> Nodeid Votes Name
> 1 1 cl15-02 (local)
> 2 1 cl15-08
> 3 1 cl15-09
>
> ---------------------------------------------------------------------------
> /etc/corosync/corosync.conf:
>
> [root@cl15-02 ~]# cat /etc/corosync/corosync.conf
> totem {
> version: 2
> secauth: off
> cluster_name: gfs_cluster
> transport: udpu
> }
>
> nodelist {
> node {
> ring0_addr: cl15-02
> nodeid: 1
> }
>
> node {
> ring0_addr: cl15-08
> nodeid: 2
> }
>
> node {
> ring0_addr: cl15-09
> nodeid: 3
> }
> }
>
> quorum {
> provider: corosync_votequorum
> }
>
> logging {
> debug: on
You have debug logging on. At a guess I would say that the config file
with the other interface in it doesn't :)
Chrissie
> to_logfile: yes
> logfile: /var/log/cluster/corosync.log
> to_syslog: yes
> }
>
--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster
--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster
