2 node cluster crashes after a fixed time repeatedly

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



I have a 2 node cluster running version 2.2.0 of the cluster engine.

# corosync -v
Corosync Cluster Engine, version '2.2.0.7-e1e8f'
Copyright (c) 2006-2009 Red Hat, Inc.
#

When the two nodes are started, the cluster forms somewhat reliably every time.

The trouble is that when the cluster is up which I see using the cpgtool:

# corosync-cpgtool
Group Name           PID       Node ID
cabal.1-wb.7-b03478afa1f9f9f0b0f6bf18268d7a7507f49ac3-posixlock
             28933     833136832 (192.168.168.49)
             27353    2108205248 (192.168.168.125)
cabal.1-wb.7-b03478afa1f9f9f0b0f6bf18268d7a7507f49ac3-gc
             28933     833136832 (192.168.168.49)
             27353    2108205248 (192.168.168.125)
cabal.1-wb.7-b03478afa1f9f9f0b0f6bf18268d7a7507f49ac3-ringstate
             27454     833136832 (192.168.168.49)
             27353    2108205248 (192.168.168.125)
             26428    2108205248 (192.168.168.125)
             26411    2108205248 (192.168.168.125)
             26408    2108205248 (192.168.168.125)
             26405    2108205248 (192.168.168.125)
             26394    2108205248 (192.168.168.125)
             26367    2108205248 (192.168.168.125)
             26365    2108205248 (192.168.168.125)
             26363    2108205248 (192.168.168.125)
             28933     833136832 (192.168.168.49)
             27521     833136832 (192.168.168.49)
             27515     833136832 (192.168.168.49)
             27502     833136832 (192.168.168.49)
             27499     833136832 (192.168.168.49)
             27496     833136832 (192.168.168.49)
             27485     833136832 (192.168.168.49)
             27458     833136832 (192.168.168.49)
             27456     833136832 (192.168.168.49)

After a while (typically ~4 minutes) the two nodes end up not being able to talk to each other and each node tries to do request a rentransmit of missing messages. Finally both of them give up and form their own cluster.

However, running a tcpdump when the nodes are up,  shows that packets are flowing between the nodes. Comparing the timestamps of the corosync log with the tcpdump log, you see that while the tcpdump shows the UDP packets flowing between the nodes, the corosync log still keeps spinning in the retransmit loop
before finally the orf_token timer firing and declaring the other node dead.

Is this is a known bug or am I doing something wrong?

Both the node run on a MIPS processor. The underlyinh kernel is a stock Debian 3.5.7 kernel built for the mips arch. Kernelwise there hasnt been much tweaks done. 

The node are named oneblox-0082.local and oneblox-0083.local

Corosync log of oneblox-0082.local:
<snip>
Apr 03 10:39:41 notice  [TOTEM ] totemsrp.c:2614 Retransmit List: 1389 138a
Apr 03 10:39:42 notice  [TOTEM ] totemsrp.c:2614 Retransmit List: 1389 138a
Apr 03 10:39:42 notice  [TOTEM ] totemsrp.c:2614 Retransmit List: 1389 138a
Apr 03 10:39:42 notice  [TOTEM ] totemsrp.c:2614 Retransmit List: 1389 138a
Apr 03 10:39:42 notice  [TOTEM ] totemsrp.c:2614 Retransmit List: 1389 138a
Apr 03 10:39:42 notice  [TOTEM ] totemsrp.c:2614 Retransmit List: 1389 138a
Apr 03 10:39:43 notice  [TOTEM ] totemsrp.c:2614 Retransmit List: 1389 138a
Apr 03 10:39:43 notice  [TOTEM ] totemsrp.c:2614 Retransmit List: 1389 138a
Apr 03 10:39:43 notice  [TOTEM ] totemsrp.c:2614 Retransmit List: 1389 138a
Apr 03 10:39:43 notice  [TOTEM ] totemsrp.c:2614 Retransmit List: 1389 138a
Apr 03 10:39:43 notice  [TOTEM ] totemsrp.c:2614 Retransmit List: 1389 138a
Apr 03 10:39:43 notice  [TOTEM ] totemsrp.c:2614 Retransmit List: 1389 138a
Apr 03 10:39:44 notice  [TOTEM ] totemsrp.c:2614 Retransmit List: 1389 138a
Apr 03 10:39:44 notice  [TOTEM ] totemsrp.c:2614 Retransmit List: 1389 138a
Apr 03 10:39:44 notice  [TOTEM ] totemsrp.c:2614 Retransmit List: 1389 138a
Apr 03 10:39:44 notice  [TOTEM ] totemsrp.c:2614 Retransmit List: 1389 138a
Apr 03 10:39:44 notice  [TOTEM ] totemsrp.c:2614 Retransmit List: 1389 138a
Apr 03 10:39:44 error   [TOTEM ] totemsrp.c:3688 FAILED TO RECEIVE
Apr 03 10:39:46 notice  [TOTEM ] totemsrp.c:1940 A processor joined or left the membership and a new membership (192.168.168.49:55916) was formed.
Apr 03 10:39:46 notice  [MAIN  ] main.c:275 Completed service synchronization, ready to provide service.



tcpdump on oneblox-0082:
10:39:19.278212 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 338
10:39:19.278242 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:19.485450 IP oneblox-0082.local.2934 > oneblox-0083.local.2935: UDP, length 312
10:39:19.485490 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 87
10:39:19.486088 IP oneblox-0083.local.2934 > oneblox-0082.local.2935: UDP, length 142
10:39:19.486199 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 338
10:39:19.486225 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:19.693451 IP oneblox-0082.local.2934 > oneblox-0083.local.2935: UDP, length 312
10:39:19.693489 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 87
10:39:19.694079 IP oneblox-0083.local.2934 > oneblox-0082.local.2935: UDP, length 142
10:39:19.694199 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 338
10:39:19.694230 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:19.901451 IP oneblox-0082.local.2934 > oneblox-0083.local.2935: UDP, length 312
10:39:19.901490 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 87
10:39:19.902083 IP oneblox-0083.local.2934 > oneblox-0082.local.2935: UDP, length 142
10:39:19.902205 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 338
10:39:19.902235 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:20.109451 IP oneblox-0082.local.2934 > oneblox-0083.local.2935: UDP, length 312
10:39:20.109490 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 87
10:39:20.110083 IP oneblox-0083.local.2934 > oneblox-0082.local.2935: UDP, length 142
10:39:20.110204 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 338
10:39:20.110235 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:20.317450 IP oneblox-0082.local.2934 > oneblox-0083.local.2935: UDP, length 312
10:39:20.317488 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 87
10:39:20.318088 IP oneblox-0083.local.2934 > oneblox-0082.local.2935: UDP, length 142
10:39:20.318209 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 338
10:39:20.318240 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:20.525450 IP oneblox-0082.local.2934 > oneblox-0083.local.2935: UDP, length 312
10:39:20.525489 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 87
10:39:20.526086 IP oneblox-0083.local.2934 > oneblox-0082.local.2935: UDP, length 142
10:39:20.526197 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 338
10:39:20.526223 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:20.713465 IP oneblox-0082.local.2934 > oneblox-0083.local.2935: UDP, length 312
10:39:20.714128 IP oneblox-0083.local.2934 > oneblox-0082.local.2935: UDP, length 142
10:39:20.714242 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 338
10:39:20.714271 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:20.901452 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 87
10:39:20.901493 IP oneblox-0082.local.2934 > oneblox-0083.local.2935: UDP, length 312
10:39:20.902122 IP oneblox-0083.local.2934 > oneblox-0082.local.2935: UDP, length 142
10:39:20.902236 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 338
10:39:20.902264 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:21.093458 IP oneblox-0082.local.2934 > oneblox-0083.local.2935: UDP, length 312
10:39:21.094086 IP oneblox-0083.local.2934 > oneblox-0082.local.2935: UDP, length 142
10:39:21.094198 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 338
10:39:21.094227 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:21.281451 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 87
10:39:21.281489 IP oneblox-0082.local.2934 > oneblox-0083.local.2935: UDP, length 312
10:39:21.282116 IP oneblox-0083.local.2934 > oneblox-0082.local.2935: UDP, length 142
10:39:21.282225 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 338
10:39:21.282254 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:21.473460 IP oneblox-0082.local.2934 > oneblox-0083.local.2935: UDP, length 312
10:39:21.474093 IP oneblox-0083.local.2934 > oneblox-0082.local.2935: UDP, length 142
10:39:21.474204 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 338
10:39:21.474233 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:21.665451 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 87
10:39:21.665490 IP oneblox-0082.local.2934 > oneblox-0083.local.2935: UDP, length 312
10:39:21.666028 IP oneblox-0083.local.2934 > oneblox-0082.local.2935: UDP, length 142
10:39:21.666140 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 338
10:39:21.666169 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:21.857462 IP oneblox-0082.local.2934 > oneblox-0083.local.2935: UDP, length 312
10:39:21.858003 IP oneblox-0083.local.2934 > oneblox-0082.local.2935: UDP, length 142
10:39:21.858117 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 338
10:39:21.858147 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:22.049451 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 87
10:39:22.049491 IP oneblox-0082.local.2934 > oneblox-0083.local.2935: UDP, length 312
10:39:22.050042 IP oneblox-0083.local.2934 > oneblox-0082.local.2935: UDP, length 142
10:39:22.050155 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 338
10:39:22.050184 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:22.241457 IP oneblox-0082.local.2934 > oneblox-0083.local.2935: UDP, length 312
10:39:22.242419 IP oneblox-0083.local.2934 > oneblox-0082.local.2935: UDP, length 142
10:39:22.242524 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 338
10:39:22.242553 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:22.261456 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 87
10:39:22.489450 IP oneblox-0082.local.2934 > oneblox-0083.local.2935: UDP, length 312
10:39:22.489507 IP oneblox-0082.local.2934 > oneblox-0083.local.2935: UDP, length 312
10:39:22.490064 IP oneblox-0083.local.2934 > oneblox-0082.local.2935: UDP, length 142
10:39:22.490179 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 338
10:39:22.490208 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:22.681461 IP oneblox-0082.local.2934 > oneblox-0083.local.2935: UDP, length 312
10:39:22.682055 IP oneblox-0083.local.2934 > oneblox-0082.local.2935: UDP, length 142
10:39:22.682167 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 338
10:39:22.682196 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:22.873452 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 87
10:39:22.873509 IP oneblox-0082.local.2934 > oneblox-0083.local.2935: UDP, length 312
10:39:22.874101 IP oneblox-0083.local.2934 > oneblox-0082.local.2935: UDP, length 142
10:39:22.874212 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 338
10:39:22.874241 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:23.065458 IP oneblox-0082.local.2934 > oneblox-0083.local.2935: UDP, length 312
10:39:23.066044 IP oneblox-0083.local.2934 > oneblox-0082.local.2935: UDP, length 142
10:39:23.066153 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 338
10:39:23.066181 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:23.257450 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 87
10:39:23.257506 IP oneblox-0082.local.2934 > oneblox-0083.local.2935: UDP, length 312
10:39:23.258095 IP oneblox-0083.local.2934 > oneblox-0082.local.2935: UDP, length 142
10:39:23.258207 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 338
10:39:23.258236 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:23.449459 IP oneblox-0082.local.2934 > oneblox-0083.local.2935: UDP, length 312
10:39:23.450048 IP oneblox-0083.local.2934 > oneblox-0082.local.2935: UDP, length 142
10:39:23.450158 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 338
10:39:23.450187 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:23.617471 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 87
10:39:23.641455 IP oneblox-0082.local.2934 > oneblox-0083.local.2935: UDP, length 312
10:39:23.642081 IP oneblox-0083.local.2934 > oneblox-0082.local.2935: UDP, length 142
10:39:23.642186 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 338
10:39:23.642215 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:23.849451 IP oneblox-0082.local.2934 > oneblox-0083.local.2935: UDP, length 312
10:39:23.849509 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 87
10:39:23.850090 IP oneblox-0083.local.2934 > oneblox-0082.local.2935: UDP, length 142
10:39:23.850196 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 338
10:39:23.850224 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:24.061448 IP oneblox-0082.local.2934 > oneblox-0083.local.2935: UDP, length 312
10:39:24.061503 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 87
10:39:24.062094 IP oneblox-0083.local.2934 > oneblox-0082.local.2935: UDP, length 142
10:39:24.062199 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 338
10:39:24.062227 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:24.269449 IP oneblox-0082.local.2934 > oneblox-0083.local.2935: UDP, length 312
10:39:24.269542 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 87
10:39:24.270079 IP oneblox-0083.local.2934 > oneblox-0082.local.2935: UDP, length 142
10:39:24.270183 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 338
10:39:24.270212 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:24.481448 IP oneblox-0082.local.2934 > oneblox-0083.local.2935: UDP, length 312
10:39:24.481504 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 87
10:39:24.482084 IP oneblox-0083.local.2934 > oneblox-0082.local.2935: UDP, length 142
10:39:24.482190 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 338
10:39:24.482218 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:24.693448 IP oneblox-0082.local.2934 > oneblox-0083.local.2935: UDP, length 312
10:39:24.693504 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 87
10:39:24.694077 IP oneblox-0083.local.2934 > oneblox-0082.local.2935: UDP, length 142
10:39:24.694182 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 338
10:39:24.694211 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:24.905448 IP oneblox-0082.local.2934 > oneblox-0083.local.2935: UDP, length 312
10:39:24.905505 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 87
10:39:24.906076 IP oneblox-0083.local.2934 > oneblox-0082.local.2935: UDP, length 142
10:39:24.906181 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 338
10:39:24.906209 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:25.117449 IP oneblox-0082.local.2934 > oneblox-0083.local.2935: UDP, length 312
10:39:25.117503 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 87
10:39:25.118084 IP oneblox-0083.local.2934 > oneblox-0082.local.2935: UDP, length 142
10:39:25.118203 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 338
10:39:25.118232 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:25.329449 IP oneblox-0082.local.2934 > oneblox-0083.local.2935: UDP, length 312
10:39:25.329506 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 87
10:39:25.330137 IP oneblox-0083.local.2934 > oneblox-0082.local.2935: UDP, length 142
10:39:25.330246 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 338
10:39:25.330275 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:25.541460 IP oneblox-0082.local.2934 > oneblox-0083.local.2935: UDP, length 312
10:39:25.541531 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 87
10:39:25.542096 IP oneblox-0083.local.2934 > oneblox-0082.local.2935: UDP, length 142
10:39:25.542242 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 338
10:39:25.542270 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:25.753463 IP oneblox-0082.local.2934 > oneblox-0083.local.2935: UDP, length 312
10:39:25.753537 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 87
10:39:25.754140 IP oneblox-0083.local.2934 > oneblox-0082.local.2935: UDP, length 142
10:39:25.754284 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 338
10:39:25.754314 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:25.965459 IP oneblox-0082.local.2934 > oneblox-0083.local.2935: UDP, length 312
10:39:25.965524 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 87
10:39:25.966089 IP oneblox-0083.local.2934 > oneblox-0082.local.2935: UDP, length 142
10:39:25.966228 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 338
10:39:25.966258 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:26.177449 IP oneblox-0082.local.2934 > oneblox-0083.local.2935: UDP, length 312
10:39:26.177505 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 87
10:39:26.178087 IP oneblox-0083.local.2934 > oneblox-0082.local.2935: UDP, length 142
10:39:26.178193 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 338
10:39:26.178222 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:26.389449 IP oneblox-0082.local.2934 > oneblox-0083.local.2935: UDP, length 312
10:39:26.389507 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 87
10:39:26.390081 IP oneblox-0083.local.2934 > oneblox-0082.local.2935: UDP, length 142
10:39:26.390188 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 338
10:39:26.390217 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:26.601531 IP oneblox-0082.local.2934 > oneblox-0083.local.2935: UDP, length 312
10:39:26.601591 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 87
10:39:26.602164 IP oneblox-0083.local.2934 > oneblox-0082.local.2935: UDP, length 142
10:39:26.602273 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 338
10:39:26.602301 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:26.813452 IP oneblox-0082.local.2934 > oneblox-0083.local.2935: UDP, length 312
10:39:26.813513 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 87
10:39:26.814103 IP oneblox-0083.local.2934 > oneblox-0082.local.2935: UDP, length 142
10:39:26.814217 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 338
10:39:26.814246 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:27.025448 IP oneblox-0082.local.2934 > oneblox-0083.local.2935: UDP, length 312
10:39:27.025551 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 87
10:39:27.026076 IP oneblox-0083.local.2934 > oneblox-0082.local.2935: UDP, length 142
10:39:27.026177 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 338
10:39:27.026206 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:27.237454 IP oneblox-0082.local.2934 > oneblox-0083.local.2935: UDP, length 312
10:39:27.237512 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 87
10:39:27.238089 IP oneblox-0083.local.2934 > oneblox-0082.local.2935: UDP, length 142
10:39:27.238197 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 338
10:39:27.238225 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:27.449449 IP oneblox-0082.local.2934 > oneblox-0083.local.2935: UDP, length 312
10:39:27.449508 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 87
10:39:27.450084 IP oneblox-0083.local.2934 > oneblox-0082.local.2935: UDP, length 142
10:39:27.450193 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 338
10:39:27.450222 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:27.661449 IP oneblox-0082.local.2934 > oneblox-0083.local.2935: UDP, length 312
10:39:27.661507 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 87
10:39:27.662086 IP oneblox-0083.local.2934 > oneblox-0082.local.2935: UDP, length 142
10:39:27.662191 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 338
10:39:27.662219 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:27.873448 IP oneblox-0082.local.2934 > oneblox-0083.local.2935: UDP, length 312
10:39:27.873505 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 87
10:39:27.874090 IP oneblox-0083.local.2934 > oneblox-0082.local.2935: UDP, length 142
10:39:27.874195 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 338
10:39:27.874223 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:28.085448 IP oneblox-0082.local.2934 > oneblox-0083.local.2935: UDP, length 312
10:39:28.085503 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 87
10:39:28.086077 IP oneblox-0083.local.2934 > oneblox-0082.local.2935: UDP, length 142
10:39:28.086180 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 338
10:39:28.086209 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:28.277880 IP oneblox-0082.local.2934 > oneblox-0083.local.2935: UDP, length 312
10:39:28.278517 IP oneblox-0083.local.2934 > oneblox-0082.local.2935: UDP, length 142
10:39:28.278625 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 338
10:39:28.278654 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:28.317448 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 87
10:39:28.473457 IP oneblox-0082.local.2934 > oneblox-0083.local.2935: UDP, length 312
10:39:28.474095 IP oneblox-0083.local.2934 > oneblox-0082.local.2935: UDP, length 142
10:39:28.474202 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 338
10:39:28.474230 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:28.685449 IP oneblox-0082.local.2934 > oneblox-0083.local.2935: UDP, length 312
10:39:28.685506 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 87
10:39:28.686079 IP oneblox-0083.local.2934 > oneblox-0082.local.2935: UDP, length 142
10:39:28.686186 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 338
10:39:28.686215 IP oneblox-0082.local.2934 > 239.244.154.195.2935: UDP, length 410


Corosync log on oneblox-0083:

Apr 03 10:39:42 notice  [TOTEM ] totemsrp.c:2614 Retransmit List: 1383 1384 1385 1386 1387 1388 138b
Apr 03 10:39:42 notice  [TOTEM ] totemsrp.c:2614 Retransmit List: 1383 1384 1385 1386 1387 1388 138b
Apr 03 10:39:43 notice  [TOTEM ] totemsrp.c:2614 Retransmit List: 1383 1384 1385 1386 1387 1388 138b
Apr 03 10:39:43 notice  [TOTEM ] totemsrp.c:2614 Retransmit List: 1383 1384 1385 1386 1387 1388 138b
Apr 03 10:39:43 notice  [TOTEM ] totemsrp.c:2614 Retransmit List: 1383 1384 1385 1386 1387 1388 138b
Apr 03 10:39:43 notice  [TOTEM ] totemsrp.c:2614 Retransmit List: 1383 1384 1385 1386 1387 1388 138b
Apr 03 10:39:43 notice  [TOTEM ] totemsrp.c:2614 Retransmit List: 1383 1384 1385 1386 1387 1388 138b
Apr 03 10:39:43 notice  [TOTEM ] totemsrp.c:2614 Retransmit List: 1383 1384 1385 1386 1387 1388 138b
Apr 03 10:39:44 notice  [TOTEM ] totemsrp.c:2614 Retransmit List: 1383 1384 1385 1386 1387 1388 138b
Apr 03 10:39:44 notice  [TOTEM ] totemsrp.c:2614 Retransmit List: 1383 1384 1385 1386 1387 1388 138b
Apr 03 10:39:44 notice  [TOTEM ] totemsrp.c:2614 Retransmit List: 1383 1384 1385 1386 1387 1388 138b
Apr 03 10:39:44 notice  [TOTEM ] totemsrp.c:2614 Retransmit List: 1383 1384 1385 1386 1387 1388 138b
Apr 03 10:39:44 notice  [TOTEM ] totemsrp.c:2614 Retransmit List: 1383 1384 1385 1386 1387 1388 138b
Apr 03 10:39:45 notice  [TOTEM ] totemsrp.c:1639 A processor failed, forming new configuration.
Apr 03 10:39:46 notice  [TOTEM ] totemsrp.c:1940 A processor joined or left the membership and a new membership (192.168.168.125:55916) was formed.
Apr 03 10:39:46 notice  [MAIN  ] main.c:275 Completed service synchronization, ready to provide service.
Apr 03 10:39:47 notice  [TOTEM ] totemsrp.c:1940 A processor joined or left the membership and a new membership (192.168.168.125:55920) was formed.
Apr 03 10:39:47 notice  [MAIN  ] main.c:275 Completed service synchronization, ready to provide service.
Apr 03 10:39:49 notice  [TOTEM ] totemsrp.c:1940 A processor joined or left the membership and a new membership (192.168.168.125:55924) was formed.


tcpdump log of 0083.local

10:39:27.445941 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 382
10:39:27.445990 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 470
10:39:27.446035 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 422
10:39:27.446081 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 749
10:39:27.446167 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 338
10:39:27.446213 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:27.446260 IP oneblox-0083.local.2934 > oneblox-0082.local.2935: UDP, length 142
10:39:27.657794 IP oneblox-0082.local.2934 > oneblox-0083.local.2935: UDP, length 312
10:39:27.657911 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:27.657940 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 382
10:39:27.657989 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 470
10:39:27.658036 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 422
10:39:27.658083 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 749
10:39:27.658169 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 338
10:39:27.658215 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:27.658261 IP oneblox-0083.local.2934 > oneblox-0082.local.2935: UDP, length 142
10:39:27.869796 IP oneblox-0082.local.2934 > oneblox-0083.local.2935: UDP, length 312
10:39:27.869915 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:27.869944 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 382
10:39:27.869993 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 470
10:39:27.870038 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 422
10:39:27.870084 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 749
10:39:27.870171 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 338
10:39:27.870218 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:27.870265 IP oneblox-0083.local.2934 > oneblox-0082.local.2935: UDP, length 142
10:39:28.081794 IP oneblox-0082.local.2934 > oneblox-0083.local.2935: UDP, length 312
10:39:28.081902 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:28.081931 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 382
10:39:28.081981 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 470
10:39:28.082028 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 422
10:39:28.082073 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 749
10:39:28.082159 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 338
10:39:28.082205 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:28.082252 IP oneblox-0083.local.2934 > oneblox-0082.local.2935: UDP, length 142
10:39:28.274226 IP oneblox-0082.local.2934 > oneblox-0083.local.2935: UDP, length 312
10:39:28.274335 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:28.274364 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 382
10:39:28.274422 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 470
10:39:28.274469 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 422
10:39:28.274515 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 749
10:39:28.274600 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 338
10:39:28.274646 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:28.274692 IP oneblox-0083.local.2934 > oneblox-0082.local.2935: UDP, length 142
10:39:28.469802 IP oneblox-0082.local.2934 > oneblox-0083.local.2935: UDP, length 312
10:39:28.469919 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:28.469948 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 382
10:39:28.469997 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 470
10:39:28.470044 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 422
10:39:28.470090 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 749
10:39:28.470176 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 338
10:39:28.470223 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:28.470269 IP oneblox-0083.local.2934 > oneblox-0082.local.2935: UDP, length 142
10:39:28.681794 IP oneblox-0082.local.2934 > oneblox-0083.local.2935: UDP, length 312
10:39:28.681903 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:28.681932 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 382
10:39:28.681981 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 470
10:39:28.682027 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 422
10:39:28.682073 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 749
10:39:28.682162 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 338
10:39:28.682208 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:28.682254 IP oneblox-0083.local.2934 > oneblox-0082.local.2935: UDP, length 142
10:39:28.893791 IP oneblox-0082.local.2934 > oneblox-0083.local.2935: UDP, length 312
10:39:28.893903 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:28.893933 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 382
10:39:28.893982 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 470
10:39:28.894028 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 422
10:39:28.894074 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 749
10:39:28.894159 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 338
10:39:28.894205 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:28.894252 IP oneblox-0083.local.2934 > oneblox-0082.local.2935: UDP, length 142
10:39:29.101793 IP oneblox-0082.local.2934 > oneblox-0083.local.2935: UDP, length 312
10:39:29.101908 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:29.101937 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 382
10:39:29.101987 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 470
10:39:29.102033 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 422
10:39:29.102080 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 749
10:39:29.102165 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 338
10:39:29.102211 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:29.102257 IP oneblox-0083.local.2934 > oneblox-0082.local.2935: UDP, length 142
10:39:29.309788 IP oneblox-0082.local.2934 > oneblox-0083.local.2935: UDP, length 312
10:39:29.309907 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:29.309937 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 382
10:39:29.309986 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 470
10:39:29.310031 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 422
10:39:29.310078 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 749
10:39:29.310165 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 338
10:39:29.310211 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:29.310257 IP oneblox-0083.local.2934 > oneblox-0082.local.2935: UDP, length 142
10:39:29.517786 IP oneblox-0082.local.2934 > oneblox-0083.local.2935: UDP, length 312
10:39:29.517898 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:29.517927 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 382
10:39:29.517975 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 470
10:39:29.518021 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 422
10:39:29.518106 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 749
10:39:29.518154 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 338
10:39:29.518200 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:29.518247 IP oneblox-0083.local.2934 > oneblox-0082.local.2935: UDP, length 142
10:39:29.725788 IP oneblox-0082.local.2934 > oneblox-0083.local.2935: UDP, length 312
10:39:29.725905 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:29.725934 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 382
10:39:29.725983 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 470
10:39:29.726029 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 422
10:39:29.726115 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 749
10:39:29.726163 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 338
10:39:29.726208 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:29.726254 IP oneblox-0083.local.2934 > oneblox-0082.local.2935: UDP, length 142
10:39:29.933787 IP oneblox-0082.local.2934 > oneblox-0083.local.2935: UDP, length 312
10:39:29.933896 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:29.933926 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 382
10:39:29.933975 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 470
10:39:29.934021 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 422
10:39:29.934107 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 749
10:39:29.934155 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 338
10:39:29.934201 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:29.934247 IP oneblox-0083.local.2934 > oneblox-0082.local.2935: UDP, length 142
10:39:30.141787 IP oneblox-0082.local.2934 > oneblox-0083.local.2935: UDP, length 312
10:39:30.141897 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:30.141926 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 382
10:39:30.141976 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 470
10:39:30.142022 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 422
10:39:30.142108 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 749
10:39:30.142156 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 338
10:39:30.142201 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:30.142247 IP oneblox-0083.local.2934 > oneblox-0082.local.2935: UDP, length 142
10:39:30.349787 IP oneblox-0082.local.2934 > oneblox-0083.local.2935: UDP, length 312
10:39:30.349895 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:30.349925 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 382
10:39:30.349973 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 470
10:39:30.350019 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 422
10:39:30.350113 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 749
10:39:30.350160 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 338
10:39:30.350206 IP oneblox-0083.local.2934 > 239.244.154.195.2935: UDP, length 410
10:39:30.350251 IP oneblox-0083.local.2934 > oneblox-0082.local.2935: UDP, length 142
root@oneblox-0083:~#

 

Thanks
Koushik
_______________________________________________
discuss mailing list
discuss@xxxxxxxxxxxx
http://lists.corosync.org/mailman/listinfo/discuss

[Index of Archives]     [Linux Clusters]     [Corosync Project]     [Linux USB Devel]     [Linux Audio Users]     [Photo]     [Yosemite News]    [Yosemite Photos]    [Linux Kernel]     [Linux SCSI]     [X.Org]

  Powered by Linux