On Tue, Apr 3, 2012 at 10:48 AM, Unknown Asker
<unknown.asker.2012@xxxxxxxxx> wrote:
> In the name of Allah
>
> Hello
> I installed Scientific Linux 6 kernel version 2.6.32-71.el6.x86_64 as
> minimal and then compiled corosync 1.4.2 with
> gcc 4.4.4
>
> I started corosync on two VMs that are connected together with bridge.
> after about 22 hours the problem repeated.
>
> Please,help me how to solve this problem
> Also I have decided to test transport udpu and waiting for answer ...
>
> Thanks in advance
>
> ------------
> I configured corosync as follow:
>
> # Please read the corosync.conf.5 manual page
> compatibility: whitetank
>
> totem {
> version: 2
> token: 5000
> token_retransmits_before_loss_const: 20
> join: 1000
> consensus: 7500
> vsftype: none
> max_messages: 20
> secauth: off
> threads: 0
> nodeid: 1
> clear_node_high_bit: yes
> interface {
> ringnumber: 0
> bindnetaddr: 192.168.100.0
> mcastaddr: 226.94.1.1
> mcastport: 5405
> ttl: 1
> }
> }
>
> logging {
> fileline: off
> to_stderr: no
> to_logfile: yes
> to_syslog: yes
> logfile: /var/log/cluster/corosync.log
> debug: on
> timestamp: on
> logger_subsys {
> subsys: AMF
> debug: on
> }
> }
>
> amf {
> mode: disabled
> }
> "/etc/corosync/corosync.conf" 41L, 598C
> 31,2-9 Top
>
> ---------------
> logs:
>
> Apr 2 11:23:14 VM1 corosync[1086]: [MAIN ] Corosync Cluster Engine
> ('1.4.2'): started and ready to provide service.
> Apr 2 11:23:14 VM1 corosync[1086]: [MAIN ] Corosync built-in features:
> Apr 2 11:23:14 VM1 corosync[1086]: [MAIN ] Successfully read main
> configuration file '/etc/corosync/corosync.conf'.
> Apr 2 11:23:14 VM1 corosync[1086]: [TOTEM ] Initializing transport
> (UDP/IP Multicast).
> Apr 2 11:23:14 VM1 corosync[1086]: [TOTEM ] Initializing
> transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0).
> Apr 2 11:23:14 VM1 corosync[1086]: [TOTEM ] The network interface
> [192.168.100.1] is now up.
> Apr 2 11:23:14 VM1 corosync[1086]: [SERV ] Service engine loaded:
> corosync extended virtual synchrony service
> Apr 2 11:23:14 VM1 corosync[1086]: [SERV ] Service engine loaded:
> corosync configuration service
> Apr 2 11:23:14 VM1 corosync[1086]: [SERV ] Service engine loaded:
> corosync cluster closed process group service v1.01
> Apr 2 11:23:14 VM1 corosync[1086]: [SERV ] Service engine loaded:
> corosync cluster config database access v1.01
> Apr 2 11:23:14 VM1 corosync[1086]: [SERV ] Service engine loaded:
> corosync profile loading service
> Apr 2 11:23:14 VM1 corosync[1086]: [SERV ] Service engine loaded:
> corosync cluster quorum service v0.1
> Apr 2 11:23:14 VM1 corosync[1086]: [MAIN ] Compatibility mode set
> to whitetank. Using V1 and V2 of the synchronization engine.
> Apr 2 11:23:14 VM1 corosync[1086]: [TOTEM ] A processor joined or
> left the membership and a new membership was formed.
> Apr 2 11:23:14 VM1 corosync[1086]: [CPG ] chosen downlist: sender
> r(0) ip(192.168.100.1) ; members(old:0 left:0)
> Apr 2 11:23:14 VM1 corosync[1086]: [MAIN ] Completed service
> synchronization, ready to provide service.
> Apr 2 11:39:04 VM1 corosync[1086]: [TOTEM ] A processor joined or
> left the membership and a new membership was formed.
> Apr 2 11:39:04 VM1 corosync[1086]: [CPG ] chosen downlist: sender
> r(0) ip(192.168.100.1) ; members(old:1 left:0)
> Apr 2 11:39:04 VM1 corosync[1086]: [MAIN ] Completed service
> synchronization, ready to provide service.
> Apr 3 09:00:16 VM1 corosync[1086]: [TOTEM ] A processor failed,
> forming new configuration.
> Apr 3 09:00:16 VM1 corosync[1086]: [TOTEM ] Process pause detected
> for 22514 ms, flushing membership messages.
> Apr 3 09:00:32 VM1 corosync[1086]: [TOTEM ] A processor joined or
> left the membership and a new membership was formed.
> Apr 3 09:00:34 VM1 corosync[1086]: [CPG ] chosen downlist: sender
> r(0) ip(192.168.100.1) ; members(old:2 left:0)
> Apr 3 09:00:35 VM1 corosync[1086]: [MAIN ] Completed service
> synchronization, ready to provide service.
> Apr 3 09:05:56 VM1 corosync[1086]: [TOTEM ] A processor failed,
> forming new configuration.
> Apr 3 09:05:56 VM1 corosync[1086]: [TOTEM ] A processor joined or
> left the membership and a new membership was formed.
> Apr 3 09:05:56 VM1 corosync[1086]: [CPG ] chosen downlist: sender
> r(0) ip(192.168.100.1) ; members(old:2 left:0)
> Apr 3 09:05:56 VM1 corosync[1086]: [MAIN ] Completed service
> synchronization, ready to provide service.
> Apr 3 10:05:43 VM1 init: tty (/dev/tty1) main process ended, respawning
>
> 709,1 Bot
>
> ------------
> and this is the output of corosync-blackbox:
>
> Starting replay: head [22276] tail [0]
> rec=[1] Log Message=Corosync Cluster Engine ('1.4.2'): started and
> ready to provide service.
> rec=[2] Log Message=Corosync built-in features:
> rec=[3] Log Message=Successfully read main configuration file
> '/etc/corosync/corosync.conf'.
> rec=[4] Log Message=Token Timeout (5000 ms) retransmit timeout (247 ms)
> rec=[5] Log Message=token hold (187 ms) retransmits before loss (20 retrans)
> rec=[6] Log Message=join (1000 ms) send_join (0 ms) consensus (7500
> ms) merge (200 ms)
> rec=[7] Log Message=downcheck (1000 ms) fail to recv const (2500 msgs)
> rec=[8] Log Message=seqno unchanged const (30 rotations) Maximum
> network MTU 1402
> rec=[9] Log Message=window size per rotation (50 messages) maximum
> messages per rotation (20 messages)
> rec=[10] Log Message=missed count const (5 messages)
> rec=[11] Log Message=send threads (0 threads)
> rec=[12] Log Message=RRP token expired timeout (247 ms)
> rec=[13] Log Message=RRP token problem counter (2000 ms)
> rec=[14] Log Message=RRP threshold (10 problem count)
> rec=[15] Log Message=RRP multicast threshold (100 problem count)
> rec=[16] Log Message=RRP automatic recovery check timeout (1000 ms)
> rec=[17] Log Message=RRP mode set to none.
> rec=[18] Log Message=heartbeat_failures_allowed (0)
> rec=[19] Log Message=max_network_delay (50 ms)
> rec=[20] Log Message=HeartBeat is Disabled. To enable set
> heartbeat_failures_allowed > 0
> rec=[21] Log Message=Initializing transport (UDP/IP Multicast).
> rec=[22] Log Message=Initializing transmit/receive security:
> libtomcrypt SOBER128/SHA1HMAC (mode 0).
> rec=[23] Log Message=you are using ipc api v2
> rec=[24] Log Message=Receive multicast socket recv buffer size (262142 bytes).
> rec=[25] Log Message=Transmit multicast socket send buffer size (262142 bytes).
> rec=[26] Log Message=The network interface [192.168.100.1] is now up.
> rec=[27] Log Message=Created or loaded sequence id 0.192.168.100.1 for
> this ring.
> rec=[28] Log Message=Service engine loaded: corosync extended virtual
> synchrony service
> rec=[29] Log Message=Service engine loaded: corosync configuration service
> rec=[30] Log Message=Service engine loaded: corosync cluster closed
> process group service v1.01
> rec=[31] Log Message=Service engine loaded: corosync cluster config
> database access v1.01
> rec=[32] Log Message=Service engine loaded: corosync profile loading service
> rec=[33] Log Message=Service engine loaded: corosync cluster quorum service v0.1
> rec=[34] Log Message=Compatibility mode set to whitetank. Using V1
> and V2 of the synchronization engine.
> rec=[35] Log Message=entering GATHER state from 15.
> rec=[36] Log Message=Creating commit token because I am the rep.
> rec=[37] Log Message=Saving state aru 0 high seq received 0
> rec=[38] Log Message=Storing new sequence id for ring 4
> rec=[39] Log Message=entering COMMIT state.
> rec=[40] Log Message=got commit token
> rec=[41] Log Message=entering RECOVERY state.
> rec=[42] Log Message=position [0] member 192.168.100.1:
> rec=[43] Log Message=previous ring seq 0 rep 192.168.100.1
> rec=[44] Log Message=aru 0 high delivered 0 received flag 1
> rec=[45] Log Message=Did not need to originate any messages in recovery.
> rec=[46] Log Message=got commit token
> rec=[47] Log Message=Sending initial ORF token
> rec=[48] Log Message=token retrans flag is 0 my set retrans flag0
> retrans queue empty 1 count 0, aru 0
> rec=[49] Log Message=install seq 0 aru 0 high seq received 0
> rec=[50] Log Message=token retrans flag is 0 my set retrans flag0
> retrans queue empty 1 count 1, aru 0
> rec=[51] Log Message=install seq 0 aru 0 high seq received 0
> rec=[52] Log Message=token retrans flag is 0 my set retrans flag0
> retrans queue empty 1 count 2, aru 0
> rec=[53] Log Message=install seq 0 aru 0 high seq received 0
> rec=[54] Log Message=token retrans flag is 0 my set retrans flag0
> retrans queue empty 1 count 3, aru 0
> rec=[55] Log Message=install seq 0 aru 0 high seq received 0
> rec=[56] Log Message=retrans flag count 4 token aru 0 install seq 0 aru 0 0
> rec=[57] Log Message=Resetting old ring state
> rec=[58] Log Message=recovery to regular 1-0
> rec=[59] Log Message=Delivering to app 1 to 0
> rec=[60] Log Message=This node is within the primary component and
> will provide service.
> rec=[61] Log Message=entering OPERATIONAL state.
> rec=[62] Log Message=A processor joined or left the membership and a
> new membership was formed.
> rec=[63] Log Message=mcasted message added to pending queue
> rec=[64] Log Message=Delivering 0 to 1
> rec=[65] Log Message=Delivering MCAST message with seq 1 to pending
> delivery queue
> rec=[66] Log Message=confchg entries 1
> rec=[67] Log Message=Barrier Start Received From 1
> rec=[68] Log Message=Barrier completion status for nodeid 1 = 1.
> rec=[69] Log Message=Synchronization barrier completed
> rec=[70] Log Message=Synchronization actions starting for (dummy CLM service)
> rec=[71] Log Message=mcasted message added to pending queue
> rec=[72] Log Message=Delivering 1 to 2
> rec=[73] Log Message=Delivering MCAST message with seq 2 to pending
> delivery queue
> rec=[74] Log Message=mcasted message added to pending queue
> rec=[75] Log Message=releasing messages up to and including 1
> rec=[76] Log Message=Delivering 2 to 3
> rec=[77] Log Message=Delivering MCAST message with seq 3 to pending
> delivery queue
> rec=[78] Log Message=confchg entries 1
> rec=[79] Log Message=Barrier Start Received From 1
> rec=[80] Log Message=Barrier completion status for nodeid 1 = 1.
> rec=[81] Log Message=Synchronization barrier completed
> rec=[82] Log Message=Committing synchronization for (dummy CLM service)
> rec=[83] Log Message=Synchronization actions starting for (dummy AMF service)
> rec=[84] Log Message=releasing messages up to and including 2
> rec=[85] Log Message=mcasted message added to pending queue
> rec=[86] Log Message=releasing messages up to and including 3
> rec=[87] Log Message=Delivering 3 to 4
> rec=[88] Log Message=Delivering MCAST message with seq 4 to pending
> delivery queue
> rec=[89] Log Message=confchg entries 1
> rec=[90] Log Message=Barrier Start Received From 1
> rec=[91] Log Message=Barrier completion status for nodeid 1 = 1.
> rec=[92] Log Message=Synchronization barrier completed
> rec=[93] Log Message=Committing synchronization for (dummy AMF service)
> rec=[94] Log Message=Synchronization actions starting for (dummy CKPT service)
> rec=[95] Log Message=mcasted message added to pending queue
> rec=[96] Log Message=releasing messages up to and including 4
> rec=[97] Log Message=Delivering 4 to 5
> rec=[98] Log Message=Delivering MCAST message with seq 5 to pending
> delivery queue
> rec=[99] Log Message=confchg entries 1
> rec=[100] Log Message=Barrier Start Received From 1
> rec=[101] Log Message=Barrier completion status for nodeid 1 = 1.
> rec=[102] Log Message=Synchronization barrier completed
> rec=[103] Log Message=Committing synchronization for (dummy CKPT service)
> rec=[104] Log Message=Synchronization actions starting for (dummy EVT service)
> rec=[105] Log Message=mcasted message added to pending queue
> rec=[106] Log Message=Delivering 5 to 6
> rec=[107] Log Message=Delivering MCAST message with seq 6 to pending
> delivery queue
> rec=[108] Log Message=mcasted message added to pending queue
> rec=[109] Log Message=releasing messages up to and including 5
> rec=[110] Log Message=Delivering 6 to 7
> rec=[111] Log Message=Delivering MCAST message with seq 7 to pending
> delivery queue
> rec=[112] Log Message=releasing messages up to and including 6
> rec=[113] Log Message=mcasted message added to pending queue
> rec=[114] Log Message=releasing messages up to and including 7
> rec=[115] Log Message=Delivering 7 to 8
> rec=[116] Log Message=Delivering MCAST message with seq 8 to pending
> delivery queue
> rec=[117] Log Message=confchg entries 1
> rec=[118] Log Message=Barrier Start Received From 1
> rec=[119] Log Message=Barrier completion status for nodeid 1 = 1.
> rec=[120] Log Message=Synchronization barrier completed
> rec=[121] Log Message=Committing synchronization for (dummy EVT service)
> rec=[122] Log Message=Synchronization actions starting for (corosync
> cluster closed process group service v1.01)
> rec=[123] Log Message=mcasted message added to pending queue
> rec=[124] Log Message=mcasted message added to pending queue
> rec=[125] Log Message=Delivering 8 to a
> rec=[126] Log Message=Delivering MCAST message with seq 9 to pending
> delivery queue
> rec=[127] Log Message=Delivering MCAST message with seq a to pending
> delivery queue
> rec=[128] Log Message=comparing: sender r(0) ip(192.168.100.1) ;
> members(old:0 left:0)
> rec=[129] Log Message=chosen downlist: sender r(0) ip(192.168.100.1) ;
> members(old:0 left:0)
> rec=[130] Log Message=mcasted message added to pending queue
> rec=[131] Log Message=releasing messages up to and including 8
> rec=[132] Log Message=Delivering a to b
> rec=[133] Log Message=Delivering MCAST message with seq b to pending
> delivery queue
> rec=[134] Log Message=confchg entries 1
> rec=[135] Log Message=Barrier Start Received From 1
> rec=[136] Log Message=Barrier completion status for nodeid 1 = 1.
> rec=[137] Log Message=Synchronization barrier completed
> rec=[138] Log Message=Committing synchronization for (corosync cluster
> closed process group service v1.01)
> rec=[139] Log Message=mcasted message added to pending queue
> rec=[140] Log Message=releasing messages up to and including a
> rec=[141] Log Message=Delivering b to c
> rec=[142] Log Message=Delivering MCAST message with seq c to pending
> delivery queue
> rec=[143] Log Message=Completed service synchronization, ready to
> provide service.
> rec=[144] Log Message=releasing messages up to and including b
> rec=[145] Log Message=releasing messages up to and including c
> rec=[146] Log Message=entering GATHER state from 11.
> rec=[147] Log Message=Creating commit token because I am the rep.
> rec=[148] Log Message=Saving state aru c high seq received c
> rec=[149] Log Message=Storing new sequence id for ring c
> rec=[150] Log Message=entering COMMIT state.
> rec=[151] Log Message=got commit token
> rec=[152] Log Message=entering RECOVERY state.
> rec=[153] Log Message=TRANS [0] member 192.168.100.1:
> rec=[154] Log Message=position [0] member 192.168.100.1:
> rec=[155] Log Message=previous ring seq 4 rep 192.168.100.1
> rec=[156] Log Message=aru c high delivered c received flag 1
> rec=[157] Log Message=position [1] member 192.168.100.2:
> rec=[158] Log Message=previous ring seq 8 rep 192.168.100.2
> rec=[159] Log Message=aru c high delivered c received flag 1
> rec=[160] Log Message=Did not need to originate any messages in recovery.
> rec=[161] Log Message=got commit token
> rec=[162] Log Message=Sending initial ORF token
> rec=[163] Log Message=token retrans flag is 0 my set retrans flag0
> retrans queue empty 1 count 0, aru 0
> rec=[164] Log Message=install seq 0 aru 0 high seq received 0
> rec=[165] Log Message=token retrans flag is 0 my set retrans flag0
> retrans queue empty 1 count 1, aru 0
> rec=[166] Log Message=install seq 0 aru 0 high seq received 0
> rec=[167] Log Message=token retrans flag is 0 my set retrans flag0
> retrans queue empty 1 count 2, aru 0
> rec=[168] Log Message=install seq 0 aru 0 high seq received 0
> rec=[169] Log Message=token retrans flag is 0 my set retrans flag0
> retrans queue empty 1 count 3, aru 0
> rec=[170] Log Message=install seq 0 aru 0 high seq received 0
> rec=[171] Log Message=retrans flag count 4 token aru 0 install seq 0 aru 0 0
> rec=[172] Log Message=Resetting old ring state
> rec=[173] Log Message=recovery to regular 1-0
> rec=[174] Log Message=Delivering to app d to c
> rec=[175] Log Message=This node is within the primary component and
> will provide service.
> rec=[176] Log Message=entering OPERATIONAL state.
> rec=[177] Log Message=A processor joined or left the membership and a
> new membership was formed.
> rec=[178] Log Message=Received ringid(192.168.100.1:12) seq 1
> rec=[179] Log Message=Delivering 0 to 1
> rec=[180] Log Message=Delivering MCAST message with seq 1 to pending
> delivery queue
> rec=[181] Log Message=confchg entries 2
> rec=[182] Log Message=Barrier Start Received From 2
> rec=[183] Log Message=Barrier completion status for nodeid 1 = 0.
> rec=[184] Log Message=Barrier completion status for nodeid 2 = 1.
> rec=[185] Log Message=mcasted message added to pending queue
> rec=[186] Log Message=Delivering 1 to 2
> rec=[187] Log Message=Delivering MCAST message with seq 2 to pending
> delivery queue
> rec=[188] Log Message=confchg entries 2
> rec=[189] Log Message=Barrier Start Received From 1
> rec=[190] Log Message=Barrier completion status for nodeid 1 = 1.
> rec=[191] Log Message=Barrier completion status for nodeid 2 = 1.
> rec=[192] Log Message=Synchronization barrier completed
> rec=[193] Log Message=Synchronization actions starting for (dummy CLM service)
> rec=[194] Log Message=Received ringid(192.168.100.1:12) seq 2
> rec=[195] Log Message=mcasted message added to pending queue
> rec=[196] Log Message=releasing messages up to and including 1
> rec=[197] Log Message=Delivering 2 to 3
> rec=[198] Log Message=Delivering MCAST message with seq 3 to pending
> delivery queue
> rec=[199] Log Message=Received ringid(192.168.100.1:12) seq 3
> rec=[200] Log Message=Received ringid(192.168.100.1:12) seq 4
> rec=[201] Log Message=Delivering 3 to 4
> rec=[202] Log Message=Delivering MCAST message with seq 4 to pending
> delivery queue
> rec=[203] Log Message=mcasted message added to pending queue
> rec=[204] Log Message=releasing messages up to and including 2
> rec=[205] Log Message=Delivering 4 to 5
> rec=[206] Log Message=Delivering MCAST message with seq 5 to pending
> delivery queue
> rec=[207] Log Message=confchg entries 2
> rec=[208] Log Message=Barrier Start Received From 1
> rec=[209] Log Message=Barrier completion status for nodeid 1 = 1.
> rec=[210] Log Message=Barrier completion status for nodeid 2 = 0.
> rec=[211] Log Message=Received ringid(192.168.100.1:12) seq 5
> rec=[212] Log Message=Received ringid(192.168.100.1:12) seq 6
> rec=[213] Log Message=Delivering 5 to 6
> rec=[214] Log Message=Delivering MCAST message with seq 6 to pending
> delivery queue
> rec=[215] Log Message=confchg entries 2
> rec=[216] Log Message=Barrier Start Received From 2
> rec=[217] Log Message=Barrier completion status for nodeid 1 = 1.
> rec=[218] Log Message=Barrier completion status for nodeid 2 = 1.
> rec=[219] Log Message=Synchronization barrier completed
> rec=[220] Log Message=Committing synchronization for (dummy CLM service)
> rec=[221] Log Message=Synchronization actions starting for (dummy AMF service)
> rec=[222] Log Message=releasing messages up to and including 4
> rec=[223] Log Message=releasing messages up to and including 6
> rec=[224] Log Message=Received ringid(192.168.100.1:12) seq 7
> rec=[225] Log Message=Delivering 6 to 7
> rec=[226] Log Message=Delivering MCAST message with seq 7 to pending
> delivery queue
> rec=[227] Log Message=confchg entries 2
> rec=[228] Log Message=Barrier Start Received From 2
> rec=[229] Log Message=Barrier completion status for nodeid 1 = 0.
> rec=[230] Log Message=Barrier completion status for nodeid 2 = 1.
> rec=[231] Log Message=mcasted message added to pending queue
> rec=[232] Log Message=Delivering 7 to 8
> rec=[233] Log Message=Delivering MCAST message with seq 8 to pending
> delivery queue
> rec=[234] Log Message=confchg entries 2
> rec=[235] Log Message=Barrier Start Received From 1
> rec=[236] Log Message=Barrier completion status for nodeid 1 = 1.
> rec=[237] Log Message=Barrier completion status for nodeid 2 = 1.
> rec=[238] Log Message=Synchronization barrier completed
> rec=[239] Log Message=Committing synchronization for (dummy AMF service)
> rec=[240] Log Message=Synchronization actions starting for (dummy CKPT service)
> rec=[241] Log Message=Received ringid(192.168.100.1:12) seq 8
> rec=[242] Log Message=releasing messages up to and including 7
> rec=[243] Log Message=mcasted message added to pending queue
> rec=[244] Log Message=releasing messages up to and including 8
> rec=[245] Log Message=Delivering 8 to 9
> rec=[246] Log Message=Delivering MCAST message with seq 9 to pending
> delivery queue
> rec=[247] Log Message=confchg entries 2
> rec=[248] Log Message=Barrier Start Received From 1
> rec=[249] Log Message=Barrier completion status for nodeid 1 = 1.
> rec=[250] Log Message=Barrier completion status for nodeid 2 = 0.
> rec=[251] Log Message=Received ringid(192.168.100.1:12) seq 9
> rec=[252] Log Message=Received ringid(192.168.100.1:12) seq a
> rec=[253] Log Message=Delivering 9 to a
> rec=[254] Log Message=Delivering MCAST message with seq a to pending
> delivery queue
> rec=[255] Log Message=confchg entries 2
> rec=[256] Log Message=Barrier Start Received From 2
> rec=[257] Log Message=Barrier completion status for nodeid 1 = 1.
> rec=[258] Log Message=Barrier completion status for nodeid 2 = 1.
> rec=[259] Log Message=Synchronization barrier completed
> rec=[260] Log Message=Committing synchronization for (dummy CKPT service)
> rec=[261] Log Message=Synchronization actions starting for (dummy EVT service)
> rec=[262] Log Message=Received ringid(192.168.100.1:12) seq b
> rec=[263] Log Message=Delivering a to b
> rec=[264] Log Message=Delivering MCAST message with seq b to pending
> delivery queue
> rec=[265] Log Message=mcasted message added to pending queue
> rec=[266] Log Message=releasing messages up to and including a
> rec=[267] Log Message=Delivering b to c
> rec=[268] Log Message=Delivering MCAST message with seq c to pending
> delivery queue
> rec=[269] Log Message=Received ringid(192.168.100.1:12) seq c
> rec=[270] Log Message=mcasted message added to pending queue
> rec=[271] Log Message=releasing messages up to and including b
> rec=[272] Log Message=Delivering c to d
> rec=[273] Log Message=Delivering MCAST message with seq d to pending
> delivery queue
> rec=[274] Log Message=Received ringid(192.168.100.1:12) seq d
> rec=[275] Log Message=Received ringid(192.168.100.1:12) seq e
> rec=[276] Log Message=Delivering d to e
> rec=[277] Log Message=Delivering MCAST message with seq e to pending
> delivery queue
> rec=[278] Log Message=releasing messages up to and including c
> rec=[279] Log Message=releasing messages up to and including e
> rec=[280] Log Message=Received ringid(192.168.100.1:12) seq f
> rec=[281] Log Message=Delivering e to f
> rec=[282] Log Message=Delivering MCAST message with seq f to pending
> delivery queue
> rec=[283] Log Message=confchg entries 2
> rec=[284] Log Message=Barrier Start Received From 2
> rec=[285] Log Message=Barrier completion status for nodeid 1 = 0.
> rec=[286] Log Message=Barrier completion status for nodeid 2 = 1.
> rec=[287] Log Message=mcasted message added to pending queue
> rec=[288] Log Message=Delivering f to 10
> rec=[289] Log Message=Delivering MCAST message with seq 10 to pending
> delivery queue
> rec=[290] Log Message=confchg entries 2
> rec=[291] Log Message=Barrier Start Received From 1
> rec=[292] Log Message=Barrier completion status for nodeid 1 = 1.
> rec=[293] Log Message=Barrier completion status for nodeid 2 = 1.
> rec=[294] Log Message=Synchronization barrier completed
> rec=[295] Log Message=Committing synchronization for (dummy EVT service)
> rec=[296] Log Message=Synchronization actions starting for (corosync
> cluster closed process group service v1.01)
> rec=[297] Log Message=mcasted message added to pending queue
> rec=[298] Log Message=Received ringid(192.168.100.1:12) seq 10
> rec=[299] Log Message=mcasted message added to pending queue
> rec=[300] Log Message=releasing messages up to and including f
> rec=[301] Log Message=Delivering 10 to 12
> rec=[302] Log Message=Delivering MCAST message with seq 11 to pending
> delivery queue
> rec=[303] Log Message=Delivering MCAST message with seq 12 to pending
> delivery queue
> rec=[304] Log Message=Received ringid(192.168.100.1:12) seq 11
> rec=[305] Log Message=Received ringid(192.168.100.1:12) seq 12
> rec=[306] Log Message=Received ringid(192.168.100.1:12) seq 13
> rec=[307] Log Message=Delivering 12 to 13
> rec=[308] Log Message=Delivering MCAST message with seq 13 to pending
> delivery queue
> rec=[309] Log Message=Received ringid(192.168.100.1:12) seq 14
> rec=[310] Log Message=Delivering 13 to 14
> rec=[311] Log Message=Delivering MCAST message with seq 14 to pending
> delivery queue
> rec=[312] Log Message=comparing: sender r(0) ip(192.168.100.2) ;
> members(old:1 left:0)
> rec=[313] Log Message=comparing: sender r(0) ip(192.168.100.1) ;
> members(old:1 left:0)
> rec=[314] Log Message=chosen downlist: sender r(0) ip(192.168.100.1) ;
> members(old:1 left:0)
> rec=[315] Log Message=mcasted message added to pending queue
> rec=[316] Log Message=releasing messages up to and including 10
> rec=[317] Log Message=Delivering 14 to 15
> rec=[318] Log Message=Delivering MCAST message with seq 15 to pending
> delivery queue
> rec=[319] Log Message=confchg entries 2
> rec=[320] Log Message=Barrier Start Received From 1
> rec=[321] Log Message=Barrier completion status for nodeid 1 = 1.
> rec=[322] Log Message=Barrier completion status for nodeid 2 = 0.
> rec=[323] Log Message=Received ringid(192.168.100.1:12) seq 15
> rec=[324] Log Message=Received ringid(192.168.100.1:12) seq 16
> rec=[325] Log Message=Delivering 15 to 16
> rec=[326] Log Message=Delivering MCAST message with seq 16 to pending
> delivery queue
> rec=[327] Log Message=confchg entries 2
> rec=[328] Log Message=Barrier Start Received From 2
> rec=[329] Log Message=Barrier completion status for nodeid 1 = 1.
> rec=[330] Log Message=Barrier completion status for nodeid 2 = 1.
> rec=[331] Log Message=Synchronization barrier completed
> rec=[332] Log Message=Committing synchronization for (corosync cluster
> closed process group service v1.01)
> rec=[333] Log Message=mcasted message added to pending queue
> rec=[334] Log Message=releasing messages up to and including 14
> rec=[335] Log Message=Delivering 16 to 17
> rec=[336] Log Message=Delivering MCAST message with seq 17 to pending
> delivery queue
> rec=[337] Log Message=Received ringid(192.168.100.1:12) seq 17
> rec=[338] Log Message=Received ringid(192.168.100.1:12) seq 18
> rec=[339] Log Message=Delivering 17 to 18
> rec=[340] Log Message=Delivering MCAST message with seq 18 to pending
> delivery queue
> rec=[341] Log Message=Completed service synchronization, ready to
> provide service.
> rec=[342] Log Message=releasing messages up to and including 16
> rec=[343] Log Message=releasing messages up to and including 18
> rec=[344] Log Message=The token was lost in the OPERATIONAL state.
> rec=[345] Log Message=A processor failed, forming new configuration.
> rec=[346] Log Message=Receive multicast socket recv buffer size (262142 bytes).
> rec=[347] Log Message=Transmit multicast socket send buffer size (262142 bytes).
> rec=[348] Log Message=entering GATHER state from 2.
> rec=[349] Log Message=Process pause detected for 22514 ms, flushing
> membership messages.
> rec=[350] Log Message=Creating commit token because I am the rep.
> rec=[351] Log Message=Saving state aru 18 high seq received 18
> rec=[352] Log Message=Storing new sequence id for ring 10
> rec=[353] Log Message=entering COMMIT state.
> rec=[354] Log Message=got commit token
> rec=[355] Log Message=entering RECOVERY state.
> rec=[356] Log Message=TRANS [0] member 192.168.100.1:
> rec=[357] Log Message=TRANS [1] member 192.168.100.2:
> rec=[358] Log Message=position [0] member 192.168.100.1:
> rec=[359] Log Message=previous ring seq c rep 192.168.100.1
> rec=[360] Log Message=aru 18 high delivered 18 received flag 1
> rec=[361] Log Message=position [1] member 192.168.100.2:
> rec=[362] Log Message=previous ring seq c rep 192.168.100.1
> rec=[363] Log Message=aru 18 high delivered 18 received flag 1
> rec=[364] Log Message=Did not need to originate any messages in recovery.
> rec=[365] Log Message=The token was lost in the RECOVERY state.
> rec=[366] Log Message=Restoring instance->my_aru 18 my high seq received 18
> rec=[367] Log Message=entering GATHER state from 5.
> rec=[368] Log Message=Creating commit token because I am the rep.
> rec=[369] Log Message=Storing new sequence id for ring 14
> rec=[370] Log Message=entering COMMIT state.
> rec=[371] Log Message=got commit token
> rec=[372] Log Message=entering RECOVERY state.
> rec=[373] Log Message=TRANS [0] member 192.168.100.1:
> rec=[374] Log Message=TRANS [1] member 192.168.100.2:
> rec=[375] Log Message=position [0] member 192.168.100.1:
> rec=[376] Log Message=previous ring seq c rep 192.168.100.1
> rec=[377] Log Message=aru 18 high delivered 18 received flag 1
> rec=[378] Log Message=position [1] member 192.168.100.2:
> rec=[379] Log Message=previous ring seq c rep 192.168.100.1
> rec=[380] Log Message=aru 18 high delivered 18 received flag 1
> rec=[381] Log Message=Did not need to originate any messages in recovery.
> rec=[382] Log Message=got commit token
> rec=[383] Log Message=Sending initial ORF token
> rec=[384] Log Message=token retrans flag is 0 my set retrans flag0
> retrans queue empty 1 count 0, aru 0
> rec=[385] Log Message=install seq 0 aru 0 high seq received 0
> rec=[386] Log Message=token retrans flag is 0 my set retrans flag0
> retrans queue empty 1 count 1, aru 0
> rec=[387] Log Message=install seq 0 aru 0 high seq received 0
> rec=[388] Log Message=token retrans flag is 0 my set retrans flag0
> retrans queue empty 1 count 2, aru 0
> rec=[389] Log Message=install seq 0 aru 0 high seq received 0
> rec=[390] Log Message=token retrans flag is 0 my set retrans flag0
> retrans queue empty 1 count 3, aru 0
> rec=[391] Log Message=install seq 0 aru 0 high seq received 0
> rec=[392] Log Message=retrans flag count 4 token aru 0 install seq 0 aru 0 0
> rec=[393] Log Message=Resetting old ring state
> rec=[394] Log Message=recovery to regular 1-0
> rec=[395] Log Message=Delivering to app 19 to 18
> rec=[396] Log Message=This node is within the primary component and
> will provide service.
> rec=[397] Log Message=entering OPERATIONAL state.
> rec=[398] Log Message=A processor joined or left the membership and a
> new membership was formed.
> rec=[399] Log Message=mcasted message added to pending queue
> rec=[400] Log Message=Received ringid(192.168.100.1:20) seq 1
> rec=[401] Log Message=Delivering 0 to 1
> rec=[402] Log Message=Delivering MCAST message with seq 1 to pending
> delivery queue
> rec=[403] Log Message=confchg entries 2
> rec=[404] Log Message=Barrier Start Received From 2
> rec=[405] Log Message=Barrier completion status for nodeid 1 = 0.
> rec=[406] Log Message=Barrier completion status for nodeid 2 = 1.
> rec=[407] Log Message=Delivering 1 to 2
> rec=[408] Log Message=Delivering MCAST message with seq 2 to pending
> delivery queue
> rec=[409] Log Message=confchg entries 2
> rec=[410] Log Message=Barrier Start Received From 1
> rec=[411] Log Message=Barrier completion status for nodeid 1 = 1.
> rec=[412] Log Message=Barrier completion status for nodeid 2 = 1.
> rec=[413] Log Message=Synchronization barrier completed
> rec=[414] Log Message=Synchronization actions starting for (dummy CLM service)
> rec=[415] Log Message=Received ringid(192.168.100.1:20) seq 2
> rec=[416] Log Message=mcasted message added to pending queue
> rec=[417] Log Message=releasing messages up to and including 1
> rec=[418] Log Message=Delivering 2 to 3
> rec=[419] Log Message=Delivering MCAST message with seq 3 to pending
> delivery queue
> rec=[420] Log Message=Received ringid(192.168.100.1:20) seq 3
> rec=[421] Log Message=Received ringid(192.168.100.1:20) seq 4
> rec=[422] Log Message=Delivering 3 to 4
> rec=[423] Log Message=Delivering MCAST message with seq 4 to pending
> delivery queue
> rec=[424] Log Message=mcasted message added to pending queue
> rec=[425] Log Message=releasing messages up to and including 2
> rec=[426] Log Message=Delivering 4 to 5
> rec=[427] Log Message=Delivering MCAST message with seq 5 to pending
> delivery queue
> rec=[428] Log Message=confchg entries 2
> rec=[429] Log Message=Barrier Start Received From 1
> rec=[430] Log Message=Barrier completion status for nodeid 1 = 1.
> rec=[431] Log Message=Barrier completion status for nodeid 2 = 0.
> rec=[432] Log Message=Received ringid(192.168.100.1:20) seq 5
> rec=[433] Log Message=Received ringid(192.168.100.1:20) seq 6
> rec=[434] Log Message=Delivering 5 to 6
> rec=[435] Log Message=Delivering MCAST message with seq 6 to pending
> delivery queue
> rec=[436] Log Message=confchg entries 2
> rec=[437] Log Message=Barrier Start Received From 2
> rec=[438] Log Message=Barrier completion status for nodeid 1 = 1.
> rec=[439] Log Message=Barrier completion status for nodeid 2 = 1.
> rec=[440] Log Message=Synchronization barrier completed
> rec=[441] Log Message=Committing synchronization for (dummy CLM service)
> rec=[442] Log Message=Synchronization actions starting for (dummy AMF service)
> rec=[443] Log Message=releasing messages up to and including 4
> rec=[444] Log Message=releasing messages up to and including 6
> rec=[445] Log Message=Received ringid(192.168.100.1:20) seq 7
> rec=[446] Log Message=Delivering 6 to 7
> rec=[447] Log Message=Delivering MCAST message with seq 7 to pending
> delivery queue
> rec=[448] Log Message=confchg entries 2
> rec=[449] Log Message=Barrier Start Received From 2
> rec=[450] Log Message=Barrier completion status for nodeid 1 = 0.
> rec=[451] Log Message=Barrier completion status for nodeid 2 = 1.
> rec=[452] Log Message=mcasted message added to pending queue
> rec=[453] Log Message=Delivering 7 to 8
> rec=[454] Log Message=Delivering MCAST message with seq 8 to pending
> delivery queue
> rec=[455] Log Message=confchg entries 2
> rec=[456] Log Message=Barrier Start Received From 1
> rec=[457] Log Message=Barrier completion status for nodeid 1 = 1.
> rec=[458] Log Message=Barrier completion status for nodeid 2 = 1.
> rec=[459] Log Message=Synchronization barrier completed
> rec=[460] Log Message=Committing synchronization for (dummy AMF service)
> rec=[461] Log Message=Synchronization actions starting for (dummy CKPT service)
> rec=[462] Log Message=Received ringid(192.168.100.1:20) seq 8
> rec=[463] Log Message=releasing messages up to and including 7
> rec=[464] Log Message=mcasted message added to pending queue
> rec=[465] Log Message=releasing messages up to and including 8
> rec=[466] Log Message=Delivering 8 to 9
> rec=[467] Log Message=Delivering MCAST message with seq 9 to pending
> delivery queue
> rec=[468] Log Message=confchg entries 2
> rec=[469] Log Message=Barrier Start Received From 1
> rec=[470] Log Message=Barrier completion status for nodeid 1 = 1.
> rec=[471] Log Message=Barrier completion status for nodeid 2 = 0.
> rec=[472] Log Message=Received ringid(192.168.100.1:20) seq 9
> rec=[473] Log Message=Received ringid(192.168.100.1:20) seq a
> rec=[474] Log Message=Delivering 9 to a
> rec=[475] Log Message=Delivering MCAST message with seq a to pending
> delivery queue
> rec=[476] Log Message=confchg entries 2
> rec=[477] Log Message=Barrier Start Received From 2
> rec=[478] Log Message=Barrier completion status for nodeid 1 = 1.
> rec=[479] Log Message=Barrier completion status for nodeid 2 = 1.
> rec=[480] Log Message=Synchronization barrier completed
> rec=[481] Log Message=Committing synchronization for (dummy CKPT service)
> rec=[482] Log Message=Synchronization actions starting for (dummy EVT service)
> rec=[483] Log Message=Received ringid(192.168.100.1:20) seq b
> rec=[484] Log Message=Delivering a to b
> rec=[485] Log Message=Delivering MCAST message with seq b to pending
> delivery queue
> rec=[486] Log Message=mcasted message added to pending queue
> rec=[487] Log Message=releasing messages up to and including a
> rec=[488] Log Message=Delivering b to c
> rec=[489] Log Message=Delivering MCAST message with seq c to pending
> delivery queue
> rec=[490] Log Message=Received ringid(192.168.100.1:20) seq c
> rec=[491] Log Message=mcasted message added to pending queue
> rec=[492] Log Message=releasing messages up to and including b
> rec=[493] Log Message=Delivering c to d
> rec=[494] Log Message=Delivering MCAST message with seq d to pending
> delivery queue
> rec=[495] Log Message=Received ringid(192.168.100.1:20) seq d
> rec=[496] Log Message=Received ringid(192.168.100.1:20) seq e
> rec=[497] Log Message=Delivering d to e
> rec=[498] Log Message=Delivering MCAST message with seq e to pending
> delivery queue
> rec=[499] Log Message=releasing messages up to and including c
> rec=[500] Log Message=releasing messages up to and including e
> rec=[501] Log Message=Received ringid(192.168.100.1:20) seq f
> rec=[502] Log Message=Delivering e to f
> rec=[503] Log Message=Delivering MCAST message with seq f to pending
> delivery queue
> rec=[504] Log Message=confchg entries 2
> rec=[505] Log Message=Barrier Start Received From 2
> rec=[506] Log Message=Barrier completion status for nodeid 1 = 0.
> rec=[507] Log Message=Barrier completion status for nodeid 2 = 1.
> rec=[508] Log Message=mcasted message added to pending queue
> rec=[509] Log Message=Delivering f to 10
> rec=[510] Log Message=Delivering MCAST message with seq 10 to pending
> delivery queue
> rec=[511] Log Message=confchg entries 2
> rec=[512] Log Message=Barrier Start Received From 1
> rec=[513] Log Message=Barrier completion status for nodeid 1 = 1.
> rec=[514] Log Message=Barrier completion status for nodeid 2 = 1.
> rec=[515] Log Message=Synchronization barrier completed
> rec=[516] Log Message=Committing synchronization for (dummy EVT service)
> rec=[517] Log Message=Synchronization actions starting for (corosync
> cluster closed process group service v1.01)
> rec=[518] Log Message=mcasted message added to pending queue
> rec=[519] Log Message=Received ringid(192.168.100.1:20) seq 10
> rec=[520] Log Message=mcasted message added to pending queue
> rec=[521] Log Message=releasing messages up to and including f
> rec=[522] Log Message=Delivering 10 to 12
> rec=[523] Log Message=Delivering MCAST message with seq 11 to pending
> delivery queue
> rec=[524] Log Message=Delivering MCAST message with seq 12 to pending
> delivery queue
> rec=[525] Log Message=Received ringid(192.168.100.1:20) seq 11
> rec=[526] Log Message=Received ringid(192.168.100.1:20) seq 12
> rec=[527] Log Message=Received ringid(192.168.100.1:20) seq 13
> rec=[528] Log Message=Delivering 12 to 13
> rec=[529] Log Message=Delivering MCAST message with seq 13 to pending
> delivery queue
> rec=[530] Log Message=Received ringid(192.168.100.1:20) seq 14
> rec=[531] Log Message=Delivering 13 to 14
> rec=[532] Log Message=Delivering MCAST message with seq 14 to pending
> delivery queue
> rec=[533] Log Message=comparing: sender r(0) ip(192.168.100.2) ;
> members(old:2 left:0)
> rec=[534] Log Message=comparing: sender r(0) ip(192.168.100.1) ;
> members(old:2 left:0)
> rec=[535] Log Message=chosen downlist: sender r(0) ip(192.168.100.1) ;
> members(old:2 left:0)
> rec=[536] Log Message=mcasted message added to pending queue
> rec=[537] Log Message=releasing messages up to and including 10
> rec=[538] Log Message=Delivering 14 to 15
> rec=[539] Log Message=Delivering MCAST message with seq 15 to pending
> delivery queue
> rec=[540] Log Message=confchg entries 2
> rec=[541] Log Message=Barrier Start Received From 1
> rec=[542] Log Message=Barrier completion status for nodeid 1 = 1.
> rec=[543] Log Message=Barrier completion status for nodeid 2 = 0.
> rec=[544] Log Message=Received ringid(192.168.100.1:20) seq 15
> rec=[545] Log Message=releasing messages up to and including 14
> rec=[546] Log Message=Received ringid(192.168.100.1:20) seq 16
> rec=[547] Log Message=Delivering 15 to 16
> rec=[548] Log Message=Delivering MCAST message with seq 16 to pending
> delivery queue
> rec=[549] Log Message=confchg entries 2
> rec=[550] Log Message=Barrier Start Received From 2
> rec=[551] Log Message=Barrier completion status for nodeid 1 = 1.
> rec=[552] Log Message=Barrier completion status for nodeid 2 = 1.
> rec=[553] Log Message=Synchronization barrier completed
> rec=[554] Log Message=Committing synchronization for (corosync cluster
> closed process group service v1.01)
> rec=[555] Log Message=mcasted message added to pending queue
> rec=[556] Log Message=Delivering 16 to 17
> rec=[557] Log Message=Delivering MCAST message with seq 17 to pending
> delivery queue
> rec=[558] Log Message=Received ringid(192.168.100.1:20) seq 17
> rec=[559] Log Message=Received ringid(192.168.100.1:20) seq 18
> rec=[560] Log Message=Delivering 17 to 18
> rec=[561] Log Message=Delivering MCAST message with seq 18 to pending
> delivery queue
> rec=[562] Log Message=Completed service synchronization, ready to
> provide service.
> rec=[563] Log Message=releasing messages up to and including 16
> rec=[564] Log Message=releasing messages up to and including 18
> rec=[565] Log Message=The token was lost in the OPERATIONAL state.
> rec=[566] Log Message=A processor failed, forming new configuration.
> rec=[567] Log Message=Receive multicast socket recv buffer size (262142 bytes).
> rec=[568] Log Message=Transmit multicast socket send buffer size (262142 bytes).
> rec=[569] Log Message=entering GATHER state from 2.
> rec=[570] Log Message=Creating commit token because I am the rep.
> rec=[571] Log Message=Saving state aru 18 high seq received 18
> rec=[572] Log Message=Storing new sequence id for ring 18
> rec=[573] Log Message=entering COMMIT state.
> rec=[574] Log Message=got commit token
> rec=[575] Log Message=entering RECOVERY state.
> rec=[576] Log Message=TRANS [0] member 192.168.100.1:
> rec=[577] Log Message=TRANS [1] member 192.168.100.2:
> rec=[578] Log Message=position [0] member 192.168.100.1:
> rec=[579] Log Message=previous ring seq 14 rep 192.168.100.1
> rec=[580] Log Message=aru 18 high delivered 18 received flag 1
> rec=[581] Log Message=position [1] member 192.168.100.2:
> rec=[582] Log Message=previous ring seq 14 rep 192.168.100.1
> rec=[583] Log Message=aru 18 high delivered 18 received flag 1
> rec=[584] Log Message=Did not need to originate any messages in recovery.
> rec=[585] Log Message=got commit token
> rec=[586] Log Message=Sending initial ORF token
> rec=[587] Log Message=token retrans flag is 0 my set retrans flag0
> retrans queue empty 1 count 0, aru 0
> rec=[588] Log Message=install seq 0 aru 0 high seq received 0
> rec=[589] Log Message=token retrans flag is 0 my set retrans flag0
> retrans queue empty 1 count 1, aru 0
> rec=[590] Log Message=install seq 0 aru 0 high seq received 0
> rec=[591] Log Message=token retrans flag is 0 my set retrans flag0
> retrans queue empty 1 count 2, aru 0
> rec=[592] Log Message=install seq 0 aru 0 high seq received 0
> rec=[593] Log Message=token retrans flag is 0 my set retrans flag0
> retrans queue empty 1 count 3, aru 0
> rec=[594] Log Message=install seq 0 aru 0 high seq received 0
> rec=[595] Log Message=retrans flag count 4 token aru 0 install seq 0 aru 0 0
> rec=[596] Log Message=Resetting old ring state
> rec=[597] Log Message=recovery to regular 1-0
> rec=[598] Log Message=Delivering to app 19 to 18
> rec=[599] Log Message=This node is within the primary component and
> will provide service.
> rec=[600] Log Message=entering OPERATIONAL state.
> rec=[601] Log Message=A processor joined or left the membership and a
> new membership was formed.
> rec=[602] Log Message=Received ringid(192.168.100.1:24) seq 1
> rec=[603] Log Message=Delivering 0 to 1
> rec=[604] Log Message=Delivering MCAST message with seq 1 to pending
> delivery queue
> rec=[605] Log Message=confchg entries 2
> rec=[606] Log Message=Barrier Start Received From 2
> rec=[607] Log Message=Barrier completion status for nodeid 1 = 0.
> rec=[608] Log Message=Barrier completion status for nodeid 2 = 1.
> rec=[609] Log Message=mcasted message added to pending queue
> rec=[610] Log Message=Delivering 1 to 2
> rec=[611] Log Message=Delivering MCAST message with seq 2 to pending
> delivery queue
> rec=[612] Log Message=confchg entries 2
> rec=[613] Log Message=Barrier Start Received From 1
> rec=[614] Log Message=Barrier completion status for nodeid 1 = 1.
> rec=[615] Log Message=Barrier completion status for nodeid 2 = 1.
> rec=[616] Log Message=Synchronization barrier completed
> rec=[617] Log Message=Synchronization actions starting for (dummy CLM service)
> rec=[618] Log Message=Received ringid(192.168.100.1:24) seq 2
> rec=[619] Log Message=mcasted message added to pending queue
> rec=[620] Log Message=releasing messages up to and including 1
> rec=[621] Log Message=Delivering 2 to 3
> rec=[622] Log Message=Delivering MCAST message with seq 3 to pending
> delivery queue
> rec=[623] Log Message=Received ringid(192.168.100.1:24) seq 3
> rec=[624] Log Message=Received ringid(192.168.100.1:24) seq 4
> rec=[625] Log Message=Delivering 3 to 4
> rec=[626] Log Message=Delivering MCAST message with seq 4 to pending
> delivery queue
> rec=[627] Log Message=mcasted message added to pending queue
> rec=[628] Log Message=releasing messages up to and including 2
> rec=[629] Log Message=Delivering 4 to 5
> rec=[630] Log Message=Delivering MCAST message with seq 5 to pending
> delivery queue
> rec=[631] Log Message=confchg entries 2
> rec=[632] Log Message=Barrier Start Received From 1
> rec=[633] Log Message=Barrier completion status for nodeid 1 = 1.
> rec=[634] Log Message=Barrier completion status for nodeid 2 = 0.
> rec=[635] Log Message=Received ringid(192.168.100.1:24) seq 5
> rec=[636] Log Message=Received ringid(192.168.100.1:24) seq 6
> rec=[637] Log Message=Delivering 5 to 6
> rec=[638] Log Message=Delivering MCAST message with seq 6 to pending
> delivery queue
> rec=[639] Log Message=confchg entries 2
> rec=[640] Log Message=Barrier Start Received From 2
> rec=[641] Log Message=Barrier completion status for nodeid 1 = 1.
> rec=[642] Log Message=Barrier completion status for nodeid 2 = 1.
> rec=[643] Log Message=Synchronization barrier completed
> rec=[644] Log Message=Committing synchronization for (dummy CLM service)
> rec=[645] Log Message=Synchronization actions starting for (dummy AMF service)
> rec=[646] Log Message=releasing messages up to and including 4
> rec=[647] Log Message=releasing messages up to and including 6
> rec=[648] Log Message=Received ringid(192.168.100.1:24) seq 7
> rec=[649] Log Message=Delivering 6 to 7
> rec=[650] Log Message=Delivering MCAST message with seq 7 to pending
> delivery queue
> rec=[651] Log Message=confchg entries 2
> rec=[652] Log Message=Barrier Start Received From 2
> rec=[653] Log Message=Barrier completion status for nodeid 1 = 0.
> rec=[654] Log Message=Barrier completion status for nodeid 2 = 1.
> rec=[655] Log Message=mcasted message added to pending queue
> rec=[656] Log Message=Delivering 7 to 8
> rec=[657] Log Message=Delivering MCAST message with seq 8 to pending
> delivery queue
> rec=[658] Log Message=confchg entries 2
> rec=[659] Log Message=Barrier Start Received From 1
> rec=[660] Log Message=Barrier completion status for nodeid 1 = 1.
> rec=[661] Log Message=Barrier completion status for nodeid 2 = 1.
> rec=[662] Log Message=Synchronization barrier completed
> rec=[663] Log Message=Committing synchronization for (dummy AMF service)
> rec=[664] Log Message=Synchronization actions starting for (dummy CKPT service)
> rec=[665] Log Message=Received ringid(192.168.100.1:24) seq 8
> rec=[666] Log Message=releasing messages up to and including 7
> rec=[667] Log Message=mcasted message added to pending queue
> rec=[668] Log Message=releasing messages up to and including 8
> rec=[669] Log Message=Delivering 8 to 9
> rec=[670] Log Message=Delivering MCAST message with seq 9 to pending
> delivery queue
> rec=[671] Log Message=confchg entries 2
> rec=[672] Log Message=Barrier Start Received From 1
> rec=[673] Log Message=Barrier completion status for nodeid 1 = 1.
> rec=[674] Log Message=Barrier completion status for nodeid 2 = 0.
> rec=[675] Log Message=Received ringid(192.168.100.1:24) seq 9
> rec=[676] Log Message=Received ringid(192.168.100.1:24) seq a
> rec=[677] Log Message=Delivering 9 to a
> rec=[678] Log Message=Delivering MCAST message with seq a to pending
> delivery queue
> rec=[679] Log Message=confchg entries 2
> rec=[680] Log Message=Barrier Start Received From 2
> rec=[681] Log Message=Barrier completion status for nodeid 1 = 1.
> rec=[682] Log Message=Barrier completion status for nodeid 2 = 1.
> rec=[683] Log Message=Synchronization barrier completed
> rec=[684] Log Message=Committing synchronization for (dummy CKPT service)
> rec=[685] Log Message=Synchronization actions starting for (dummy EVT service)
> rec=[686] Log Message=Received ringid(192.168.100.1:24) seq b
> rec=[687] Log Message=Delivering a to b
> rec=[688] Log Message=Delivering MCAST message with seq b to pending
> delivery queue
> rec=[689] Log Message=mcasted message added to pending queue
> rec=[690] Log Message=releasing messages up to and including a
> rec=[691] Log Message=Delivering b to c
> rec=[692] Log Message=Delivering MCAST message with seq c to pending
> delivery queue
> rec=[693] Log Message=Received ringid(192.168.100.1:24) seq c
> rec=[694] Log Message=mcasted message added to pending queue
> rec=[695] Log Message=releasing messages up to and including b
> rec=[696] Log Message=Delivering c to d
> rec=[697] Log Message=Delivering MCAST message with seq d to pending
> delivery queue
> rec=[698] Log Message=Received ringid(192.168.100.1:24) seq d
> rec=[699] Log Message=Received ringid(192.168.100.1:24) seq e
> rec=[700] Log Message=Delivering d to e
> rec=[701] Log Message=Delivering MCAST message with seq e to pending
> delivery queue
> rec=[702] Log Message=releasing messages up to and including c
> rec=[703] Log Message=releasing messages up to and including e
> rec=[704] Log Message=Received ringid(192.168.100.1:24) seq f
> rec=[705] Log Message=Delivering e to f
> rec=[706] Log Message=Delivering MCAST message with seq f to pending
> delivery queue
> rec=[707] Log Message=confchg entries 2
> rec=[708] Log Message=Barrier Start Received From 2
> rec=[709] Log Message=Barrier completion status for nodeid 1 = 0.
> rec=[710] Log Message=Barrier completion status for nodeid 2 = 1.
> rec=[711] Log Message=mcasted message added to pending queue
> rec=[712] Log Message=Delivering f to 10
> rec=[713] Log Message=Delivering MCAST message with seq 10 to pending
> delivery queue
> rec=[714] Log Message=confchg entries 2
> rec=[715] Log Message=Barrier Start Received From 1
> rec=[716] Log Message=Barrier completion status for nodeid 1 = 1.
> rec=[717] Log Message=Barrier completion status for nodeid 2 = 1.
> rec=[718] Log Message=Synchronization barrier completed
> rec=[719] Log Message=Committing synchronization for (dummy EVT service)
> rec=[720] Log Message=Synchronization actions starting for (corosync
> cluster closed process group service v1.01)
> rec=[721] Log Message=mcasted message added to pending queue
> rec=[722] Log Message=Received ringid(192.168.100.1:24) seq 10
> rec=[723] Log Message=mcasted message added to pending queue
> rec=[724] Log Message=releasing messages up to and including f
> rec=[725] Log Message=Delivering 10 to 12
> rec=[726] Log Message=Delivering MCAST message with seq 11 to pending
> delivery queue
> rec=[727] Log Message=Delivering MCAST message with seq 12 to pending
> delivery queue
> rec=[728] Log Message=Received ringid(192.168.100.1:24) seq 11
> rec=[729] Log Message=Received ringid(192.168.100.1:24) seq 12
> rec=[730] Log Message=Received ringid(192.168.100.1:24) seq 13
> rec=[731] Log Message=Delivering 12 to 13
> rec=[732] Log Message=Delivering MCAST message with seq 13 to pending
> delivery queue
> rec=[733] Log Message=Received ringid(192.168.100.1:24) seq 14
> rec=[734] Log Message=Delivering 13 to 14
> rec=[735] Log Message=Delivering MCAST message with seq 14 to pending
> delivery queue
> rec=[736] Log Message=comparing: sender r(0) ip(192.168.100.2) ;
> members(old:2 left:0)
> rec=[737] Log Message=comparing: sender r(0) ip(192.168.100.1) ;
> members(old:2 left:0)
> rec=[738] Log Message=chosen downlist: sender r(0) ip(192.168.100.1) ;
> members(old:2 left:0)
> rec=[739] Log Message=mcasted message added to pending queue
> rec=[740] Log Message=releasing messages up to and including 10
> rec=[741] Log Message=Delivering 14 to 15
> rec=[742] Log Message=Delivering MCAST message with seq 15 to pending
> delivery queue
> rec=[743] Log Message=confchg entries 2
> rec=[744] Log Message=Barrier Start Received From 1
> rec=[745] Log Message=Barrier completion status for nodeid 1 = 1.
> rec=[746] Log Message=Barrier completion status for nodeid 2 = 0.
> rec=[747] Log Message=Received ringid(192.168.100.1:24) seq 15
> rec=[748] Log Message=Received ringid(192.168.100.1:24) seq 16
> rec=[749] Log Message=Delivering 15 to 16
> rec=[750] Log Message=Delivering MCAST message with seq 16 to pending
> delivery queue
> rec=[751] Log Message=confchg entries 2
> rec=[752] Log Message=Barrier Start Received From 2
> rec=[753] Log Message=Barrier completion status for nodeid 1 = 1.
> rec=[754] Log Message=Barrier completion status for nodeid 2 = 1.
> rec=[755] Log Message=Synchronization barrier completed
> rec=[756] Log Message=Committing synchronization for (corosync cluster
> closed process group service v1.01)
> rec=[757] Log Message=mcasted message added to pending queue
> rec=[758] Log Message=releasing messages up to and including 14
> rec=[759] Log Message=Delivering 16 to 17
> rec=[760] Log Message=Delivering MCAST message with seq 17 to pending
> delivery queue
> rec=[761] Log Message=Received ringid(192.168.100.1:24) seq 17
> rec=[762] Log Message=Received ringid(192.168.100.1:24) seq 18
> rec=[763] Log Message=Delivering 17 to 18
> rec=[764] Log Message=Delivering MCAST message with seq 18 to pending
> delivery queue
> rec=[765] Log Message=Completed service synchronization, ready to
> provide service.
> rec=[766] Log Message=releasing messages up to and including 16
> rec=[767] Log Message=releasing messages up to and including 18
> rec=[768] Log Message=lib_init_fn: conn=0x2070ff0
> rec=[769] Log Message=exit_fn for conn=0x2070ff0
> rec=[770] Log Message=lib_init_fn: conn=0x2070dc0
> rec=[771] Log Message=exit_fn for conn=0x2070dc0
> rec=[772] Log Message=lib_init_fn: conn=0x2070dc0
> rec=[773] Log Message=exit_fn for conn=0x2070dc0
> rec=[774] Log Message=lib_init_fn: conn=0x2070dc0
> Finishing replay: records found [774]
_______________________________________________
discuss mailing list
discuss@xxxxxxxxxxxx
http://lists.corosync.org/mailman/listinfo/discuss
